Crawl4AI Setup Guide: LLM-Ready Web Scraping for Local RAG (with Ollama)

Short answer: pip install -U crawl4ai && crawl4ai-setup gets you Crawl4AI v0.9.2 (Apache-2.0, ~76.1k GitHub stars as of August 2026) — the best free tool for turning web pages into LLM-ready markdown. On our test machine it crawled a docs page to clean markdown in 6.4 seconds, its content filter cut 74% of the page's boilerplate, and the complete crawl → embed (Ollama) → store (ChromaDB) → query loop finished in under 8 seconds. No API key, no cloud, $0.

That last part is the whole reason this tool belongs on this site. Most "web scraping for RAG" tutorials end with an API bill. Crawl4AI is a Python library that runs on your machine, outputs markdown that drops straight into the local RAG pipeline we already built, and asks for nothing. This guide is the setup, a real crawl-to-answer demo with measured timings, and the honest list of where it falls short.

---

What Crawl4AI Is (and What It Is Not) {#what-is-crawl4ai}

Answer first: Crawl4AI is an open-source Python crawler whose output format is markdown instead of raw HTML — because markdown is what LLMs and RAG pipelines actually want. It drives a real headless browser, so JavaScript-rendered pages work, and its content filters strip navigation and boilerplate before the text ever reaches your embedder.

The pitch in one comparison. All figures are GitHub star counts and licenses as listed on each project's repo, checked August 2026:

Firecrawl is the bigger project and a genuinely good product — but its core is AGPL and its center of gravity is a hosted API. Scrapy is the veteran for industrial scraping, but it hands you raw extraction and leaves the LLM-prep layer to you. Crawl4AI sits exactly in our lane: permissive license, pip-installable, markdown out, nothing phones home.

What it is not: a point-and-click app (it is a library — you write Python), and not the fastest option for plain static HTML, where a requests-based fetcher will always beat a real browser.

---

Install in 3 Commands {#install}

Answer first: pip install -U crawl4ai, then crawl4ai-setup, then crawl4ai-doctor to verify — that is the entire install, per the official docs. On our machine crawl4ai-setup finished in 52 seconds, almost all of it a 93.5 MiB Chrome Headless Shell (Chromium 149) download via Playwright.

# 1. The library (we got v0.9.2, the current release)
pip install -U crawl4ai

# 2. Browser dependencies for regular + undetected modes
crawl4ai-setup

# 3. Sanity check the whole install
crawl4ai-doctor

We ran this on Python 3.12 in a fresh venv; crawl4ai-setup finished with a clean "Post-installation setup completed!" and the doctor passed. Two version notes worth your time: pin the version in anything you deploy — the project shipped five releases between early June and mid-July 2026 (0.8.8 through 0.9.2, per the GitHub releases page) — and use a virtual environment, because the dependency stack is substantial. Optional extras exist for heavier features — pip install "crawl4ai[torch]" for PyTorch-based clustering, [transformer] for local HF models, [all] for everything (per the official installation docs) — but the base install is all this guide needs.

---

First Crawl: Any Page to Markdown in Six Lines {#first-crawl}

Answer first: six lines of Python turn a URL into clean markdown — our first crawl took 6.44 seconds including browser startup; repeat crawls in the same session are faster since the browser is already up.

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://docs.crawl4ai.com/core/installation/")
        print(result.markdown)

asyncio.run(main())

That is the official quickstart pattern, and it works exactly as advertised. result.markdown is real markdown — headings, lists, code fences preserved — not the tag-soup text you get from naive HTML-to-text conversion. For a one-off page into a notes file, you can stop reading here.

For RAG, you should not stop here, because raw markdown still contains everything on the page: nav bars, footers, cookie banners, "edit this page" links. That junk gets embedded, retrieved, and quoted at you later. Which is why the next section is the one that matters.

---

fit_markdown: Cutting the Boilerplate Before It Poisons Retrieval {#fit-markdown}

Answer first: attach a PruningContentFilter and read result.markdown.fit_markdown instead — on our test page it cut 11,798 characters of raw markdown down to 3,043 (74% removed), and everything it removed was navigation and boilerplate.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

prune = PruningContentFilter(
    threshold=0.45,          # lower keeps more, higher prunes more
    threshold_type="dynamic",
    min_word_threshold=5,    # drop nodes with fewer than 5 words
)
config = CrawlerRunConfig(
    markdown_generator=DefaultMarkdownGenerator(content_filter=prune)
)

# inside the crawler:
result = await crawler.arun(url=URL, config=config)
clean = result.markdown.fit_markdown   # filtered
full  = result.markdown.raw_markdown   # everything

The pruning filter scores DOM nodes by text density and link density and drops the low-value ones — no LLM involved, so it costs milliseconds. There is a second filter worth knowing: BM25ContentFilter(user_query="...", bm25_threshold=1.2) keeps only content relevant to a query, which is useful when you crawl broad pages but only care about one topic. Both attach the same way, via DefaultMarkdownGenerator(content_filter=...).

A 74% cut sounds aggressive, and on some pages it is — see limitations for when to lower the threshold. But for docs sites, blogs, and news pages, this single feature is the difference between a RAG index that answers questions and one that confidently retrieves your cookie policy. If you want the theory of why clean chunks beat big chunks, our local RAG setup guide covers it.

---

The Full Crawl-to-RAG Pipeline: Measured, Under 8 Seconds {#crawl-to-rag}

Answer first: crawl → prune → chunk → embed with Ollama's nomic-embed-text → store in ChromaDB → answer a query ran end-to-end in under 8 seconds on our Apple M3 Pro (18GB) — and every piece of it is free and local.

These are our own measurements (Crawl4AI 0.9.2, Python 3.12, macOS, August 2026), crawling the Crawl4AI installation docs page as the test document:

The code, condensed — this is the same architecture as our Ollama + ChromaDB RAG pipeline, with Crawl4AI replacing the file loader as the ingestion step:

import asyncio, chromadb, ollama
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

URL = "https://docs.crawl4ai.com/core/installation/"

async def crawl_clean(url):
    prune = PruningContentFilter(threshold=0.45, threshold_type="dynamic", min_word_threshold=5)
    cfg = CrawlerRunConfig(markdown_generator=DefaultMarkdownGenerator(content_filter=prune))
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url=url, config=cfg)
    return result.markdown.fit_markdown

def chunk(text, size=1000, overlap=150):
    out, i = [], 0
    while i < len(text):
        out.append(text[i:i + size])
        i += size - overlap
    return out

md = asyncio.run(crawl_clean(URL))
chunks = chunk(md)

embs = [ollama.embeddings(model="nomic-embed-text", prompt=c)["embedding"] for c in chunks]

client = chromadb.PersistentClient(path="./crawl_rag_db")
col = client.get_or_create_collection("web_docs")
col.add(ids=[f"c{i}" for i in range(len(chunks))], embeddings=embs, documents=chunks)

question = "How do I install Crawl4AI and set up the browser?"
q_emb = ollama.embeddings(model="nomic-embed-text", prompt=question)["embedding"]
hits = col.query(query_embeddings=[q_emb], n_results=2)
print(hits["documents"][0][0])   # feed these chunks to your chat model as context

From here, generation is one more Ollama call with the retrieved chunks as context — the pipeline guide has that step, plus a FastAPI wrapper. We also timed the loop at corpus scale, with generation included: an 8-page deep crawl of the Crawl4AI docs (7.3s) produced 50 chunks, batch-embedding them via ollama.embed(model="nomic-embed-text", input=chunks) took 2.4s, the ChromaDB write 0.1s, and retrieving the top 4 chunks plus generating an answer with llama3.2:3b took 8.8s — 18.6 seconds total from URL to a correct, grounded answer. We asked "How do I get fit_markdown instead of raw markdown?", a question answerable only from the crawled pages, and the 3B model pointed at PruningContentFilter plus DefaultMarkdownGenerator and reproduced a working config. A model that small getting documentation questions right is retrieval doing its job, not the model being clever. If you are choosing an embedding model, nomic-embed-text is our default for a reason, but the trade-offs are in the local embeddings guide; and if ChromaDB is not your vector store, the same code shape works with anything in our vector database comparison.

Hardware note: none of this needs a GPU. The crawl is network-bound, nomic-embed-text is a 274MB model that runs fine on CPU, and ChromaDB is just disk. A modest machine from our local agent hardware guide handles the whole stack; the GPU only starts mattering at the generation step, and even there a 3B-8B model is plenty for grounded Q&A — the current picks are in our best Ollama models for 8GB VRAM rundown.

---

Deep Crawling: A Whole Docs Site in One Call {#deep-crawling}

Answer first: BFSDeepCrawlStrategy(max_depth=1, max_pages=8) crawled 8 pages of the Crawl4AI docs in 7.3 seconds on our machine, pruning 115,652 characters of raw markdown down to 51,048 of RAG-ready content (a 56% cut) — a whole small knowledge base in one function call.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
from crawl4ai.deep_crawling.filters import FilterChain, DomainFilter, URLPatternFilter

config = CrawlerRunConfig(
    markdown_generator=md_generator,   # same pruning setup as above
    deep_crawl_strategy=BFSDeepCrawlStrategy(
        max_depth=1,            # start page + 1 level of links
        include_external=False, # stay on the domain
        max_pages=8,            # hard cap — always set one
        filter_chain=FilterChain([
            DomainFilter(allowed_domains=["docs.example.com"]),
            URLPatternFilter(patterns=["*guide*", "*tutorial*"]),
        ]),
    ),
    check_robots_txt=True,      # defaults to False — turn it on
)

results = await crawler.arun(url="https://docs.example.com/", config=config)
for r in results:                       # a list, one result per page
    if r.success:
        index(r.markdown.fit_markdown)  # your chunk/embed/store from above

Breadth-first search, domain and URL-pattern filters, an optional score_threshold for URL relevance scoring, and stream=True if you want results as an async iterator instead of a list at the end. Three rules we would insist on: always set max_pages (depth grows link counts brutally fast), keep include_external=False unless you genuinely want to leave the site, and set check_robots_txt=True — we checked the installed v0.9.2 and it defaults to False, which is the difference between a polite corpus builder and an accidental nuisance.

This is the point where Crawl4AI stops being a scraping tool and becomes an ingestion engine — point it at your company wiki, your product docs, or a vendor's manual, and the RAG agent on top of it suddenly knows things.

---

Structured Extraction with Ollama (No API Key) {#llm-extraction}

Answer first: LLMExtractionStrategy with LLMConfig(provider="ollama/llama3.2") pulls structured JSON out of pages using your local Ollama model — the docs confirm local models need no API token.

Markdown is right for RAG, but sometimes you want fields — prices, specs, titles — as JSON. Crawl4AI can run an LLM over the crawled content during the crawl, and the provider string format is ollama/<model-tag>, so any model you have pulled works:

from pydantic import BaseModel
from crawl4ai import LLMExtractionStrategy, LLMConfig, CrawlerRunConfig

class Product(BaseModel):
    name: str
    price: str

strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(provider="ollama/llama3.2"),  # no api_token needed locally
    schema=Product.model_json_schema(),
    extraction_type="schema",
    instruction="Extract every product name and price on the page.",
    chunk_token_threshold=1000,
    apply_chunking=True,
    input_format="markdown",
)
config = CrawlerRunConfig(extraction_strategy=strategy)
result = await crawler.arun(url=URL, config=config)
print(result.extracted_content)   # JSON matching your schema

Honest advice: use this selectively. Every chunk of every page becomes an LLM call, and on local hardware that turns a 7-second crawl into minutes on a big site. For repetitive page structures, Crawl4AI also has CSS-selector-based extraction that costs nothing — reach for the LLM strategy when the structure varies too much for selectors.

---

The Docker Option {#docker}

Answer first: docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:latest gives you Crawl4AI as a self-hosted REST service with a dashboard at localhost:11235 — use it when the consumer is not Python.

docker pull unclecode/crawl4ai:latest
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest
# dashboard: http://localhost:11235/dashboard

These commands are from the project README (the docs site's installation page still shows an older experimental image tag — trust the README here). The Docker route makes sense if your pipeline is in Node or n8n-style automation and you want crawling as an HTTP endpoint rather than a library import. For a pure-Python RAG stack, pip is simpler and one less moving part.

---

Honest Limitations {#limitations}

Answer first: Crawl4AI is the right default for local RAG ingestion, but it is a real browser (heavy), its pruning filter can over-cut, LLM extraction is slow on local models, and none of this exempts you from robots.txt and terms of service.

• It ships a browser. crawl4ai-setup pulled a 93.5 MiB headless Chromium on top of a substantial Python dependency stack, and every live browser instance holds real RAM while it runs. For static HTML at scale, a requests-based fetcher is far cheaper — Crawl4AI earns its weight on JavaScript-rendered pages.
• Pruning is a heuristic, not a reader. Our 74% cut was accurate on a docs page, but density-based filtering can eat sparse-but-real content — short FAQs, tables of numbers, changelogs. Spot-check fit_markdown against raw_markdown on each new site and lower threshold if content is going missing.
• First-crawl latency. 6.44s of our single-page time was dominated by browser startup. Batch your crawls in one session (as the deep-crawl demo does — 8 pages in 7.3s) rather than paying startup per page.
• The release cadence is brisk. Five releases in roughly six weeks this summer (0.8.8 on June 4 through 0.9.2 on July 15, per GitHub). The APIs on this page were stable across them, but pin your version in production and read release notes before upgrading.
• LLM extraction multiplies cost. Schema extraction calls your model per chunk. On local hardware, keep it for the pages that need it; use CSS extraction or plain fit_markdown everywhere else.
• Docs occasionally lag the code. The installation page's Docker section still describes an older experimental image while the README documents the current one. When they disagree, the README and release notes win.
• Anti-bot walls exist. Crawl4AI has stealth features, but heavily protected sites (login walls, aggressive bot detection) will still block or captcha you. That is a property of the modern web, not a bug in the tool.
• Legality is on you. Respect robots.txt — remembering that check_robots_txt defaults to False — rate-limit yourself, and read the terms of the sites you crawl, especially before feeding the output into anything commercial.

None of these change the verdict. For getting web content into a local RAG pipeline at $0, nothing else we have tested combines a permissive license, a pip install, and markdown output this clean.

---

Sources {#sources}

• unclecode/crawl4ai — GitHub repository: star count, license, install and Docker commands (retrieved August 2026)
• Crawl4AI releases — version history: v0.9.2 (July 15, 2026) and the June-July release cadence
• Crawl4AI quickstart and installation docs — config classes, raw vs fit markdown, optional extras
• Crawl4AI deep-crawl docs — BFS/DFS/BestFirst strategies, filters, parameters
• Crawl4AI LLM extraction docs — LLMConfig Ollama provider format, no-token local usage
• firecrawl/firecrawl and scrapy/scrapy — comparison table star counts and licenses (retrieved August 2026)
• Our own test runs, August 2026 — all timings, character counts, and the check_robots_txt default verification on Crawl4AI 0.9.2, Python 3.12, Apple M3 Pro (18GB RAM), Ollama with nomic-embed-text and llama3.2:3b

---

FAQ {#faq}