Build a RAG Chatbot in 30 Minutes
A working RAG chatbot you can ask questions about your own documents — running 100% locally, no API keys, no cloud.
- 1
Install the stack
Run `pip install langchain langchain-ollama langchain-community chromadb pypdf streamlit` in a fresh virtualenv. Pull a small embedding model and a chat model with `ollama pull nomic-embed-text` and `ollama pull llama3.1:8b`. The whole install takes ~5 minutes on a normal connection.
- 2
Index your documents
Drop a folder of PDFs or .md files anywhere on disk. Use `langchain_community.document_loaders.DirectoryLoader` to load them, then `RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120)` to chunk. Pass the chunks through `OllamaEmbeddings(model="nomic-embed-text")` and store them in `Chroma.from_documents(...)`. This is your retrieval index.
- 3
Wire the retrieval chain
Build a `RetrievalQA` chain: `RetrievalQA.from_chain_type(llm=ChatOllama(model="llama3.1:8b"), retriever=vectordb.as_retriever(search_kwargs={"k": 4}), return_source_documents=True)`. Now any question gets answered using your documents as context — and you get the source chunks back so you can show citations in the UI.
- 4
Add the chat UI
Create `app.py` with `st.text_input("Ask a question")` and call the chain on every submit. Render the answer + the source documents as expandable cards. Run `streamlit run app.py` and you have a working chatbot at localhost:8501. From here you can swap in better chunking, hybrid search (BM25 + vectors), or a reranker — but the baseline already works.
- 5
Where it gets real
The 30-minute version handles a few PDFs nicely. Production RAG needs eval sets (Recall@k, MRR, nDCG), reranking with a cross-encoder, hybrid retrieval, query rewriting, and observability for retrieval quality. That entire stack — including 10 hands-on chapters with code — is the RAG Systems course on Local AI Master.
Continue with the full RAG Systems course on Local AI Master.
This page is one chapter of a structured course covering everything from foundations to production. Try Pro free for 7 days — full access to all 264 chapters across 10 courses, no charge until day 8, cancel anytime.