// LLM Engineering 2026 ⏱ ~3 min read

RAG Tutorial 2026: Build a Production Chatbot with LangChain + ChromaDB

Ayub Shah

· 📅 April 2026 · 👤 ML engineers & data scientists

⚡ Quick Answer

This RAG tutorial teaches you to build a production-ready retrieval-augmented chatbot with LangChain and ChromaDB in 2026. Learn the complete pipeline: document ingestion, chunking strategies (800 chars with 120 overlap), embeddings with text-embedding-3-small, MMR retrieval for diverse results, FastAPI serving, and RAGAS evaluation (faithfulness + answer relevancy). Includes production checklist: cache embeddings, rerankers, and logging. Start with recursive character splitting; graduate to hybrid search and parent document retrieval as you scale.

87%of ML projects never reach production

50k+searches per month

25minto complete this tutorial

Looking for a complete RAG tutorial for 2026? This guide shows you how to build a production-ready retrieval-augmented chatbot using LangChain and ChromaDB — with real code, chunking strategies, MMR retrieval, evaluation, and a deployment checklist.

This RAG tutorial is designed for ML engineers and developers who want to move beyond demos and build RAG pipelines that survive contact with real users. No prior RAG experience required.

Table of Contents

01What RAG Actually IsEssential
02Architecture OverviewSystem
03Project Setup & DependenciesSetup
04Document Ingestion PipelineStep 1
05Chunking Strategies That MatterStep 2
06Embeddings and ChromaDBStep 3
07Retrieval Chain + FastAPI ServerStep 4
08Evaluating Your RAG PipelineTesting
09Production ChecklistChecklist
10Where to Go From HereNext

01 What RAG Actually Is — and What It Isn't

Let's be direct: most RAG tutorial content online shows you how to ask a question about a single PDF. That's not RAG in production. That's a weekend demo.

Retrieval-Augmented Generation is the pattern of giving an LLM access to an external knowledge base at query time. Instead of relying purely on training-time knowledge, you fetch relevant context dynamically and include it in the prompt. The model answers based on what you gave it — not just what it memorized.

Why does this RAG tutorial matter in production?

Your data changes

LLMs have a cutoff date. Your internal docs, support tickets, and product FAQs don't freeze in time.

You can cite sources

Every answer has a paper trail — you know exactly which documents produced it.

It's cheaper than fine-tuning

Updating a vector store costs almost nothing compared to retraining.

You stay in control

You decide what the model can and cannot see.

⚠️

COMMON MISCONCEPTION

RAG is not a replacement for fine-tuning. Fine-tuning changes how the model reasons and responds. RAG gives it access to new facts. They serve different goals and often work best together.

02 Architecture Overview

Before touching any code — here's the complete pipeline in two phases. The key insight: ingestion and querying are separate concerns. Keep them that way in your codebase.

Raw Docs
→
Chunker
→
Embedder
→
ChromaDB

// Ingestion pipeline (runs offline / on schedule)

User Query
→
Embed Query
→
Vector Search
→
LLM
→
Answer

// Query pipeline (runs at request time)

03 Project Setup & Dependencies

BASHterminal
python -m venv .venv
source .venv/bin/activate

pip install langchain langchain-openai langchain-community \
            chromadb openai tiktoken \
            pypdf python-dotenv fastapi uvicorn

				
					
				1
2
3
4
5
6

						python -m venv .venv
source .venv/bin/activate
 
pip install langchain langchain-openai langchain-community \
            chromadb openai tiktoken \
            pypdf python-dotenv fastapi uvicorn

					

			

ENV.env
OPENAI_API_KEY=sk-...
CHROMA_PERSIST_DIR=./chroma_db
EMBED_MODEL=text-embedding-3-small
LLM_MODEL=gpt-4o-mini

				
					
				1
2
3
4

						OPENAI_API_KEY=sk-...
CHROMA_PERSIST_DIR=./chroma_db
EMBED_MODEL=text-embedding-3-small
LLM_MODEL=gpt-4o-mini

					

			

💡

MODEL CHOICE

text-embedding-3-small is the sweet spot for embeddings in 2026 — cheap, fast, and accurate. For the LLM, gpt-4o-mini gives great quality-to-cost. Swap in Claude or Gemini if you prefer — LangChain abstracts the backend cleanly.

04 Document Ingestion Pipeline

PYTHONingest.py
import os
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

load_dotenv()

def build_vector_store(docs_dir="./data/docs"):
    loader = DirectoryLoader(docs_dir, glob="**/*.pdf", loader_cls=PyPDFLoader)
    docs = loader.load()
    
    splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120)
    chunks = splitter.split_documents(docs)
    
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vectordb = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")
    return vectordb

				
					
				1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

						import os
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
 
load_dotenv()
 
def build_vector_store(docs_dir="./data/docs"):
    loader = DirectoryLoader(docs_dir, glob="**/*.pdf", loader_cls=PyPDFLoader)
    docs = loader.load()
    
    splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120)
    chunks = splitter.split_documents(docs)
    
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vectordb = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")
    return vectordb

					

			

05 Chunking Strategies That Actually Matter

This is where most RAG tutorial content cuts corners — and where most production RAG pipelines silently fail. How you split determines whether retrieval returns useful context or incoherent garbage.

📌

RULE OF THUMB

A RAG pipeline is only as good as the chunks it retrieves. Bad chunking means the right information is split across boundaries the model can never bridge.

Strategy	Best For	Gotcha
Fixed-size	Uniform data, fast indexing	Breaks mid-sentence constantly
Recursive character	General purpose	Not structure-aware
Semantic	Long articles, papers	Slower; embeds at split time

For most production use cases, recursive character splitting with 800 characters and ~15% overlap (120 chars) is where to start. Your chunk size should be smaller than the "useful unit of information" in your documents.

06 Embeddings and ChromaDB

An embedding is a high-dimensional vector representing the semantic meaning of text. Two chunks about the same concept should have vectors pointing in roughly the same direction — even if they share no words in common.

ChromaDB is our vector store. It runs in-process, persists to disk, and has a clean Python API. It's the right call for most teams building their first production RAG. When you scale past ~500k chunks, look at Pinecone, Weaviate, or pgvector.

07 Retrieval Chain + FastAPI Server

PYTHONretriever.py
from langchain_openai import ChatOpenAI
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

def build_rag_chain(vectordb):
    retriever = vectordb.as_retriever(
        search_type="mmr",
        search_kwargs={"k": 5, "fetch_k": 20}
    )
    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.1)
    
    prompt = ChatPromptTemplate.from_messages([
        ("system", "Answer using ONLY the context below.\n\n{context}"),
        ("human", "{input}"),
    ])
    
    qa_chain = create_stuff_documents_chain(llm, prompt)
    return create_retrieval_chain(retriever, qa_chain)

				
					
				1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

						from langchain_openai import ChatOpenAI
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate
 
def build_rag_chain(vectordb):
    retriever = vectordb.as_retriever(
        search_type="mmr",
        search_kwargs={"k": 5, "fetch_k": 20}
    )
    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.1)
    
    prompt = ChatPromptTemplate.from_messages([
        ("system", "Answer using ONLY the context below.\n\n{context}"),
        ("human", "{input}"),
    ])
    
    qa_chain = create_stuff_documents_chain(llm, prompt)
    return create_retrieval_chain(retriever, qa_chain)

					

			

📌

WHY MMR MATTERS

Default similarity search returns near-duplicate chunks. MMR (Maximum Marginal Relevance) trades a little similarity for diversity, giving broader coverage and noticeably better answers.

08 Evaluating Your RAG Pipeline

Two metrics that matter most in practice:

Context Precision — What fraction of retrieved chunks were actually relevant?
Faithfulness — Does the answer stick to the retrieved context?

For a rigorous setup, use RAGAS — a Python library for evaluating faithfulness, answer relevancy, and context recall using LLM-based judges.

PYTHONevaluate.py
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy

result = evaluate(dataset, metrics=[faithfulness, answer_relevancy])
print(f"Faithfulness: {result['faithfulness']:.2f}")

				1
2
3
4
5

						from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
 
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy])
print(f"Faithfulness: {result['faithfulness']:.2f}")

09 Production Checklist

✓

Cache embeddings

Redis cache cuts OpenAI bills significantly.

✓

Handle stale documents

Track document hashes, re-ingest only changed files.

✓

Add a reranker

Cohere Rerank or BAAI/bge-reranker delivers the biggest quality improvement.

✓

Log everything

Question, chunks, answer, latency, feedback.

✓

Access control from day one

Use ChromaDB's where filters.

10 Where to Go From Here

💬

Conversational Memory

Add ConversationBufferWindowMemory for multi-turn chat.

🔍

Hybrid Search

Combine vector similarity with BM25 via EnsembleRetriever.

📄

Parent Document Retrieval

Index small chunks, retrieve parent documents for better context.

🤖

Self-RAG

Model decides whether retrieval is needed based on the query.

✦

FINAL WORD

The most important discipline is measurement. A RAG system without evals is just vibes engineering. Start measuring from day one.

📖 External resources: LangChain Documentation • ChromaDB • RAGAS Documentation

📚