Build a RAG Chatbot with LangChain and ChromaDB

Learn how to build a Retrieval-Augmented Generation (RAG) chatbot using LangChain and ChromaDB for intelligent document querying and context-aware responses.

Large Language Models (LLMs) are exceptionally powerful at understanding natural language, reasoning, and generating human-like text. However, they suffer from fundamental limitations when asked about private corporate documentation, proprietary knowledge bases, or recent events outside their training data. Fine-tuning models is often prohibitively expensive and slow for dynamic data.

This is where **Retrieval-Augmented Generation (RAG)** comes in. By combining an LLM with an external vector search database, you can dynamically retrieve relevant context from your documents and inject it directly into the prompt. This comprehensive guide walks you through building a production-ready RAG chatbot in Python using **LangChain** and **ChromaDB**.

1. Understanding RAG Architecture

A standard RAG pipeline operates across two primary phases: ingestion and retrieval-generation.

Data Ingestion & Chunking: Raw documents (PDFs, text files, markdown, or web pages) are loaded, parsed into smaller, manageable text chunks, and converted into dense vector embeddings using an embedding model.
Vector Storage: These embeddings are stored in a high-performance vector database like ChromaDB, optimized for fast similarity search.
Query Processing: When a user asks a question, the query is converted into a vector embedding and matched against the vector database to retrieve the top-k most relevant document chunks.
Generation: The retrieved chunks are bundled together with the user's prompt as context and sent to the LLM to generate an accurate, hallucination-free answer.

2. Environment Setup and Dependencies

Let's start by setting up our Python virtual environment and installing the required libraries, including LangChain, ChromaDB, and OpenAI integrations.

              
C

            # Create and activate a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows use: venv\Scripts\activate

# Install LangChain, ChromaDB, and OpenAI client libraries
pip install langchain langchain-openai chromadb sentence-transformers tiktoken

Make sure to set your OpenAI API key as an environment variable before running your Python scripts:

              
C

            export OPENAI_API_KEY="your-openai-api-key-here"

3. Loading and Chunking Source Documents

To build our knowledge base, we need to load documents and split them into smaller chunks so that our retrieval mechanism can pinpoint precise sections of text.

              
            
C

            
            from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Load local document
loader = TextLoader("knowledge_base.txt")
docs = loader.load()

# Split documents into overlapping chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = text_splitter.split_documents(docs)

print(f"Successfully generated {len(chunks)} text chunks.")
          

4. Storing Embeddings in ChromaDB

ChromaDB is an open-source vector database designed to store and query embeddings efficiently. LangChain provides a seamless wrapper for Chroma integration.

              
            
C

            
            from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

# Initialize embedding model
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Create persistent Chroma vector store from chunks
vector_store = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

print("Vector database successfully initialized and persisted.")
          

5. Building the Conversational RAG Chain

Now that our vector database is populated, we can configure our retriever and set up a conversational chain using LangChain's expression language (LCEL).

              
            
C

            
            from langchain_openai import ChatOpenAI
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

# Initialize LLM
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.2)

# Configure retriever
retriever = vector_store.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 3}
)

# Define prompt template for RAG
system_prompt = (
    "You are an AI assistant for question-answering tasks. "
    "Use the following pieces of retrieved context to answer "
    "the question. If you don't know the answer, say that you "
    "don't know.\n\n"
    "{context}"
)

prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human", "{input}"),
])

# Create document combining and retrieval chains
question_answer_chain = create_stuff_documents_chain(llm, prompt)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)

# Execute a query
response = rag_chain.invoke({"input": "What are the core features of our product?"})
print("AI Response:", response["answer"])
          

Conclusion and Best Practices

Optimize Chunk Size: Experiment with different chunk sizes (e.g., 300 to 1000 tokens) depending on whether your source data consists of short FAQs or long technical manuals.
Hybrid Search: Combine dense vector search with sparse keyword search (BM25) for improved retrieval accuracy on domain-specific terminology.
Persistent Storage: Always configure a persist directory in ChromaDB to avoid re-embedding your entire corpus on every application restart.

By pairing LangChain with ChromaDB, you can build responsive, highly accurate AI chatbots that ground their reasoning strictly in your proprietary documentation.