The Problem with Pure LLMs
LLMs are trained on static data. They can't access real-time information, internal documents, or your company's knowledge base. They also "hallucinate" — generating plausible but incorrect answers.
RAG solves this by retrieving relevant documents first, then using them as context for the LLM.
How RAG Works
- ●Index — Split documents into chunks, embed them, store in a vector database
- ●Retrieve — When a query comes in, find the most relevant chunks
- ●Generate — Pass retrieved chunks + query to the LLM for an answer
Simple RAG Implementation
python
class="text-purple-400 font-medium">from langchain_community.document_loaders class="text-purple-400 font-medium">import TextLoader
class="text-purple-400 font-medium">from langchain.text_splitter class="text-purple-400 font-medium">import RecursiveCharacterTextSplitter
class="text-purple-400 font-medium">from langchain_community.vectorstores class="text-purple-400 font-medium">import FAISS
class="text-purple-400 font-medium">from langchain_openai class="text-purple-400 font-medium">import OpenAIEmbeddings, ChatOpenAI
class="text-purple-400 font-medium">from langchain.chains class="text-purple-400 font-medium">import RetrievalQA
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">1. Load documents
loader = TextLoader(class="text-emerald-class="text-amber-300">400">"docs/my_notes.txt")
documents = loader.load()
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">2. Split into chunks
splitter = RecursiveCharacterTextSplitter(
chunk_size=class="text-amber-300">500,
chunk_overlap=class="text-amber-300">50
)
chunks = splitter.split_documents(documents)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">3. Create vector store
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(chunks, embeddings)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">4. Create RAG chain
llm = ChatOpenAI(model=class="text-emerald-class="text-amber-300">400">"gpt-class="text-amber-300">4", temperature=class="text-amber-300">0)
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
retriever=vectorstore.as_retriever(search_kwargs={class="text-emerald-class="text-amber-300">400">"k": class="text-amber-300">3}),
return_source_documents=True
)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">5. Query
result = qa_chain.invoke({class="text-emerald-class="text-amber-300">400">"query": class="text-emerald-class="text-amber-300">400">"What is this document about?"})
print(result[class="text-emerald-class="text-amber-300">400">"result"])Chunking Strategies
python
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Fixed-size chunks (simple)
splitter = RecursiveCharacterTextSplitter(
chunk_size=class="text-amber-300">500,
chunk_overlap=class="text-amber-300">50,
separators=[class="text-emerald-class="text-amber-300">400">"\n\n", class="text-emerald-class="text-amber-300">400">"\n", class="text-emerald-class="text-amber-300">400">". ", class="text-emerald-class="text-amber-300">400">" "]
)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Semantic chunks (better)
class="text-purple-400 font-medium">from langchain_experimental.text_splitter class="text-purple-400 font-medium">import SemanticChunker
splitter = SemanticChunker(embeddings, breakpoint_threshold_type=class="text-emerald-class="text-amber-300">400">"percentile")Evaluating RAG
python
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Key metrics
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">1. Retrieval precision: Are retrieved docs relevant?
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">2. Answer faithfulness: Does the answer come class="text-purple-400 font-medium">from the docs?
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">3. Answer relevance: Does it actually answer the question?
class="text-purple-400 font-medium">from ragas class="text-purple-400 font-medium">import evaluate
class="text-purple-400 font-medium">from ragas.metrics class="text-purple-400 font-medium">import faithfulness, answer_relevancy
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Run evaluation
result = evaluate(
dataset=eval_dataset,
metrics=[faithfulness, answer_relevancy]
)Key Takeaways
- ●RAG grounds LLM responses in real documents, reducing hallucination
- ●Good chunking and embedding strategies are crucial for retrieval quality
- ●Always set
temperature=0for factual RAG applications - ●Evaluate both retrieval quality and answer quality separately