RAG + Vector
Databases

By the end of this session, you will:

LLMs are powerful but have critical limitations

Choose the right approach for your use case

The offline step: prepare your knowledge base for search

Indexing Steps

1. Load documents (PDF, web, etc.)

2. Split into chunks

3. Embed each chunk

4. Store in Vector DB

---

One-time setup — run once per document update

The runtime step: answer questions using your knowledge

Query Steps

1. User asks question

2. Embed the query

3. Search vector DB

4. Retrieve top K chunks

5. Augment prompt

6. LLM generates answer

---

Every query — runs in real-time

Combining indexing and retrieval into a unified system

Indexing (Offline)

Load → Chunk → Embed → Store in Vector DB

Retrieval (Runtime)

Query → Embed → Search → Retrieve Top K

Generation

Augment Prompt → LLM → Grounded Answer

Imagine a space where every sentence has a position

Choose based on your use case

Before embedding, you need to split documents into chunks

Token Limits

Models have limits (512–8192 tokens)

Precision

Smaller chunks = precise retrieval

Balance

Too large = diluted; too small = lost context

Different approaches for different use cases

Specialized storage optimized for high-dimensional vectors

SELECT * FROM docs
WHERE title = 'RAG'

# Find 5 most similar
db.similarity_search(
  query_vector, k=5
)

Pick the right tool for your needs

Cosine Similarity — The most common metric

Measures the angle between vectors

-1.0 Opposite · 0.0 Unrelated · 1.0 Identical

Euclidean (L2)

Straight-line distance between points.

Dot Product

Similar to cosine but scale-sensitive.

Basic: embed query → cosine search → return top K. Here's how to do better:

Format retrieved chunks into the LLM prompt

# The RAG prompt template
prompt = """
You are a helpful assistant. Answer the question based ONLY on
the following context. If the context doesn't contain the answer,
say "I don't have enough information to answer that."

Context:
{retrieved_chunks}

Question: {user_question}
"""

Install dependencies and load your documents

# Install dependencies

pip install langchain langchain-openai langchain-community chromadb

# Load documents

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("your_document.pdf")
documents = loader.load()

Split documents into manageable pieces

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", " ", ""]
)

chunks = text_splitter.split_documents(documents)
print(f"Split into {len(chunks)} chunks")

Convert chunks to vectors and store in ChromaDB

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

Test your retrieval before wiring up the LLM

# Simple similarity search
results = vectorstore.similarity_search(
    "What are the key findings?",
    k=3
)

for doc in results:
    print(doc.page_content[:200])
    print(doc.metadata)
    print("---")

Connect retrieval to the LLM using modern agent pattern

from langchain_openai import ChatOpenAI
from langchain.agents import create_agent
from langchain.tools import tool

model = ChatOpenAI(model="gpt-4o-mini", temperature=0)

@tool(response_format="content_and_artifact")
def retrieve_context(query: str):
    """Retrieve information from documents."""
    docs = vectorstore.similarity_search(query, k=3)
    serialized = "\n\n".join(
        f"Source: {doc.metadata}\nContent: {doc.page_content}"
        for doc in docs
    )
    return serialized, docs

tools = [retrieve_context]
agent = create_agent(model, tools, system_prompt="Use the tool to answer queries.")

for event in agent.stream(
    {"messages": [{"role": "user", "content": "What is RAG?"}]},
    stream_mode="values"
):
    print(event["messages"][-1].content)

Know the symptoms and fixes

You can't improve what you can't measure

Once your basic RAG works, explore:

The essential points to remember

Extend your Week 1 POST /ask on Render into a RAG API — vector store, POST /ingest for new docs, and retrieval-first /ask — on data you care about.