Vector Stores and Embeddings

🧑‍🏫 Sabse pehle — simple mein samjho#

Agar tumhare paas 10,000 text chunks (paragraphs) hain, aur user sawal puchta hai "How to reset password?", toh computer ko kaise pata chalega ki konsa chunk padhna hai? Normal search (keyword search) fail ho jayegi agar document mein "reset password" ke bajaye "recover account" likha ho. Yahi par Embeddings aate hain. Embeddings text ko numbers (math vectors) mein badal dete hain. "Apple" aur "Orange" ke numbers aas-paas honge. Fir in numbers ko hum Vector Store (database) mein save kar lete hain. Jab user sawal puchta hai, hum us sawal ko bhi number mein badalte hain aur Vector Store se uske sabse kareeb (semantic match) wale paragraphs nikal lete hain!

1. Embeddings#

An embedding model takes a piece of text and converts it into a massive array of floating-point numbers (a vector). This vector represents the "meaning" or "concept" of the text.

from langchain_openai import OpenAIEmbeddings

# Initialize the embedding model
embeddings_model = OpenAIEmbeddings()

# Embed a single query
vector = embeddings_model.embed_query("Apple is a fruit")

print(len(vector)) # Usually 1536 dimensions for OpenAI models!
print(vector[:5])  # [-0.012, 0.034, -0.001, 0.056, ...]

Note: Generating embeddings costs money/credits (though much less than generating text). You only generate embeddings once when saving documents, and once per user query.

2. Vector Stores#

A Vector Store is a specialized database designed to store these massive arrays of numbers and perform extremely fast mathematical comparisons (Cosine Similarity) to find vectors that are close to each other.

Popular Vector Stores:

• Chroma / FAISS: Great for local development, runs in-memory or saves to a local file.
• Pinecone / Weaviate: Hosted cloud solutions for massive production scale.
• PostgreSQL (pgvector): If you already use Postgres, you can add a plugin to make it a vector store!

Creating a Vector Store#

Let's take the all_splits (the chunked documents from the previous section) and save them into a Chroma vector store.

# pip install chromadb
from langchain_community.vectorstores import Chroma

# This single line does three massive things:
# 1. Takes all the text chunks.
# 2. Sends them to OpenAI to convert them into embeddings (numbers).
# 3. Saves both the text and the numbers into the Chroma database.
vectorstore = Chroma.from_documents(
    documents=all_splits, 
    embedding=embeddings_model
)

Searching the Vector Store (Similarity Search)#

Now that the database is populated, we can search it! This does NOT use the LLM to generate an answer. It simply returns the pieces of text that mathematically match the meaning of the query.

question = "What are the main capabilities of autonomous agents?"

# Perform a similarity search and return the top 4 matching chunks
matching_docs = vectorstore.similarity_search(question, k=4)

print(f"Found {len(matching_docs)} relevant chunks.")

# Print the text of the most relevant chunk
print(matching_docs[0].page_content)

Now we have successfully retrieved the raw paragraphs containing the answer. In the next section, we will feed these paragraphs to the LLM so it can read them and generate a human-friendly response (RAG).