Document Loaders and Splitters

🧑‍🏫 Sabse pehle — simple mein samjho#

LLMs (jaise ChatGPT) ko sab kuch nahi pata hota. Agar tum chahate ho ki wo tumhari company ki private PDF padh kar answer de, toh tumhe wo PDF LLM ko bhejni padegi. Par LLM ki ek limit hoti hai (Context Window) — tum usko ek baar mein poori 500 page ki book nahi bhej sakte. Isliye hum pehle Document Loaders se file read karte hain, aur fir Text Splitters se us book ko chhote-chhote paragraphs (chunks) mein kaat dete hain taaki aage unko search kiya ja sake.

1. Document Loaders#

LangChain provides hundreds of Document Loaders to read data from almost any source: PDFs, CSVs, Notion databases, YouTube transcripts, Web pages, or AWS S3 buckets.

A Document Loader always returns a list of Document objects. A Document contains two things:

1. page_content: The actual extracted text string.
2. metadata: A dictionary containing info about the text (e.g., source file name, page number, URL).

Example: Loading a Webpage#

# You might need to install beautifulsoup4: pip install bs4
from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://lilianweng.github.io/posts/2023-06-23-agent/")

# .load() fetches the page and extracts the text
docs = loader.load()

print(f"Loaded {len(docs)} documents.")
print(docs[0].page_content[:100]) # Print the first 100 characters
print(docs[0].metadata) # {'source': 'https://...', 'title': 'LLM Powered Autonomous Agents'}

2. Text Splitters (Chunking)#

Once we have the raw text, we must divide it into smaller pieces. This is crucial for two reasons:

1. Context Limits: We cannot pass massive documents into the LLM prompt.
2. Search Accuracy: Later, when we search for the answer to a user's question, it's much easier to find the exact answer if the document is cut into specific, focused paragraphs rather than massive chapters.

The RecursiveCharacterTextSplitter#

This is the recommended text splitter in LangChain. It tries to keep related pieces of text together. It splits by paragraphs first. If a paragraph is too long, it splits by sentences. If a sentence is too long, it splits by words.

from langchain.text_splitter import RecursiveCharacterTextSplitter

# Initialize the splitter
text_splitter = RecursiveCharacterTextSplitter(
    # How big each chunk should be (in characters)
    chunk_size=1000,
    
    # Overlap ensures that a sentence split across two chunks doesn't lose its context
    chunk_overlap=200, 
    
    # Optional: Adds a metadata field to each chunk indicating where it starts
    add_start_index=True 
)

# Pass the documents from the loader into the splitter
all_splits = text_splitter.split_documents(docs)

print(f"Split {len(docs)} document into {len(all_splits)} smaller chunks.")

Now that we have our text neatly divided into 1000-character chunks, we need a way to search through them instantly to find the chunks relevant to the user's question. This leads us to Embeddings and Vector Stores in the next section.