RAG chunking lifecycle: How document chunking strategies impact AI response quality

Introduction

RAG Chunking Demo

1 / 4

What is RAG?

• Chunking – Break documents into smaller, useful pieces
• Embedding – Turn those chunks into vector representations
• Retrieval – Find the chunks most relevant to a given query
• Generation – Craft a response using the retrieved chunks

Why chunking matters

• LLMs have relatively short attention spans (aka token limits)
• Better chunks = better retrieval (semantic matching thrives on coherence)
• Messy chunks = lost meaning
• Efficient chunks = faster, smarter systems

The RAG chunking lifecycle

Stage 1: Document chunking

1. No Chunking
   • Pros: Full context preserved
   • Cons: Too much content for specific queries
2. Fixed-Size Chunking
   • Pros: Simple, predictable
   • Cons: Can split ideas awkwardly
3. Delimiter-Based Chunking
   • Pros: Honors natural breaks
   • Cons: Uneven chunk sizes
4. Recursive Chunking
   • Pros: Flexible, structure-aware
   • Cons: Slightly more complex
5. Semantic Chunking
   • Pros: Keeps related ideas together
   • Cons: Requires deeper processing (LLM-based or embedding-based)

Tools and Libraries to Make This Easier

• LangChain – Comes with RecursiveCharacterTextSplitter, TokenTextSplitter, and more. Easy to use, production-tested.
• LlamaIndex – Offers advanced semantic chunking with SemanticSplitterNodeParser using LLMs.
• Unstructured.io – Great for extracting content from real-world formats (PDFs, HTML, DOCX) into clean, structured chunks.
• spaCy / NLTK – Traditional NLP tools for sentence/paragraph/token splitting. Still great for rule-based chunking.

Should semantic chunking use an LLM?

• Interpret the text's meaning
• Segment based on conceptual breaks (not just formatting)
• Preserve context that traditional methods might split apart

• Prompting an LLM to segment content
• Using embeddings and clustering
• Using paragraph boundaries as a proxy (still common in practice)

Stage 2: Vector embedding

• Model quality – Better models = better representations
• Dimensions – More dimensions can mean richer context, but also more storage
• Garbage in, garbage out – Better chunks yield better vectors

Stage 3: Vector retrieval

• Similarity metric – Cosine is common, but not your only option
• k-NN tuning – How many chunks should you return?
• Re-ranking – Adjust scores post-retrieval
• Filtering – Metadata-based filters narrow the field

Stage 4: Response generation

• Prompt formatting – Structure the context clearly
• Explicit instructions – Tell the model what you want
• Source tracking – Keep receipts if citations matter
• Output clarity – Format for readability

Chunking in action: demo time

• Fixed-size chunking (200 characters): Retrieved equipment mentions, but missed policy context. Accurate, but not complete.
• Semantic chunking: Nailed it. Retrieved the full policy section with all relevant details. The response was rich and clear.

Best practices for chunking

1. Start with structure – Understand your document layout
2. Preserve meaning – Don't split mid-thought
3. Test chunk sizes – One size rarely fits all
4. Overlap chunks – Helps preserve context across boundaries
5. Tag everything – Metadata improves filtering and relevance
6. Use hierarchy – Nested chunking for structured content
7. Monitor & refine – Let real results guide improvements

Sample Code for Chunking

javascriptexport function chunkByDelimiter(text, delimiter) {
  return text.split(delimiter).filter(chunk => chunk.trim());
}

export function chunkByFixedSize(text, chunkSize, overlap) {
  const chunks = [];
  let i = 0;
  while (i < text.length) {
    chunks.push(text.slice(i, i + chunkSize));
    i += chunkSize - overlap;
  }
  return chunks;
}

export function recursiveChunk(text, maxSize, overlap) {
  const chunks = [];
  let start = 0;
  while (start < text.length) {
    let end = Math.min(start + maxSize, text.length);
    let breakPoint = text.lastIndexOf('.', end);
    if (breakPoint <= start) breakPoint = end;
    chunks.push(text.slice(start, breakPoint + 1));
    start = breakPoint - overlap;
  }
  return chunks;
}

export function semanticChunk(text) {
  const sentences = text.match(/[^.!?]+[.!?]+/g) || [text];
  const chunks = [];
  let current = '';
  for (const sentence of sentences) {
    if (current.length + sentence.length > 100) {
      chunks.push(current);
      current = sentence;
    } else {
      current += sentence;
    }
  }
  if (current) chunks.push(current);
  return chunks;
}

Conclusion

Additional resources

• MongoDB Atlas Vector Search Docs
• OpenAI Embeddings API
• LangChain RAG Guide
• Chunking Deep Dive – Pinecone