Code With Coffie
  • HOME
  • ABOUT US
  • PORTFOLIO
  • AI
    • AGENTIC AI
    • Generative AI
    • LangChain
    • LangGraph
    • LLM
    • MCP
    • RAG
  • TUTORIAL
    • SYSTEM DESIGN
    • MYSQL
      • DATETIME
    • DSA
      • LEETCODE
    • GIT
    • Docker
    • INTERVIEW
    • PROGRAMME
      • STAR PATTERN PROGRAMME
  • PYTHON
    • DJANGO
    • FLASK
    • FastAPI
    • Matplotlib
    • NumPy
    • Pandas
    • STREAMLIT
  • JAVASCRIPT
    • Vue.js
  • PHP
    • PHP OOPS
    • LARAVEL
    • WORDPRESS
  • NEXTERP
  • Home
  • Blog
  • RAG
  • Chunking & Document Splitting in RAG: Complete Guide
Chunking and Document Splitting

Chunking & Document Splitting in RAG: Complete Guide

Sep 21, 2026 by codewithhemu

When building a RAG (Retrieval-Augmented Generation) application, one of the most important steps is deciding how to split your documents before creating embeddings.

Large documents are usually not sent directly to an embedding model or an LLM. Instead, they are divided into smaller sections called chunks.

This process is known as document chunking or document splitting.

In simple words:

Chunking is the process of breaking a large document into smaller, meaningful pieces so that relevant information can be efficiently retrieved by an AI system.

Chunking may look like a simple preprocessing step, but it can have a major impact on the quality of a RAG application.

In this article, you will learn:

  • What document chunking is
  • Why chunking is needed in RAG
  • How document splitting works
  • Different chunking strategies
  • Fixed-size chunking
  • Character-based chunking
  • Token-based chunking
  • Sentence-based chunking
  • Recursive character splitting
  • Semantic chunking
  • Chunk overlap
  • How chunk size affects RAG
  • Common chunking mistakes
  • Best practices
  • A practical RAG chunking workflow

What is Chunking?

Chunking means dividing a large document into smaller pieces that can be processed and retrieved independently.

Suppose you have a 100-page PDF containing:

Python Basics
Python Functions
Python Classes
Python APIs
Python File Handling
Python Exceptions

Instead of treating the entire PDF as one piece of information, you can split it into smaller chunks:

Document
   ↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4
Chunk 5
...

Each chunk can then be converted into an embedding and stored in a Vector Database.


Why is Chunking Needed in RAG?

Chunking is important because large documents can contain a huge amount of information.

Suppose a user asks:

How does Python exception handling work?

Your document may contain 100 pages.

You don’t want to send all 100 pages to the LLM.

Instead, the RAG system should retrieve only the relevant section:

Python Documentation
       ↓
Chunking
       ↓
Exception Handling Chunk
       ↓
Embedding
       ↓
Vector Database
       ↓
User Question
       ↓
Relevant Chunk
       ↓
LLM
       ↓
Answer

This makes retrieval more focused.


Chunking in the RAG Pipeline

Chunking is part of the document ingestion process.

A typical RAG pipeline looks like this:

Documents
    ↓
Document Loading
    ↓
Text Extraction
    ↓
Document Splitting
    ↓
Chunks
    ↓
Embeddings
    ↓
Vector Database
    ↓
User Query
    ↓
Retrieval
    ↓
LLM
    ↓
Answer

So chunking happens before embeddings and retrieval.


What is a Document Chunk?

A document chunk is a smaller section of a larger document.

For example:

Original Document:

Python is a high-level programming language.
It supports functions, classes, modules,
exception handling, file processing, and APIs.

After splitting:

Chunk 1:
Python is a high-level programming language.

Chunk 2:
It supports functions, classes, modules,
exception handling, file processing, and APIs.

Each chunk can have its own embedding.


Chunking Example

Suppose you have this document:

Python is a programming language.

Python supports functions.

Python supports object-oriented programming.

Python can be used to build APIs.

Python is widely used in artificial intelligence.

You could split it into:

Chunk 1:
Python is a programming language.

Chunk 2:
Python supports functions.

Chunk 3:
Python supports object-oriented programming.

Chunk 4:
Python can be used to build APIs.

Chunk 5:
Python is widely used in artificial intelligence.

Each chunk can then be embedded separately.


Chunking and Embeddings

Chunking and embeddings are closely connected.

The process looks like this:

Large Document
      ↓
Chunking
      ↓
Small Chunks
      ↓
Embedding Model
      ↓
Vectors
      ↓
Vector Database

For example:

Chunk:
Python can be used to build REST APIs.

↓

Embedding Model

↓

[0.21, 0.73, 0.18, 0.52, ...]

The vector is then stored in the Vector Database.


What Happens If You Don’t Chunk Documents?

Suppose you have a very large document and create only one embedding for the entire document.

The embedding represents the document at a broad level.

Now imagine the user asks:

How does authentication work?

The system may know that the document is generally about software development, but it may not retrieve the exact section about authentication effectively.

Chunking allows different parts of the document to have their own embeddings.

Large Document
      ↓
Authentication Chunk
API Chunk
Database Chunk
Security Chunk
Deployment Chunk

This makes targeted retrieval possible.


What is Chunk Size?

Chunk size refers to how much content is included in each chunk.

For example:

Chunk Size = 500 tokens

means each chunk contains approximately 500 tokens.

Or:

Chunk Size = 1,000 characters

means each chunk contains approximately 1,000 characters.

The ideal chunk size depends on the document type and retrieval task.

There is no universal chunk size that works perfectly for every RAG application.


Small Chunks vs Large Chunks

Chunk size creates a trade-off.

Small Chunks

Example:

100–200 tokens

Advantages:

  • More focused retrieval
  • Less irrelevant information
  • Easier to retrieve specific facts

Disadvantages:

  • Important context may be missing
  • More chunks are created
  • Related information may be separated

Large Chunks

Example:

1,000–2,000 tokens

Advantages:

  • More context
  • Related information stays together
  • Better for some complex explanations

Disadvantages:

  • More irrelevant information may be retrieved
  • More tokens may be sent to the LLM
  • Retrieval can become less precise

The goal is to find a useful balance between precision and context.


What is Chunk Overlap?

Chunk overlap means allowing consecutive chunks to share some content.

For example, suppose you have:

Chunk Size = 500 tokens
Overlap = 100 tokens

Then the chunks might look conceptually like:

Chunk 1:
Tokens 1 → 500

Chunk 2:
Tokens 401 → 900

Chunk 3:
Tokens 801 → 1300

The 100-token overlap helps preserve context between chunks.


Why is Chunk Overlap Useful?

Imagine a sentence or explanation starts near the end of one chunk and continues into the next chunk.

Without overlap:

Chunk 1:
The authentication process starts by validating

Chunk 2:
the user's access token...

The meaning can become fragmented.

With overlap:

Chunk 1:
The authentication process starts by validating
the user's access token...

Chunk 2:
validating the user's access token and checking
its expiration...

The repeated context can help retrieval preserve continuity.


How Much Chunk Overlap Should You Use?

There is no universal number.

Common approaches might use a modest overlap such as:

10%–20%

of the chunk size, but the appropriate value depends on the content and retrieval task.

For example:

Chunk Size:
500 tokens

Overlap:
50–100 tokens

can be a reasonable starting point for experimentation.

Always evaluate the results using your actual documents and queries.


Different Document Chunking Strategies

There are several ways to split documents.

Common strategies include:

  1. Fixed-size chunking
  2. Character-based chunking
  3. Token-based chunking
  4. Sentence-based chunking
  5. Recursive character splitting
  6. Paragraph-based chunking
  7. Markdown-aware chunking
  8. HTML-aware chunking
  9. Semantic chunking

Let’s understand them one by one.


1. Fixed-Size Chunking

Fixed-size chunking divides the document into pieces of a predefined size.

For example:

Chunk Size = 500 characters

The document becomes:

Chunk 1 → 0–500
Chunk 2 → 500–1000
Chunk 3 → 1000–1500

Advantages

  • Very simple
  • Easy to implement
  • Predictable chunk sizes

Disadvantages

  • Can split sentences
  • Can split paragraphs
  • May break important context

Fixed-size chunking is useful for simple experiments but may not always be ideal for production RAG systems.


2. Character-Based Chunking

Character-based splitting divides content according to character count.

For example:

Chunk Size = 1,000 characters

The document is split every 1,000 characters.

Example:

Document
 ↓
1,000 characters
 ↓
1,000 characters
 ↓
1,000 characters

This approach is easy but does not necessarily understand the structure or meaning of the document.


3. Token-Based Chunking

Token-based chunking splits text based on tokens rather than characters.

For example:

Chunk Size = 500 tokens

This can be useful because LLMs process text in tokens.

Conceptually:

Text
 ↓
Tokenizer
 ↓
Tokens
 ↓
Chunk every N tokens

Token-based chunking provides more predictable control over the amount of text sent through model APIs.


What is a Token?

A token is a unit of text processed by language models.

A token may represent:

  • A complete word
  • Part of a word
  • Punctuation
  • Other pieces of text

For example, a sentence may be broken into multiple tokens.

The exact tokenization depends on the tokenizer and language.


4. Sentence-Based Chunking

Sentence-based chunking splits documents according to sentence boundaries.

For example:

Sentence 1.
Sentence 2.
Sentence 3.
Sentence 4.

can become:

Chunk 1:
Sentence 1.
Sentence 2.

Chunk 2:
Sentence 3.
Sentence 4.

This helps avoid cutting sentences in the middle.


5. Paragraph-Based Chunking

Paragraph-based chunking uses paragraph boundaries.

For example:

Paragraph 1

Paragraph 2

Paragraph 3

Paragraph 4

can become:

Chunk 1:
Paragraph 1
Paragraph 2

Chunk 2:
Paragraph 3
Paragraph 4

This can work well for documents where paragraphs represent meaningful units of information.


6. Recursive Character Splitting

Recursive character splitting is a popular strategy in RAG frameworks.

Instead of blindly cutting text at one fixed character count, the splitter tries multiple separators.

For example:

Paragraph
    ↓
Sentence
    ↓
Word
    ↓
Character

Conceptually:

Large Text
   ↓
Split by paragraphs
   ↓
If still too large
   ↓
Split by sentences
   ↓
If still too large
   ↓
Split by words

This approach tries to preserve the natural structure of the document while keeping chunks within the desired size.


Example of Recursive Splitting

Suppose your document is:

Python is a programming language.

It is widely used for web development,
automation, data analysis, and AI.

Python supports object-oriented programming.

A recursive splitter might first try paragraph boundaries.

If a paragraph is still too large, it can split it further using sentence or other separators.

The result can preserve more meaningful context than simple fixed-size splitting.


7. Markdown-Aware Chunking

If your documentation is written in Markdown, you can use its structure.

For example:

# Python

## Functions

Functions allow you to organize reusable code.

## Classes

Classes are used for object-oriented programming.

Instead of treating the entire document as plain text, you can preserve the heading structure.

For example:

Chunk:
Heading: Python
Section: Functions
Content: Functions allow you to organize reusable code.

This can improve retrieval quality because the chunk retains useful context.


8. HTML-Aware Chunking

Web pages often contain:

<h1>Python Tutorial</h1>

<h2>Functions</h2>

<p>Functions allow reusable code...</p>

An HTML-aware splitter can use the document structure to create meaningful chunks.

For example:

Page
 ↓
Heading
 ↓
Section
 ↓
Paragraph

This is often better than simply splitting the raw HTML string every N characters.


9. Semantic Chunking

Semantic chunking attempts to divide a document based on meaning rather than only character or token counts.

For example, consider:

Python is a programming language.

Functions allow developers to create reusable blocks of code.

Classes provide a structure for object-oriented programming.

Python can also be used to build APIs.

A semantic splitter attempts to keep related ideas together.

Conceptually:

Programming Introduction
        ↓
Chunk 1

Functions
        ↓
Chunk 2

Object-Oriented Programming
        ↓
Chunk 3

API Development
        ↓
Chunk 4

Semantic chunking can be more sophisticated, but it may require additional processing and evaluation.


Structure-Aware Chunking

For structured documents, the document’s organization can be more important than a fixed chunk size.

For example:

Book
 ├── Chapter 1
 │    ├── Section 1
 │    └── Section 2
 │
 ├── Chapter 2
 │    ├── Section 1
 │    └── Section 2

A good chunking strategy can preserve:

  • Chapter
  • Section
  • Heading
  • Paragraph
  • List
  • Table
  • Metadata

This can make retrieved chunks easier for an LLM to understand.


Chunking Tables

Tables can be especially difficult.

Consider:

ProductPriceStock
Laptop₹50,00010
Monitor₹15,00020

Simply converting this table into plain text may lose relationships between headers and values.

A better approach is to preserve the table structure where possible.

For example:

Product: Laptop
Price: ₹50,000
Stock: 10

This can make the information easier to retrieve.


Chunking PDFs

PDFs require special attention because their internal structure may not match their visual layout.

A PDF can contain:

  • Text
  • Headings
  • Tables
  • Images
  • Footnotes
  • Headers
  • Page numbers
  • Multiple columns

A good PDF pipeline may look like:

PDF
 ↓
PDF Parser
 ↓
Text + Structure
 ↓
Clean Content
 ↓
Chunking
 ↓
Embeddings

Simply extracting raw text and splitting it blindly can produce poor chunks.


Chunking Code Documentation

Code should generally not be chunked exactly like normal prose.

Suppose you have:

def calculate_total(price, quantity):
    return price * quantity

Splitting the function in the middle would make the retrieved code incomplete.

For code documentation, try to preserve meaningful units such as:

  • Functions
  • Classes
  • Methods
  • Modules
  • Code blocks
  • Documentation comments

For example:

Function:
calculate_total()

Description:
Calculates total price based on price and quantity.

Code:
...

Chunking FAQs

FAQs are usually naturally structured.

For example:

Question:
How do I reset my password?

Answer:
Click "Forgot Password" and follow the instructions.

It can be useful to keep each question and answer together.

Bad chunk:

Question in Chunk 1
Answer in Chunk 2

Better:

Chunk:
Question + Answer

This preserves the relationship between the question and its answer.


Chunking Product Documentation

Product documentation may contain:

Installation
Configuration
Authentication
API Reference
Troubleshooting
Examples

Each section can become a meaningful retrieval unit.

For example:

Product Documentation
       ↓
Authentication
       ↓
API Keys
       ↓
Token Authentication
       ↓
Error Handling

This structure can help the retrieval system return more relevant context.


How Chunk Size Affects Retrieval

Imagine a user asks:

How do I configure API authentication?

Very Small Chunks

You might retrieve:

"API authentication uses tokens..."

but miss the configuration instructions.

Very Large Chunks

You might retrieve:

Authentication
+
API Reference
+
Error Handling
+
Examples
+
Deployment

Now the LLM receives lots of irrelevant information.

Balanced Chunks

A useful chunk might contain:

API Authentication

API keys are generated from the developer dashboard.

Add the API key to the Authorization header...

This provides both the relevant information and enough context.


Chunking and Retrieval Quality

A RAG system can be represented as:

Good Documents
      ↓
Good Chunking
      ↓
Good Embeddings
      ↓
Good Retrieval
      ↓
Good Context
      ↓
Better Answer

If chunking is poor:

Poor Chunking
      ↓
Poor Retrieval
      ↓
Poor Context
      ↓
Poor Answer

This is why chunking deserves careful attention.


Common Chunking Mistakes

Mistake 1: Using One Chunk Size for Everything

A 500-token chunk may work for one document type but not another.

For example:

FAQ
Technical Documentation
Legal Document
Research Paper
Code

may all require different strategies.


Mistake 2: Ignoring Document Structure

Splitting at arbitrary positions can destroy headings and relationships.


Mistake 3: No Overlap When Context Requires It

Important information may be split between chunks.


Mistake 4: Excessive Overlap

Too much overlap creates duplicate information and increases storage and retrieval volume.


Mistake 5: Chunks That Are Too Large

Large chunks can introduce irrelevant context.


Mistake 6: Chunks That Are Too Small

Tiny chunks may lose the context needed to understand the information.


Mistake 7: Ignoring Metadata

A chunk should often retain useful metadata such as:

Document
Page
Section
Heading
Category
URL
Author
Date

Best Practices for Document Chunking

1. Preserve Meaning

Try to keep related information together.


2. Respect Document Structure

Use:

  • Headings
  • Paragraphs
  • Sections
  • Lists
  • Tables
  • Code blocks

when possible.


3. Start with a Reasonable Chunk Size

There is no universal value.

Start with a reasonable range and test it against real queries.


4. Use Chunk Overlap Carefully

Overlap can help preserve context but too much overlap can increase duplication.


5. Store Metadata

Metadata can help with filtering and source attribution.


6. Evaluate Retrieval

Don’t judge chunking only by how the chunks look.

Test actual user questions.

For example:

Question
   ↓
Retrieve Top 5 Chunks
   ↓
Are the correct chunks present?

Measure retrieval quality before optimizing generation.


7. Use Different Strategies for Different Data

For example:

PDF → Structure-aware chunking
FAQ → Question-answer chunks
Code → Function/class chunks
Markdown → Heading-aware chunks
Web pages → HTML-aware chunks

Chunking in a Production RAG System

A production system may look like:

                DATA SOURCES
                     ↓
              Document Loaders
                     ↓
              Content Cleaning
                     ↓
              Structure Detection
                     ↓
                 Chunking
                     ↓
                Metadata
                     ↓
              Embedding Model
                     ↓
              Vector Database
                     ↓
                Retrieval
                     ↓
                 Reranking
                     ↓
                  Context
                     ↓
                    LLM
                     ↓
               Final Answer

Chunking is only one part of the complete retrieval pipeline, but it can strongly influence downstream performance.


Chunking with Python

A simple conceptual Python implementation could look like:

text = """
Python is a programming language.
Python supports functions and classes.
Python is widely used in AI development.
"""

chunk_size = 50

chunks = [
    text[i:i + chunk_size]
    for i in range(0, len(text), chunk_size)
]

for chunk in chunks:
    print(chunk)

This is a basic fixed-size approach.

In real RAG applications, you would generally use more sophisticated document splitting logic that respects sentence, paragraph, heading, or token boundaries.


Chunking with a RAG Framework

Frameworks such as LangChain and LlamaIndex provide document splitting utilities that can help developers implement different chunking strategies.

A conceptual pipeline is:

Document Loader
      ↓
Text Splitter
      ↓
Chunks
      ↓
Embedding Model
      ↓
Vector Store

The exact implementation depends on the framework and the type of documents you are processing.


How to Choose the Right Chunking Strategy

Ask these questions:

1. What type of data do I have?

PDF?
Website?
Markdown?
FAQ?
Code?
Research paper?

2. Does the document have a clear structure?

If yes, use that structure.

3. Do users ask specific questions?

If yes, smaller and focused chunks may work well.

4. Do answers require multiple paragraphs of context?

If yes, larger chunks or overlap may be useful.

5. Are retrieval results accurate?

Evaluate the system using real questions.


A Practical Starting Point

For a beginner RAG project, you can start with:

Chunk Size:
~500–1,000 tokens

Overlap:
~50–150 tokens

These are starting points, not universal rules.

Then test different configurations.

For example:

Experiment 1:
500 tokens / 50 overlap

Experiment 2:
800 tokens / 100 overlap

Experiment 3:
1,000 tokens / 150 overlap

Compare retrieval quality using a representative set of questions.


Chunking vs Document Splitting

You may hear both terms:

  • Chunking
  • Document Splitting
  • Text Splitting

In many RAG discussions, these terms refer to the process of breaking a large document into smaller pieces.

The terminology may differ between libraries and implementations, but the basic goal is the same:

Create retrieval-friendly pieces of information from larger documents.


Frequently Asked Questions

What is chunking in RAG?

Chunking is the process of splitting large documents into smaller pieces before generating embeddings and storing them for retrieval.

Why is document splitting important in RAG?

Good document splitting helps the retrieval system find focused and relevant information instead of retrieving an entire large document.

What is a good chunk size for RAG?

There is no single ideal chunk size. A practical starting point is often a few hundred tokens, followed by evaluation and tuning based on the document type and user queries.

What is chunk overlap?

Chunk overlap means repeating some content between consecutive chunks to help preserve context across chunk boundaries.

What happens if chunks are too small?

Very small chunks may lose important context and make retrieved information incomplete.

What happens if chunks are too large?

Very large chunks may contain irrelevant information and increase the amount of context sent to the LLM.

What is semantic chunking?

Semantic chunking attempts to divide text according to meaningful changes in topic or context rather than relying only on fixed character or token lengths.

Should every document use the same chunking strategy?

Not necessarily. Different document types may benefit from different splitting strategies.

Is chunking required for RAG?

For most document-based RAG systems, some form of document segmentation is highly useful because retrieval works better when information is stored in manageable, meaningful units. The exact implementation can vary.

Does chunking affect RAG accuracy?

Yes. Poor chunking can cause relevant information to be missed or retrieved with insufficient context, while well-designed chunking can improve retrieval quality.


Conclusion

Chunking and Document Splitting are fundamental steps in building an effective RAG application.

The basic process is:

Large Document
      ↓
Document Splitting
      ↓
Meaningful Chunks
      ↓
Embeddings
      ↓
Vector Database
      ↓
Semantic Retrieval
      ↓
Relevant Context
      ↓
LLM
      ↓
Answer

The goal is not simply to create small pieces of text.

The real goal is to create meaningful, retrieval-friendly chunks that contain enough context to answer user questions.

A good chunking strategy should consider:

  • Document structure
  • Chunk size
  • Chunk overlap
  • Metadata
  • Content type
  • Retrieval quality
  • User questions

For example, a PDF, FAQ, Markdown document, website, research paper, and source code may all require different chunking approaches.

  • Share:
Previous Article Introduction to Agentic AI: How It Works, Types, Architecture & Applications
Next Article What are Embeddings in AI? Complete Guide with Examples
No comments yet! You be the first to comment.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

category

  • AGENTIC AI (2)
  • DATETIME (6)
  • DJANGO (1)
  • Docker (1)
  • DSA (22)
  • DSA PRACTICE (4)
  • GIT (1)
  • INTERVIEW (4)
  • JAVASCRIPT (69)
  • LangChain (3)
  • LARAVEL (41)
  • LeetCode (4)
  • MYSQL (45)
  • PHP (21)
  • PHP OOPS (16)
  • PROGRAMME (1)
  • PYTHON (11)
  • RAG (5)
  • REACT JS (6)
  • STAR PATTERN PROGRAMME (7)
  • SYSTEM DESIGN (2)
  • Uncategorized (21)
  • Vue.js (5)
  • WORDPRESS (15)

Archives

  • October 2026
  • September 2026
  • July 2026
  • June 2026
  • May 2026
  • March 2026
  • October 2025
  • September 2025
  • August 2025
  • July 2025
  • June 2025
  • May 2025
  • April 2025
  • March 2025
  • February 2025
  • January 2025
  • January 2023

Tags

Certificates Education Instructor Languages School Member

Building reliable software solutions for modern businesses. Sharing practical tutorials and real-world project insights to help developers grow with confidence.

GET HELP

  • Home
  • Portfolio
  • Privacy Policy
  • Terms & Conditions
  • Disclaimer
  • Contact Us

PROGRAMS

  • Software Development
  • Performance Optimization
  • System Architecture
  • Project Consultation
  • Technical Mentorship

CONTACT US

  • Netaji Subhash Place (NSP) Delhi
  • Tel: + (91) 8287315524
  • Email: contact@codewithcoffie.com

Copyright © 2026 LearnPress LMS | Powered by LearnPress LMS