What is text chunking?

Text chunking splits a large document into smaller pieces — chunks — that fit inside an LLM's context window or an embedding model's input limit. It's a foundational step in RAG (retrieval-augmented generation): each chunk is embedded, stored in a vector database, and retrieved when a user asks a question. This tool lets you chunk text online and preview the result before you commit it to a pipeline.

What chunk size should I use for RAG?

It depends on your embedding model and content. A safe default is 512 tokens with 50 tokens of overlap. Use 128–256 tokens when you need precise retrieval over short facts, and 512–1024 tokens when answers need more surrounding context. Most embedding models are tuned for the 256–512 token range, so going far beyond that can hurt embedding quality.

What is chunk overlap and how much do I need?

Overlap makes each chunk repeat the last N tokens of the previous chunk, so context that sits on a boundary isn't lost. A common starting point is 10–20% of the chunk size — for 512-token chunks, that's 50–100 tokens. More overlap improves boundary recall but increases storage and embedding cost. This tool highlights the overlapping region of each chunk so you can see exactly what's duplicated.

Token-based vs character-based chunking: what's the difference?

Character-based splitting counts raw characters; token-based splitting counts the tokens your model actually reads and bills for. They diverge because one token is roughly 3.5–4 English characters (fewer for code, punctuation, and non-Latin scripts). If you size chunks by characters but your model measures tokens, a chunk that looks small can quietly exceed a 512-token limit and get truncated. Sizing by tokens keeps chunks aligned with the model's real budget.

How does this tool count tokens?

It estimates tokens with a fast character heuristic of roughly 3.8 characters per token, which is close to typical English prose for GPT and Claude tokenizers. It's an approximation, not an exact byte-pair tokenizer, so treat the counts as a reliable planning guide rather than an exact bill. For an exact count, pair it with a dedicated token counter.

How do I chunk code, Markdown, or tables?

Structured content breaks the assumptions behind sentence and paragraph splitting. For code, prefer paragraph-based splitting — which respects blank-line boundaries — or split on function and class boundaries in your own pipeline. For Markdown, keep headings attached to the section they introduce and avoid cutting through tables or fenced code blocks. Token-based splitting with generous overlap is the safest default when structure is irregular.

What is semantic chunking?

Semantic chunking groups sentences by meaning rather than by a fixed size: it embeds sentences, measures similarity between neighbours, and starts a new chunk when the topic shifts. It can improve retrieval on dense documents but is slower and needs an embedding model at chunk time. Fixed-size and recursive splitting remain the pragmatic default for most pipelines; reach for semantic chunking when retrieval quality plateaus.

Which chunking method should I use?

Token-based splitting gives the most predictable sizes and is the best general default. Sentence-based splitting preserves natural boundaries and suits narrative or conversational text. Paragraph-based splitting keeps structured documents intact. This tool offers all three, so you can preview each against your own text and pick the one that produces the cleanest chunks.

Text Chunking Tool

Why Chunking Matters for RAG

If you’re building anything with retrieval-augmented generation, chunking is where most pipelines quietly succeed or fail. The idea is simple: you’ve got documents that are too long to feed directly into an embedding model or an LLM’s context window, so you need to break them into smaller pieces. Those pieces get embedded, stored in a vector database, and retrieved when a user asks a question.

The tricky part? How you split the text has a massive impact on retrieval quality. Cut in the wrong place and you’ll lose context that ties two ideas together. Make chunks too small and the embeddings won’t capture enough meaning. Make them too big and you’ll blow past your embedding model’s token limit or dilute the semantic signal with irrelevant text.

How to Use This Chunking Tool

You don’t need to stand up a pipeline to see how your text will chunk — everything runs live in your browser:

Paste your text into the input box. Chunking runs automatically as you type, and re-runs whenever you change a setting, so there’s no wait.
Pick a method — token-based, sentence-based, or paragraph-based — in the settings panel. Each one splits the same text differently; switch between them to compare.
Set your chunk size and overlap in tokens. The defaults (512 tokens, 50 overlap) are a sensible starting point for most RAG setups.
Read the preview. Every chunk is numbered and labelled with its estimated token count, and the region it overlaps with the previous chunk is highlighted so you can see exactly what’s duplicated.
Check the stats bar for the chunk count, average tokens per chunk, total tokens, and the effective overlap.
Export when it looks right. Hit Copy Chunks as JSON to grab the whole array, ready to drop into your embedding or vector-database code.

Because nothing leaves your browser, you can safely preview proprietary or sensitive documents — the text is never uploaded to a server.

Chunking Strategies Explained

There are a few common approaches, and each has its sweet spot.

Fixed-Size (Token-Based) Chunking

This is the simplest method. You pick a target size — say 512 tokens — and split the text into pieces of roughly that length, breaking at word boundaries so you don’t slice words in half. It’s predictable and easy to reason about, which is why it’s the default in most tutorials.

The downside is that it doesn’t care about meaning. A chunk might end mid-paragraph or even mid-sentence. For many use cases that’s perfectly fine, especially when you add overlap. But for documents where logical structure matters — legal contracts, technical specs, research papers — you might want something smarter.

Sentence-Based Chunking

Instead of counting tokens, this approach splits on sentence boundaries first, then groups sentences together until you hit your target chunk size. The result is chunks that always end at a natural stopping point.

This works well for narrative text, blog posts, documentation, and anything where sentences carry complete thoughts. It won’t help much with bullet-point-heavy content or code, where “sentences” aren’t really a thing.

Paragraph-Based Chunking

Paragraph splitting uses double newlines as boundaries. It’s a natural fit for well-structured documents where each paragraph covers a distinct topic. You group paragraphs until you reach the target size, keeping logical sections intact.

The catch is that paragraph lengths vary wildly. Some documents have one-line paragraphs; others have 500-word walls of text. You’ll often end up with uneven chunk sizes, which can affect retrieval consistency.

Recursive Character Splitting

This is what LangChain popularized, and it’s probably the most practical approach for production systems. The idea: try to split on the largest meaningful boundary first (double newlines), then fall back to single newlines, then sentences, then words. You recurse through these separators until each chunk fits within your target size.

It’s a “best of both worlds” approach — you preserve structural boundaries when possible and only break at smaller units when you have to.

Semantic Chunking

The newest approach doesn’t count anything — it follows meaning. Semantic chunking embeds each sentence, measures how similar neighbouring sentences are, and starts a fresh chunk wherever the topic shifts. The result is chunks that map to coherent ideas rather than arbitrary size limits.

It shines on dense material like research papers or knowledge bases, where a fixed window might merge two unrelated ideas or split one across a boundary. The trade-offs are real, though: you need an embedding model running at chunk time, it’s noticeably slower, and chunk sizes become unpredictable. Most teams start with recursive or token-based splitting and only move to semantic chunking when retrieval quality plateaus. This previewer focuses on the fast, deterministic token, sentence, and paragraph methods so you get instant feedback as you type.

Token-Based vs Character-Based Chunking

A lot of chunking bugs come down to a single confusion: characters are not tokens. A character is one letter or symbol. A token is the unit your model actually reads and bills for — usually a short run of characters produced by a byte-pair tokenizer. In everyday English, one token averages roughly 3.5–4 characters, but that ratio collapses for code, JSON, punctuation-heavy text, and non-Latin scripts, where a single character can cost a whole token.

Why does it matter? Because your embedding model and your LLM enforce limits in tokens, not characters. If you size chunks by character count — say “2,000 characters per chunk” — you’re only guessing at the token count, and a dense chunk can silently exceed a 512-token embedding limit and get truncated. Truncated chunks lose their tail, which is exactly where a token-based splitter would have started the next chunk.

Sizing by tokens keeps your chunks aligned with the budget the model really cares about. This tool works in tokens for that reason: it estimates the token count of every chunk (at roughly 3.8 characters per token, close to typical English) and shows it inline, so a chunk that reports “498 tokens” will comfortably fit a 512-token model while one that reports “640 tokens” won’t. When precision truly matters — billing, hard context limits — confirm the final count with a dedicated tokenizer, but for planning and previewing, token-aware sizing beats character counting every time.

The Role of Overlap

Overlap is the secret weapon that makes chunking actually work in practice. When you set an overlap of, say, 50 tokens, each chunk repeats the last 50 tokens of the previous chunk at its start. This creates redundancy at chunk boundaries.

Why does this help? Consider a paragraph that says: “The medication should be taken with food. Failure to do so may cause nausea.” If your chunk boundary falls between those two sentences and there’s no overlap, a query about medication side effects might only retrieve the second chunk — which says “Failure to do so may cause nausea” without the critical context of what should be taken with food.

With overlap, both chunks contain the full context around that boundary. The retrieval system has a much better shot at finding the relevant information regardless of where the split happened.

A good starting point is 10–20% of your chunk size. So for 512-token chunks, try 50–100 tokens of overlap. More overlap means better boundary coverage but also more storage and slightly higher embedding costs.

How Chunk Size Affects Retrieval Quality

There’s a real tension between precision and context when choosing chunk size:

Smaller chunks (128–256 tokens) give you more precise retrieval. Each chunk covers a narrower topic, so when a query matches, it’s more likely to be genuinely relevant. But you lose surrounding context, and the LLM has to piece together information from multiple small fragments.

Larger chunks (512–1024 tokens) carry more context per retrieval hit. The LLM gets a fuller picture from each chunk, which often leads to better-grounded answers. But retrieval precision drops — a large chunk might match a query because of one sentence while the rest is irrelevant noise.

The sweet spot depends on your data and your embedding model. Most embedding models (like OpenAI’s text-embedding-3-small or Cohere’s embed-v4) perform best with inputs in the 256–512 token range. That’s what they were trained on, and going significantly larger can degrade embedding quality.

Chunk Size Cheat Sheet by Content Type

There’s no universal chunk size, but these starting points cover most real-world content. Treat them as defaults to tune, not laws:

Content type	Method	Chunk size	Overlap	Why
Chat logs & support tickets	Sentence	256–384 tokens	~10%	Short, self-contained turns; precise retrieval matters more than context.
Documentation & articles	Recursive / token	512 tokens	~15%	Balanced default; preserves headings and paragraphs where possible.
Legal & contracts	Paragraph	512–768 tokens	~20%	Clauses depend on surrounding context; higher overlap protects boundaries.
Research papers	Semantic / paragraph	512–1024 tokens	~15%	Dense ideas benefit from larger, meaning-aligned windows.
Code & config	Paragraph	300–512 tokens	~10%	Split on blank lines or function boundaries; never mid-statement.
Transcripts & subtitles	Sentence	384–512 tokens	~15%	Spoken language rambles; sentence boundaries keep thoughts intact.
Product catalogs & FAQs	Token	128–256 tokens	0–10%	Each entry is atomic; small chunks maximise retrieval precision.

Start from the row that matches your data, preview it here against a representative sample, then adjust size and overlap until the chunks read cleanly and stay under your embedding model’s token limit.

Common Chunking Mistakes to Avoid

Even experienced teams trip over the same handful of chunking mistakes. Watch for these:

Zero overlap. Splitting with no overlap is the fastest way to strand context on a boundary. Unless your units are truly independent (catalog entries, FAQ pairs), keep at least 10% overlap.
Sizing by characters, not tokens. As covered above, a character budget is only a guess at the token budget. Chunks that look fine can exceed the model’s limit and get truncated.
Chunks larger than the embedding window. If a chunk exceeds the model’s max input, the tail is silently dropped before it’s ever embedded. Always keep a margin below the hard limit.
Splitting structured content naively. Cutting through a Markdown table, a fenced code block, or a JSON object produces chunks that are meaningless in isolation. Respect structural boundaries.
One size for every document type. A 512-token window that’s perfect for articles is often wrong for chat logs or code. Match the strategy to the content.
Dropping metadata. A chunk with no source, section, or title is hard to rank and impossible to cite. Attach metadata before you embed.
Never previewing. Shipping chunk parameters straight to production without looking at the output is how silent retrieval failures happen. Preview first — that’s what this tool is for.

Tips for Choosing Chunk Parameters

Here are some practical guidelines that’ve worked well in production RAG systems:

Start with 512 tokens and 50 token overlap. It’s a solid default for most document types.
Match your chunk size to your embedding model’s sweet spot. Check the model’s documentation for recommended input lengths.
Use sentence-based splitting for conversational or narrative content and token-based splitting for technical documents with mixed formatting.
Test with real queries. The best chunk size is the one that surfaces the right context for the questions your users actually ask. There’s no universal answer.
Don’t forget metadata. Attaching source info, section headers, or page numbers to each chunk makes retrieval results far more useful downstream.
Preview before you commit. That’s what this tool is for — paste your text, tweak the parameters, and see exactly how your chunks look before you push anything to your vector database.