---
title: "Text Chunker: Split by Tokens or Characters | Prompt Pebble"
description: "Split long text into chunks by tokens, characters or words, with overlap and paragraph or sentence breaks, to fit LLM context windows and embedding limits."
url: https://promptpebble.click/text-chunker/
---

Tokens · characters · words

# Text chunker

Split a long document into pieces that fit a model's context window or an embedding limit, measured in exact tokens, characters or words, with optional overlap so nothing is lost at the seams.

## How the chunker splits text

Long documents rarely fit in one prompt, and search over embeddings works best with passages of a steady size. The chunker cuts where a reader would pause: first between paragraphs, then between sentences, then between words, going finer only when one paragraph or sentence is bigger than the chunk size. With Anywhere it cuts at the exact limit, which gives equal sizes but can split a word.

Overlap repeats the end of each chunk at the start of the next, so a sentence that straddles a cut appears whole in at least one chunk. It is taken in whole pieces at the boundary you chose and highlighted in each chunk. Ten to twenty percent of the chunk size is a common place to start.

### Tokens, characters or words

Token sizes are exact for OpenAI's o200k_base and cl100k_base encodings, which load the first time you use them. Characters are Unicode code points, so an emoji is never cut in half. For other models, measure in characters at about four per token of English and leave a margin. Part labels are added on top of the chunk size.

## Tips

- For pasting into a chat, add a label such as [Part {n} of {total}] and ask the assistant to wait for the last part.
- For search and embeddings, a few hundred tokens per chunk with some overlap is a common start.
- The JSONL download records where each chunk sits in the original text.

## Questions

**What chunk size should I use?**

Work back from your limit: take the model's context window, subtract the tokens for your instructions and the reply, and split what is left.

**Why are some chunks smaller than the limit?**

The chunker stops at a paragraph, sentence or word break rather than cutting mid-thought. Choose Anywhere if every chunk must be filled to the exact size.

**Does the overlap count towards the chunk size?**

Yes. Each chunk, including the repeated text at its start, stays within the size you set. Part labels are the exception and are added on top.

**Can it split PDFs or Word files?**

Not directly. Copy the text out of the document and paste it in, or save it as plain text first; the chunker opens .txt and .md files.