What Are Tokens?
If you’ve ever hit a context limit or been surprised by an API bill, you already know tokens matter. But what actually are they?
Tokens aren’t words and they aren’t characters. They’re chunks of text that a language model breaks your input into before processing it. Think of them as the atomic units that LLMs actually “read.” The word “tokenization” itself might get split into “token” + “ization” – two tokens. A short word like “the” is typically one token. A long technical term might be three or four.
The exact split depends on the model’s tokenizer – the algorithm it uses to chop up text. Different models use different tokenizers, which is why the same paragraph produces different token counts across GPT, Claude, and Gemini.
How to Count Tokens Online (Step by Step)
Counting tokens with this tool takes about ten seconds and never sends your text anywhere:
- Paste your text into the input box above – a prompt, a document, a code snippet, or an entire conversation.
- Pick your model from the settings dropdown. The default is a current GPT model, but you can switch to Claude, Gemini, Llama, Mistral, DeepSeek, or Grok.
- Read the token count. The tool shows characters, words, and estimated tokens the instant you stop typing – no button press required.
- Check the cost and context usage. Alongside the token count you get the estimated input and output cost plus the percentage of the model’s context window your text fills.
- Compare across models. The comparison table recalculates the same text for every supported model at once, so you can see which one tokenizes your content most cheaply before you commit to an API.
Because everything runs in JavaScript on your own machine, you can paste proprietary prompts, customer data, or unreleased code without it ever leaving the browser tab.
How Tokenization Actually Works
Most modern LLMs use a technique called Byte Pair Encoding (BPE) or a close variant. Here’s the gist:
- Start with individual characters (or bytes)
- Find the most frequently occurring pair of adjacent tokens in the training data
- Merge that pair into a single new token
- Repeat thousands of times until you’ve got a vocabulary of 50K-100K tokens
OpenAI’s models use a BPE variant through their tiktoken library. Google’s Gemini models use SentencePiece, which operates on raw text (including spaces) rather than pre-tokenized words. Anthropic’s Claude uses its own BPE-based tokenizer with a vocabulary optimized for code and multilingual text.
The practical difference? Claude tends to produce fewer tokens for the same text (roughly 3.5 characters per token) compared to GPT models (roughly 4 characters per token). That gap widens with code-heavy or multilingual content.
Why Token Counts Matter
Tokens affect three things you care about:
Cost. API pricing is per-token for both input and output. If you’re building an app that sends 10K tokens per request at $10/million input tokens, that’s $0.10 per request – and it adds up fast when you’re handling thousands of users.
Context window. Every model has a maximum number of tokens it can process in a single request. GPT-5.4 handles 256K tokens, Gemini 3 stretches to 2M, and Claude Opus 4.6 sits at 200K. If your prompt plus the expected response exceeds the context window, you’ll need to trim or chunk your input.
Response quality. Longer prompts don’t always mean better results. Models can lose focus in very long contexts (the “lost in the middle” problem). Keeping your prompts concise often improves output quality while cutting costs.
Character-to-Token Ratios by Model Family
| Provider | Models | Avg. Chars/Token |
|---|---|---|
| Anthropic | Claude Opus/Sonnet/Haiku | ~3.5 |
| OpenAI | GPT-5.4, GPT-4o | ~4.0 |
| Gemini 3, Gemini 2.5 | ~4.0 | |
| Meta | Llama 4 Maverick | ~3.8 |
| Mistral | Mistral Large 3 | ~3.8 |
| DeepSeek | DeepSeek V3 | ~3.5 |
| xAI | Grok 3 | ~4.0 |
These ratios are averages for English text. Code, non-Latin scripts, and text with lots of special characters will tokenize differently – usually producing more tokens per character.
Token Counts by Content Type
Not all text tokenizes equally. The type of content you feed into a model significantly affects how many tokens it consumes:
English prose tokenizes most efficiently – averaging close to the ratios in the table above. A typical 500-word blog post runs about 650-700 tokens on GPT models and 550-600 on Claude.
Source code is more expensive. Variable names, syntax characters, and indentation all eat tokens. A 100-line Python file might run 800-1,200 tokens depending on complexity. Languages with verbose syntax (Java, C++) tokenize higher than terse ones (Python, Go).
JSON and structured data falls in between. Repeated keys and nested braces add up quickly. A 1 KB JSON payload typically runs 300-500 tokens. If you’re passing structured data to an API, consider simplifying the schema to cut costs.
Non-Latin scripts – Chinese, Japanese, Korean, Arabic – tokenize much less efficiently. A single Chinese character might consume 2-3 tokens, making the effective cost per character 3-4x higher than English. This matters significantly for multilingual applications.
Markdown and HTML carry overhead from formatting syntax. Headers, links, list markers, and tags all consume tokens without adding semantic content. Strip formatting when the model doesn’t need it to understand your input.
Quick Token Reference
To give you a rough sense of scale for English text on GPT-class models:
| Content | Approximate Tokens |
|---|---|
| 1 English word | ~1.3 tokens |
| 1 sentence (15 words) | ~20 tokens |
| 1 paragraph (100 words) | ~130 tokens |
| 1 page of text (500 words) | ~650 tokens |
| Average email | ~200-400 tokens |
| README file | ~1,000-3,000 tokens |
| Typical API request + response | ~500-2,000 tokens |
Claude models will run 10-15% lower for the same content. Code and non-Latin text will run higher.
Tips for Reducing Token Usage
Want to keep your API costs down? Here are some practical strategies:
- Be specific in your prompts. Vague instructions force the model to guess, which means longer outputs and wasted tokens on both sides.
- Use system prompts wisely. A well-crafted system prompt can replace pages of per-request instructions.
- Trim unnecessary context. Don’t dump an entire document into the prompt if the model only needs two paragraphs.
- Pick the right model. You don’t always need the flagship. For simple tasks, a smaller model like GPT-4o Mini or Claude Haiku gives you 90% of the quality at 5% of the cost.
- Cache repeated content. If your app sends the same system prompt with every request, use prompt caching (available on both OpenAI and Anthropic APIs) to avoid paying for those tokens repeatedly.
- Chunk large documents. Instead of stuffing everything into one request, break documents into chunks and process them separately. Our Chunking Preview tool can help you plan this.
Common Token-Counting Mistakes to Avoid
Even experienced developers trip over the same handful of token assumptions. Watch for these:
- Counting words instead of tokens. A 1,000-word document is not 1,000 tokens – it’s closer to 1,300 on GPT-class models, and more if it contains code or punctuation. Always count, don’t guess.
- Forgetting the output side. Your prompt tokens are only half the bill. Models charge separately for the tokens they generate, and output tokens are usually priced higher than input. A short prompt with a long completion can cost more than the reverse.
- Ignoring the system prompt and chat history. In a chat application, every turn resends the system prompt and prior messages. Those tokens count on every request, which is why long conversations get expensive even when each new message is short.
- Assuming counts are identical across models. The same paragraph produces different token counts on GPT, Claude, and Gemini because each uses a different tokenizer. Never reuse a GPT token estimate to budget a Claude request – check both.
- Underestimating non-English text and code. Chinese, Japanese, Arabic, emoji, and dense source code all consume far more tokens per character than English prose. If your app is multilingual, measure with real sample data, not English placeholders.
- Trusting an estimate as an exact count. Character-based estimators (including this one) get you within roughly 5-10%. For hard limits – like guaranteeing a request fits a context window – verify with the provider’s own tokenizer before shipping.
How This Tool Estimates Tokens
This tool uses character-to-token ratios specific to each model family. It divides your text’s character count by the model’s average characters-per-token ratio. While this won’t match the exact output of each provider’s tokenizer (which would require running their specific BPE algorithm), it’s accurate enough for cost estimation and planning purposes.
For exact counts, you’d need to use each provider’s tokenization library directly – tiktoken for OpenAI, the Anthropic SDK’s built-in counter, or Google’s token counting API. But for quick estimates and cost comparisons across models, character-based approximation gets you within 5-10% of the real number.
We also offer dedicated token counters for individual providers: our Claude Token Counter, OpenAI Token Counter, Gemini Token Counter, and Llama Token Counter each focus on a single provider’s tokenization behavior and pricing tiers.