Token Counter for GPT-5, Claude 4 & Gemini 3: Cost & Context Estimator

What is a token counter and why it matters

Models don’t read the way you and I do. They don’t see words. Before a single GPT, Claude or Gemini prompt gets processed, a tokenizer chops the text into tokens. A token might be a short word, a fragment of a longer one, a punctuation mark, a clump of digits, a newline, sometimes a lone Unicode character if you’re working in a less common script. Everything after that, reading, generating, billing, happens per token. So if you don’t know roughly how many tokens your prompt and its reply will eat, you’re basically guessing at the cost. Worse, you’re risking a silent truncation that lops off the one paragraph that mattered.

What this thing does: you paste text, it runs a per-provider estimator that tries to mirror how the GPT-4o, Claude and Gemini tokenizers actually behave, and it works out how that same content would get billed and stored on each model. I built it to lean cautious. It overcounts a little on tiny snippets and undercounts a little on dense code, which honestly is the direction you want to err when you’re planning production spend. And the text never leaves the browser, which matters a lot when what you’re pasting is a secret, customer data, or some feature nobody’s announced yet.

How tokenisation works across GPT-5, Claude 4 and Gemini 3

OpenAI runs on byte-pair encoding, BPE for short. The GPT-5 family inherits that o200k_base-style vocabulary that showed up with GPT-4o, plus some extra entries that make it leaner on code and non-English text. Anthropic’s Claude 4 (Opus 4.7, Sonnet 4.6 and Haiku 4) uses its own BPE tokenizer, also tuned for code and other languages, and it usually lands within a few percent of the OpenAI counts on comparable text. Gemini 3, both Pro and Flash, goes a different route with a SentencePiece-style tokenizer on a separate vocabulary. The upshot? It sometimes carves the exact same input into more tokens. Punctuation-heavy text and emoji are where you’ll feel it most.

1. Normalise the text so encoded characters, line endings and that stray whitespace nobody notices all match what the API actually receives.
2. Detect common chunks: words, numbers, code symbols, line breaks, the odd rare character.
3. Apply per-category cost. Short everyday words tend to be one token. Longer ones split into several. Code symbols and punctuation are usually a token apiece, line breaks count too, and Unicode-heavy text can chew through a few tokens per single character.
4. Apply a model-specific multiplier to account for how BPE and SentencePiece vocabularies disagree.
5. Project context fit and cost. Take the input estimate, add the expected output, fold in the model’s published pricing, and you’ve got a per-run figure that scales more or less linearly up to monthly volume.

Common use cases for a token counter

• Cost forecasting before a launch. Multiply the input-plus-output estimate by the monthly run count you’re projecting, and you’ll see fast whether a feature flies on a premium model like Opus 4.7 or GPT-5.5, or whether it only pencils out on something smaller like Haiku 4 or Gemini 3 Flash. A 1,500-token system prompt fired 100,000 times a month? That climbs faster than most teams brace for.
• Prompt budget design. When a system prompt, a retrieved knowledge chunk and a user message are all fighting for the same window, knowing each piece’s token count lets you carve up the budget on purpose. Beats discovering at runtime that your document got truncated.
• Model selection. A 50,000-token input is nothing on Opus 4.7 (1M window) or Gemini 3 Pro (2M). Drop it into some older model with a 32K window and suddenly it’s a squeeze. The fit chart here shows that gap in one look.
• Auditing long content pipelines. Articles, transcripts, PDFs, they almost always tokenise larger than you’d guess. Run a typical document through and you’ll find out whether a summarisation pipeline blows past the context limit on the outliers.
• RAG and embeddings sizing. Retrieval augmented generation chops documents into chunks of some target token size. Paste a representative chunk and you can confirm the splitter’s behaving before you go embed thousands of pages.
• Educational walk-throughs. Show a non-technical stakeholder that the same paragraph is 80 tokens on one model and 92 on another, and suddenly the pricing decision is a lot easier to defend.
• Latency planning. Output tokens usually drive response time more than input does. A decent guess at how long the reply runs helps you dodge timeouts on the big generations.

Limitations and privacy notes

Let’s be clear: this is an estimator, not anybody’s official tokenizer. The real byte-pair encoding behind GPT-4o, and the SentencePiece encoding Gemini leans on, are both deterministic, but running them properly means shipping multi-megabyte vocabulary files into your browser. That would drag the page down for everyone, so I didn’t. What’s here matches the official counts within a few percent on ordinary English, code and mixed content. It can wander further off on dense emoji strings, on unusual scripts, on very short snippets. When you need the number to be exact before you bill, reach for the OpenAI tiktoken library, the Anthropic token counter API, or Google’s own AI counters.

Everything’s processed inside your browser. Nothing goes to PeopleAreGeek, nothing goes to a third party. Paste your prompts, your system messages, retrieval chunks, real production samples, none of it touches a network round trip. One caveat on the pricing: the cost estimator uses published 2026 list rates, so it leaves out enterprise discounts, batch APIs and cached-input pricing. Those can slash the bill quite a bit once you’re running at real volume.

Frequently asked questions

Sources & further reading

• OpenAI: tiktoken (BPE tokenizer)
• OpenAI: API reference

Related tools and resources

Counting tokens is step one. The real wins come when you pair it with prompt engineering and a proper content audit. Here’s what to reach for next, once you know what the tokens cost you.