AI Prep • 3 min read • Reviewed 2026-11-11

Why Token Counts Differ Across OpenAI, Claude, and Gemini

How BPE vocabularies diverge across English prose, source code, JSON, and non-Latin text — and why honest estimates beat false precision.

Exact local BPE vs. estimated multipliers

OpenAI publishes open BPE rank tables (o200k_base and cl100k_base) that can run 100% locally in your browser for exact token counts. Proprietary Claude and Gemini tokenizers are not distributed for offline client bundling, meaning any purely client-side counter for those models must report an estimate range.

Tools Referenced in This Guide