AI Prep • 3 min read • Reviewed 2026-11-11
Why Token Counts Differ Across OpenAI, Claude, and Gemini
How BPE vocabularies diverge across English prose, source code, JSON, and non-Latin text — and why honest estimates beat false precision.
Exact local BPE vs. estimated multipliers
OpenAI publishes open BPE rank tables (o200k_base and cl100k_base) that can run 100% locally in your browser for exact token counts. Proprietary Claude and Gemini tokenizers are not distributed for offline client bundling, meaning any purely client-side counter for those models must report an estimate range.