What are tokens in AI and how are they counted?

Tokens are the basic units of text that AI language models work with. Before a model reads your message, the text is split into tokens, and when it replies, it produces tokens one after another. Understanding tokens helps explain pricing, limits and some odd behaviour.

A token is usually a piece of a word. Common short words like "the" or "and" are often a single token. Longer or rarer words get split into parts, so "internationalisation" might become several tokens. Spaces, punctuation and numbers also count. As a rough rule of thumb for English, one token is about three-quarters of a word, so 1,000 tokens is roughly 750 words. The exact split depends on each model's tokenizer, so the same sentence can produce slightly different counts on different models.

Language makes a big difference. Tokenizers are often trained mostly on English text, so other languages can need more tokens for the same meaning. Hindi, Tamil, Bengali and other Indian languages written in their own scripts have historically used noticeably more tokens than English for a similar sentence. Newer tokenizers have improved this, but if you are building an app for Indian-language users, measure your actual token counts rather than assuming English rates.

Tokens matter in three main ways:

1. Cost. API pricing is per token, usually quoted per million tokens. Input tokens (what you send, including conversation history and documents) and output tokens (what the model writes) are priced separately, and output usually costs more.
2. Limits. The context window, the maximum the model can consider at once, is measured in tokens. So is the maximum length of a single reply.
3. Speed. More output tokens means a longer wait, because the model generates them one at a time.

Some practical tips to manage tokens:

Send only the relevant part of a document instead of the whole thing.
Trim old conversation history or replace it with a short summary.
Ask for concise answers when you do not need long ones.
Use prompt caching, offered by several providers, to cut costs when you send the same long instructions or documents repeatedly.
Use a smaller, cheaper model for simple tasks such as classification.

Most providers offer a token counter or tokenizer tool in their documentation, and the API response usually reports exactly how many tokens each request used. Checking those numbers early helps you estimate costs before you scale up.