A token is the smallest unit of text that an AI model treats as one single thing.

The simple definition

A token is a small chunk of text. It might be a whole word like cat, a piece of a longer word like believ or able, a single character, a punctuation mark such as !, or even a space attached to the front of a word. The AI model does not see sentences or words. It sees a long row of tokens, one after another.

Take the word “unbelievable”. A human sees one word. A typical AI tokenizer might split it into three pieces: “un”, “believ”, and “able”. The word “cat” is so common that it usually stays as one token. The exact split depends on the tokenizer, which is the tool that does the chopping, and we will meet it properly in a later chapter.

Why tokens exist

Computers only understand numbers. They cannot do math on the letter “a” or on the word “apple”. So before a model can learn from text or answer a question, the text must be turned into numbers. Tokens are the middle step. First, text is cut into tokens. Then each token is swapped for a number. The model only ever handles those numbers.

Where tokens are used

Every modern language model works this way. ChatGPT from OpenAI, Gemini from Google, Claude from Anthropic, and Llama from Meta all read and write tokens. The same idea is also used for code, images, and sound, which we will cover later. Whenever an AI product shows a limit such as “128,000 tokens”, or a price “per million tokens”, it is talking about these pieces.

A handy rule of thumb

For normal English, one token is roughly four characters, or about three quarters of a word. So 100 tokens is about 75 words. A short paragraph of 75 words is about 100 tokens, and a full page of text is often somewhere near 500 to 700 tokens. These are only averages, but they are good enough for quick planning.

Tokens did not appear overnight. Researchers tried several ideas before settling on today’s approach.

Early language software was simple. It split text on spaces and treated every word as one unit. This worked for small projects, but it had a big problem. Any word that was not in the program’s word list, such as a new slang term, a name, or a spelling mistake, became an “unknown” and was thrown away. The model could not read it at all.

The fix came from an unexpected place: file compression. The story below shows how an old compression trick became the foundation of modern AI text handling.

Today almost every major language model uses some form of subword tokenization. The details differ between companies, but the big idea stays the same: break text into reusable pieces that are bigger than letters and often smaller than words.

There are three ways to cut text. Each one has a cost, and subword pieces strike the best balance.

Option one: one token per word

This feels natural, but English alone has hundreds of thousands of words, plus names, typos, and new slang every week. A word-based list would be enormous, and any word missing from the list would be lost. Languages that build long words by gluing parts together, such as German, make the problem even worse.

Option two: one token per character

Here the list is tiny. A few hundred symbols cover almost everything, and no word can ever be “unknown”. The catch is length. A single sentence becomes a very long row of tokens, and a model has to work much harder to learn that the letters c-a-t together mean a small furry animal.

Option three: subword pieces

Subword tokenizers keep common words whole and break rare words into familiar parts. The word “the” stays as one token. A rare word such as “photosynthesizing” might become a few pieces. This keeps the vocabulary at a manageable size, keeps sentences reasonably short, and still handles brand-new words.

The tokenizer is a small program that sits at the front door of every language model.

A tokenizer is the tool that turns text into tokens and turns tokens back into text. It is not the AI model itself. It is a separate piece of software that follows fixed rules learned in advance. The same tokenizer must be used when the model is trained and when it is used, or the numbers would mean different things.

flowchart LR
  A["Your text"] --> B["Tokenizer splits text into pieces"]
  B --> C["Token IDs: a list of whole numbers"]
  C --> D["Embedding layer: each ID becomes a vector"]
  D --> E["Transformer layers: attention and processing"]
  E --> F["Scores for every token in the vocabulary"]
  F --> G["One token is chosen"]
  G --> H["Detokenizer turns IDs back into text"]
    

The diagram shows a one-way trip with a return ticket. On the way in, the tokenizer and embedding layer prepare your words. In the middle, the transformer, which is the main engine of modern language models, does the thinking. On the way out, the model scores every possible next token, one is chosen, and the detokenizer joins the pieces back into normal text.

A worked example

Here is how a typical tokenizer might handle a short sentence. The exact splits differ from model to model, so treat this table as an illustration, not as the output of any one product.

The pieces are not chosen by hand. They are discovered automatically from huge amounts of text.

The most popular method is byte pair encoding, usually shortened to BPE. The word “byte” here simply means a basic unit of computer data, and a “pair” means two neighbors sitting next to each other. The method is surprisingly simple, and you can understand it without any math.

Why “byte-level” matters

A plain character-based version of BPE can still run into trouble with strange symbols. The byte-level version starts from the 256 possible values of a computer byte instead. Since every piece of text on a computer is stored as bytes, nothing can ever be “unknown”. Emoji, rare alphabets, and even broken text can always be represented. GPT-2 made this approach famous, and many models use it today.

Other tokenizer families

Every token has a number, and every tokenizer has a fixed list of all the tokens it knows.

What a vocabulary is

The vocabulary is the complete list of tokens a tokenizer knows, and each one has a fixed number called its token ID. The model never sees the word “cat”. It sees a number, such as 9246 in one tokenizer and a completely different number in another. The numbers have no meaning on their own. They are just labels, like seat numbers.

How big are real vocabularies?

Vocabulary size is a design choice, and it has changed over time. Newer models tend to use bigger lists, which helps compress text into fewer tokens, especially for languages other than English.

The vocabulary-size trade-off

A larger vocabulary means common words and phrases fit into fewer tokens, so the same text takes less space in the context window and costs less to process. The price is that the model needs a bigger table of meanings and a bigger final scoring step. Designers pick a size that balances compression against the extra memory and training cost.

Special tokens

Besides normal text pieces, every vocabulary includes a few special tokens. They are never typed by a user. They are control signals that help the model understand structure.

A language model has one core job: look at the tokens so far and guess the next one.

From an ID to a meaning

The first thing the model does with a token ID is look it up in a big table called the embedding table. An embedding is a long list of numbers, often hundreds or thousands of them, that describes what a token means and how it behaves. Tokens with similar meanings end up with similar lists. The embedding of “king” sits closer to “queen” than to “banana”.

Paying attention

Next, the tokens pass through many layers of the transformer, the design behind almost every modern language model. Its key trick is called attention. Attention lets each token look back at all the other tokens and decide which ones matter most for understanding it. In the sentence “The trophy did not fit in the suitcase because it was too big”, attention helps the model work out that “it” points to the trophy.

Guessing the next token

At the end, the model produces a score for every single token in its vocabulary. If the vocabulary has 100,000 tokens, it produces 100,000 scores. These scores are turned into chances, so “Paris” might get a very high chance after “The capital of France is”, while “banana” gets almost none. One token is then picked from those chances.

One token at a time

The chosen token is added to the end of the text, and the whole process runs again to pick the next one. This loop continues until the model picks a special end token or hits the maximum length it was allowed. An answer of 300 tokens means the model went around this loop 300 times.

flowchart TD
  A["Prompt tokens go in"] --> B["Model reads all tokens so far"]
  B --> C["Model scores every token in the vocabulary"]
  C --> D["Sampler picks one token"]
  D --> E{"End token or length limit reached?"}
  E -->|No| F["Add the new token to the sequence"]
  F --> B
  E -->|Yes| G["Stop and return the finished answer"]
    

The diagram makes one important fact visible: the model does not write the whole answer at once. It writes a token, then reads everything again including that new token, then writes the next one. This is why answers appear on screen word by word, and why longer answers take more time.

Why the same question can give different answers

The picking step usually has a little randomness. A setting called temperature controls it. With a low temperature, the model almost always picks the highest-chance token, so answers are steady and repeatable. With a higher temperature, it sometimes picks less likely tokens, so answers are more varied and creative. This is why asking the same question twice can give two different replies.

A model can only look at a limited number of tokens at one time. That limit is called the context window.

The context window is the maximum number of tokens a model can handle in a single request. It is the model’s working memory. Everything has to fit inside it: the instructions you give, the earlier chat messages, any documents you attach, your new question, and the answer the model is about to write.

flowchart LR
  subgraph W["Context window: one shared token budget"]
    A["System instructions"]
    B["Earlier chat history"]
    C["Documents or retrieved facts"]
    D["Your new question"]
    E["Room reserved for the answer"]
  end
  A --> B --> C --> D --> E
    

Look at the figure again: the answer is not free space outside the window. It lives inside the same budget. If you fill the window almost completely with a giant document, the model has very little room left to reply, and the answer may be cut short.

How context windows have grown

Early models could only see a few pages. The original GPT-3 handled 2,048 tokens. When GPT-4 launched in 2023, it came in 8,192-token and 32,768-token versions. In 2024, Google introduced Gemini 1.5 with a window of up to one million tokens, enough for several books at once. Many recent models offer hundreds of thousands of tokens. The trend is clear, but bigger windows come with their own costs.

Why a longer chat gets forgotten

When you use an AI model through a programming interface, it does not remember earlier turns by itself. The application must send the whole conversation again with every new message. A long chat therefore grows with each turn. When it reaches the limit, the application has to drop or shorten the oldest parts, and the model seems to “forget” them.

Bigger is not always better

A large window does not mean the model uses every part of it equally well. Research from 2023, often called the “lost in the middle” study, found that some models were better at using facts placed at the very beginning or end of a long input than facts buried in the middle. Also, the standard attention method gets more expensive as the input grows, because every token looks at every other token. Doubling the input can more than double the work. Engineers keep inventing ways to reduce this cost, but it never completely disappears.

Tokens are the unit of billing, the unit of speed, and the unit of limits for almost every AI service.

You pay per token

Most AI companies charge by the token, usually quoted as a price per million tokens. Input tokens (what you send) and output tokens (what the model writes) are priced separately, and output tokens usually cost more. That is because each output token needs one full trip around the generation loop from the last chapter, while input tokens can be processed together in one go.

What counts as a token on the bill

Tokens and speed

Speed has two parts. The wait before the first word appears depends mostly on how many input tokens must be read. After that, the answer streams out one token at a time, so the total time depends on how many output tokens are written. A request with a huge input and a short answer feels different from a short input with a very long answer.

Limits are counted in tokens too

Providers also limit how fast you can use their service, often as a number of tokens per minute along with a number of requests per minute. If your app suddenly sends much longer prompts, you can hit a limit even though the number of requests stayed the same. There is usually a separate setting for the maximum length of a single answer as well.

Many strange AI mistakes come from the fact that the model sees pieces, not letters.

When an AI model fails at something that seems easy to a child, tokens are often the reason. The model never looks at the individual letters of a word if that word is stored as one or two big pieces. Here are the most common surprises.

The good news is that many of these problems are being reduced. Larger vocabularies, better training data for more languages, and tools such as code execution all help. But understanding the root cause, that the model reads pieces, helps you predict where mistakes will happen and design around them.

The token idea is not limited to words. It is a general way of turning anything into a row of small units.

Modern models are multimodal, which means they handle more than one kind of data, such as text, pictures, and sound. To do this, each kind of data is converted into token-like units that the same model can process.

Once an AI feature is live, tokens become something you must protect, measure, and cap.

Security concerns tied to tokens

What to measure

You cannot control what you do not measure. Good teams log token counts for every request and watch them on a dashboard, just as they watch response time and errors.

flowchart LR
  U["User app"] --> G["API gateway with rate limits"]
  G --> T["Token counter and budget check"]
  T -->|Within budget| L["Language model service"]
  T -->|Over budget| R["Reject or shorten the request"]
  L --> M["Usage logger"]
  M --> D["Dashboard and alerts"]
  L --> U
    

In this design, nothing reaches the model without passing a budget check. The gateway limits how fast each user can send requests. The counter estimates the size of the prompt and decides whether it fits the allowed budget. After the model answers, the logger records the real usage, and the dashboard raises an alert if numbers jump unexpectedly.

A few simple habits will keep your AI features fast, cheap, and reliable.

Patterns that work

Mistakes to avoid

Here is how the ideas in this guide show up in everyday products.

You cannot change how a model thinks, but you can control how many tokens it writes and how it picks them.

Every time you call a language model, you can adjust a few settings. These settings do not change what the model knows. They change how the next-token guessing loop behaves. Learning them gives you far more control over cost, length, and style, and most of them are measured in tokens or in chances between tokens.

The maximum output length

The most important setting is the maximum output tokens limit. It tells the model to stop after a certain number of new tokens, no matter what. If the answer is not finished by then, it is simply cut off, sometimes in the middle of a sentence. Many services also report a “finish reason” with each reply, which says whether the model stopped naturally or was stopped by the length limit. Checking that reason is the easiest way to detect cut-off answers in your own app.

Stop sequences

A stop sequence is a short piece of text that tells the model to halt as soon as it writes it. For example, you might ask for a list and tell the model to stop when it reaches a special marker. This is a neat way to prevent rambling and to save tokens, because generation ends immediately instead of running to the length limit.

Choosing among likely tokens

Remember that at every step the model gives a chance to every token in its vocabulary. Settings called sampling controls decide how the final pick is made from those chances.

Streaming makes tokens visible

Because the model produces one token at a time, many services let you receive the answer as a stream, token by token, instead of waiting for the full reply. This is what makes chat apps feel quick: the first words appear almost immediately while the rest is still being written. Streaming does not reduce the total number of tokens. It only lets people start reading earlier, which greatly improves how fast the product feels.

Practical settings advice

Tokens are not only what you type. They are also how the size of a model’s training library is counted.

Before a model can answer your question, it has to learn language. This first big learning stage is called pretraining. The model reads an enormous collection of text, split into tokens, and practices one simple game over and over: look at the tokens so far and predict the next one. When it guesses wrong, its internal numbers are adjusted a tiny bit. After trillions of these small corrections, it becomes surprisingly good at language.

How training data is measured

Companies describe the size of training data in tokens, not in pages or books, because tokens are what the model actually consumes. The numbers have grown dramatically. The original GPT-3 was trained on roughly 300 billion tokens. DeepMind’s Chinchilla model, a well-known 2022 research result, used 1.4 trillion tokens. Meta said that its Llama 3 models were pretrained on more than 15 trillion tokens.

Tokens versus parameters

People often mix up tokens with parameters. Parameters are the adjustable numbers inside the model, the “knobs” that get tuned during training. Tokens are the text the model reads while tuning those knobs. The Chinchilla research suggested that, for a fixed training budget, many earlier models had too many knobs and had read too little text. A commonly quoted guideline from that work is roughly 20 training tokens per parameter. Since then, many teams have trained smaller models on far more tokens than that guideline, because a model that has read more is cheaper to run afterward.

The main stages

Clearing up four easily confused terms

Let us put every idea together by tracing a single, ordinary chatbot request and counting its tokens.

Imagine a study-help app. A student types a short question. Behind the scenes, the app builds a full request from several parts, sends it to a language model, and shows the answer. The numbers below are invented for practice, but they follow the same pattern as real systems.

sequenceDiagram
  participant U as Student
  participant A as App backend
  participant T as Tokenizer
  participant M as Language model
  U->>A: Types a question
  A->>T: Send full prompt text
  T-->>A: Token IDs and a count
  A->>M: Request with limits set
  M-->>A: Streams output tokens
  A-->>U: Shows the answer as it arrives
    

Notice that the app talks to the tokenizer before it talks to the model. This is the budget check from the earlier chapter in action. If the count were too large, the app could trim the history or refuse the request before spending any money.

The token ledger

Here is the accounting for one turn of the conversation. Think of it like a receipt that lists where every token went.

If the model’s window is 8,192 tokens, this request uses only a small slice of it. But the history part will keep growing with every turn, and after many turns it could become the largest item on the receipt. That is why apps shorten or summarize old messages.

Turning tokens into money

To see how cost works, use made-up prices just for practice: one dollar per million input tokens and three dollars per million output tokens. These are not real prices from any company. With these numbers, the input costs 812 divided by one million, multiplied by one dollar, which is about $0.0008. The output costs 180 divided by one million, multiplied by three dollars, which is about $0.0005. The whole request costs roughly $0.0014.

That looks tiny, but scale it up. If the app handles 100,000 such requests every day, the daily cost is around $135. Over a month, that is more than $4,000. If a design change cut the input by a quarter, the saving would be noticeable on every single bill. This is why experienced teams care about even small amounts of wasted tokens.

Quick answers to the questions people ask most about tokens.