Skip to main content
Bluecoders
← Tech glossary

Token (AI)

TermConcept

Token (AI) = the piece of a word

A token is the unit of text that a language model (LLM) works with: a fragment that can be a short word, part of a word, a punctuation mark or a character, produced by splitting the text with a tokenizer before passing it to the model.

The model does not read letters or words, but a sequence of tokens, and it generates its answer token by token. The token is the unit of measure everywhere: the size of the context window, generation speed and API billing, which usually distinguishes input tokens from output tokens.

Each model family has its own tokenizer. The same sentence therefore does not yield the same number of tokens from one model to another. Because tokenizers are often optimized for English, a text in French or a less represented language generally uses more tokens, so it costs more and takes up more context.

Why it matters when hiring

Understanding tokens is a prerequisite for any AI Engineer or LLM Engineer: it is what makes it possible to estimate the costs, latency and limits of an application. Ask the candidate how they reduced token usage on a real project (shorter prompts, caching, model choice). Someone who has never looked at an API bill has probably not shipped a product to production.

A recruitment need?

Describe what you need. A recruiter from your sector will call you back within 24 business hours.