What are AI tokens? Tokens per second, context length and cost
Tokens are how AI models read and write text. Knowing what they are makes speed, context length and cost much easier to understand.
Last updated October 2026
What a token is
Models don't read whole words. They split text into tokens: short pieces that are often a whole common word, part of a longer word, or a punctuation mark. A page of text is many tokens, and both what you send and what the model writes back are counted in tokens.
Speed: tokens per second
Speed is usually measured in tokens per second (often written tokens/sec): how many tokens a model writes each second. There are two parts people notice: how long before the first words appear, and how quickly the rest follows. For chat, it feels fast enough when the answer appears faster than you can comfortably read it. For long summaries, a short wait for the first words matters less than steady progress.
What affects speed
- The model: models with more parameters are usually slower on the same hardware.
- The hardware, especially the GPU or memory system and how quickly it can move data in and out of memory.
- How much text you send: long documents take longer to read before the answer starts.
- How many people are asking at once.
Context length: how much it can read at once
Context length is the most tokens a model can consider at one time, including your question, any documents and its answer. A longer context lets it read longer documents in one go, but uses more memory and takes longer. For large libraries, systems usually find the relevant passages first and give the model only those.
Why owned hardware means no per-token bill
Many hosted AI services charge for usage, often counted in tokens, on top of or instead of a subscription. When the model runs on hardware you own, there's no provider counting your tokens: the costs are the hardware, its power and its upkeep, whether you use it a little or a lot. That doesn't make it cheaper for everyone; it makes the cost predictable.