---
title: "What is token, tokenization work and context window in LLMs?"  
description: "What is token, tokenization work and context window in LLMs?"  
author: "Manish Kumar"  
published: 2026-06-11  
updated: 2026-06-14  
canonical: https://answers.mindstick.com/qa/116778/what-is-token-tokenization-work-and-context-window-in-llms  
category: "artificial-intelligence"  
tags: ["artificial intelligence", "llm"]  
reading_time: 2 minutes  

---

# What is token, tokenization work and context window in LLMs?

## Answers

### Answer by Yash Srivastava

## Token

Tokens are the basic unit of [Large Language Models](https://www.mindstick.com/news/3347/microsoft-releases-bing-chat-enterprise-and-microsoft-365-copilot-pricing-information) (LLMs). LLM cannot understand [human language](https://answers.mindstick.com/qa/98357/is-it-possible-to-teach-robots-human-language) directly in the form of sentences or words, Before [processing](https://www.mindstick.com/blog/254/background-processing-in-android) Model converts the sentences into chunks and assign by unique numerical value (Token ID) to the chunks which is called tokens.

Example:

```plaintext
I love artificial intelligence
```

[Tokenizer](https://www.mindstick.com/interview/316/what-is-string-tokenizer) will convert the sentence into:

```plaintext
["I", "love", "artificial", "intelligence"]
```

### How a token is created?

#### Step 1: Input sentence

```plaintext
What is AI?
```

#### Step 2: Tokenizer

```plaintext
["What", "is", "AI", "?"]
```

#### Step 3: Vocabulary lookup

```plaintext
What → 1578
is → 425
AI → 9821
? → 63
```

Now model gets these information instead of text:

```plaintext
[1578,425,9821,63]
```

## Tokenization

Tokenization means to divide the sentences into meaningful pieces called Tokens. Tokenization is required because computers are not able to understand the words instead of this they works on numerical values. The main problem is if we try to store each words in vocabulary like:

```plaintext
low
lower
lowest
```

Then the size of vocabulary becomes very high, Because of millions of words the storage become Huge and [training](https://www.mindstick.com/articles/198439/improve-employee-performance-with-training-films) become difficult. so we divide the word into tokens.

*Example:*

```plaintext
unhappiness
```

*into*

```plaintext
un
happy
ness
```

Now due to this model are also able to understand the new words.

Some of the famous tokenization [algorithms](https://www.mindstick.com/articles/12297/google-algorithms-why-so-important) are:

- Byte Pair [Encoding](https://www.mindstick.com/interview/752/how-is-encoding-handled-in-ajax) (BPE)
- WordPiece
- SentencePiece

There are two types of tokenizations:

- [Character](https://www.mindstick.com/blog/250/html-character-entities) Based
- Sub-word Based

## Context window

[Context Window](https://answers.mindstick.com/qa/116798/what-causes-context-window-limitations-in-modern-llms) means the maximum tokens which a model can process at a time.

*Example: Model Context Window*

```plaintext
128K Tokens
```

Which means the model can process the information of 128,000 tokens in a single request. Context also includes the older conversations. But if the context limit exceeds then the older message will dropped. Context Window is the model's [temporary](https://www.mindstick.com/forum/2292/how-to-set-temporary-path-of-jdk-in-windows) working memory not permanent memory.

#### Some models and there context size:

```plaintext
GPT 3.5  -  16k
GPT 4  -  8/32k
GPT 4 Turbo   -  128k
GPT 4o  -  128k
GPT 4.1  -  1M
Claude sonnet 4x  -  1M
Gemini 2.5   -  1M-2M
```


---

Original Source: https://answers.mindstick.com/qa/116778/what-is-token-tokenization-work-and-context-window-in-llms

Copyright © MindStick Software Pvt. Ltd. This Markdown version is provided for developers, AI systems, and offline reading.