--- title: "Tokenization Techniques in Large Language Models (LLMs)" description: "This blog is for implementation of tokenization and the different techniques for tokenization." author: "Yash Srivastava" published: 2026-06-18 updated: 2026-06-18 canonical: https://answers.mindstick.com/blog/416/tokenization-techniques-in-large-language-models-llms category: "technology" tags: ["llm", "ai model", "token"] reading_time: 4 minutes --- # Tokenization Techniques in Large Language Models (LLMs) ## What is tokenization? **Tokenization** is a process to break the text like statement or any prompt into smaller units so that [AI](https://www.mindstick.com/services/artificial-intelligence) and [Natural Language Processing](https://www.mindstick.com/articles/335424/natural-language-processing-nlp-in-ai) (NLP) can understand and process [human language](https://answers.mindstick.com/qa/98357/is-it-possible-to-teach-robots-human-language) more efficiently. *Example :* ```plaintext original text - Explain the concept of tokenization in AI system after tokenization - ["Explain", "The ", "Concept" ," Of", "Tokenization", "In", “AI”, "System"] ``` Why tokenization is important : 1. Help AI to understand the text 2. Faster process 3. Vocabulary [management](https://www.mindstick.com/articles/23490/tips-for-better-cash-flow-management) 4. Important for NLP task ## 1. Character Tokenization In [character](https://www.mindstick.com/articles/23551/an-investigate-distinctive-seafood-restaurant-for-your-image-stamp-character) Tokenization each character is [considered as](https://www.mindstick.com/forum/156093/what-is-considered-as-more-significant-creating-content-or-building-backlinks) a token. **Python [Implementation](https://www.mindstick.com/forum/33764/how-to-implementation-of-class-in-c-sharp):** ```python text = "I like ML" tokens = list(text) print(tokens) ``` ## C# Implementation: ```cs using System; class Program { static void Main() { string text= "I like ML"; char[] tokens= text.ToCharArray(); foreach(char token in tokens) { Console.WriteLine(token); } } } ``` *Output:* ```plaintext ['I',' ','l','i','k','e',' ','M','L'] ``` ## 2. Word Tokenization Word Tokenization divide the sentences in Words. ## Python Implementation: ```python text = "I like Machine Learning" tokens= text.split() print(tokens) ``` ## C# Implementation: ```cs using System; class Program { static void Main() { string text="I like Machine Learning"; string[] tokens=text.Split(' '); foreach(string token in tokens) { Console.WriteLine(token); } } } ``` *Output:* ```plaintext ['I','like','Machine','Learning'] ``` ## 3. Sentence Tokenization Sentence tokenization is used to divides the paragraph into [multiple](https://www.mindstick.com/blog/12797/iowa-is-expected-to-see-heavy-growth-in-multiple-sectors) sentences. ## Python Implementation: ```python import nltk from nltk.tokenize import sent_tokenize nltk.download("punkt") text= """I like ML. Machine Learning is amazing. LLMs are powerful.""" sentences= sent_tokenize(text) print(sentences) ``` ## C# Implementation: ```cs using System; class Program { static void Main() { string text= "I like ML. Machine Learning is amazing. LLMs are powerful."; string[] sentences = text.Split('.'); foreach(string sentence in sentences) { if (!string.IsNullOrWhiteSpace(sentence)) { Console.WriteLine(sentence.Trim()); } } } } ``` *Output:* ```plaintext ['I like ML.', 'Machine Learning is amazing.', 'LLMs are powerful.'] ``` ## 4. Regular Expression (Regex) Tokenization [Regular Expression](https://www.mindstick.com/articles/11909/java-regex-or-regular-expression-in-java) (Regex) tokenization remove the punctuations and generates clean tokens. ## Python Implementation: ```python import re text= "Hello, World! How are you?" tokens= re.findall(r"\w+", text) print(tokens) ``` ## C# Implementation: ```cs using System; using System.Text.RegularExpressions; class Program { static void Main() { string text="Hello, World! How are you?"; MatchCollection matches= Regex.Matches(text, @"\w+"); foreach(Match match in matches) { Console.WriteLine(match.Value); } } } ``` *Output:* ```plaintext ['Hello','World','How','are','you'] ``` ### ## 5. Converting Tokens into Token IDs LLMs do not process the text directly, each tokens are converted into ID. ## Python Implementation: ```python vocab= {"I": 0,"like": 1,"ML": 2} tokens = ["I","like","ML"] token_ids= [vocab[token] for token in tokens] print(token_ids) ``` ## C# Implementation: ```cs using System; using System.Collections.Generic; class Program { static void Main() { Dictionary vocab = new Dictionary() { {"I", 0}, {"like", 1}, {"ML", 2} }; string[] tokens={"I", "like", "ML"}; List tokenIds= new List(); foreach (string token in tokens) { tokenIds.Add(vocab[token]); } Console.WriteLine( "[" + string.Join(", ", tokenIds) + "]" ); } } ``` *Output:* ```plaintext [0,1,2] ``` ## 6. BERT WordPiece Tokenization Modern LLMs do not use simple word tokenization, instead of this they use BERT WordPiece tokenization. *[Install](https://www.mindstick.com/articles/12876/the-top-5-reasons-to-buy-and-install-a-tonneau-cover) for Python:* ```plaintext pip install transformers ``` *for C#:* ```plaintext dotnet add package Microsoft.ML.Tokenizers ``` ## Python Implementation: ```python from transformers import BertTokenizer tokenizer=BertTokenizer.from_pretrained("bert-base-uncased") tokens=tokenizer.tokenize("unbelievable") print(tokens) ``` ## C# Implementation: ```cs using System; using Microsoft.ML.Tokenizers; class Program { static void Main() { var tokenizer=BertTokenizer.Create( "vocab.txt", toLowercase: true ); var tokens=tokenizer.Tokenize("unbelievable"); foreach(var token in tokens) { Console.WriteLine(token); } } } ``` *Output:* ```plaintext ['un','##bel','##ie','##vable'] ``` ## 7. BERT Token IDs ## Python Implementation: ```plaintext from transformers import BertTokenizer tokenizer=BertTokenizer.from_pretrained("bert-base-uncased") encoded=tokenizer("I like ML") print(encoded["input_ids"]) ``` ## C# Implementation: ```cs using System; using Microsoft.ML.Tokenizers; class Program { static void Main() { var tokenizer= BertTokenizer.Create( "vocab.txt", toLowercase: true ); var encoding= tokenizer.EncodeToIds( "I like ML" ); foreach(var id in encoding) { Console.WriteLine(id); } } } ``` *Output:* ```plaintext [101,1045,2293,9932,102] ``` Where: ```plaintext 101 = [CLS] 102 = [SEP] ``` --- Original Source: https://answers.mindstick.com/blog/416/tokenization-techniques-in-large-language-models-llms Copyright © MindStick Software Pvt. Ltd. This Markdown version is provided for developers, AI systems, and offline reading.