--- title: "Positional Encoding and RoPE in LLMs" description: "this blog will provide the concept of positional embedding and RoPE" author: "Yash Srivastava" published: 2026-06-17 updated: 2026-06-17 canonical: https://answers.mindstick.com/blog/410/positional-encoding-and-rope-in-llms category: "technology" tags: ["llm", "ai model", "embedding"] reading_time: 5 minutes --- # Positional Encoding and RoPE in LLMs ## Introduction The foundation of some modern Large Language Models ([LLMs](https://www.mindstick.com/forum/162047/what-are-large-language-models-llms-and-how-do-they-work)) like ChatGPT, Llama, Gemini and Claude is Transformers. Transformers are capable of processing all the Tokens in parallel. Transformer process the whole sentence simultaneously instead of processing the word one by one like RNN and LSTM. Since the Transformers process the sentence in parallel which make them extremely fast, it also creates new problem like the model does not know the order of words in a sentence. *example:* ```plaintext Dog bites man and Man bites dog ``` contain exactly the same words but have completely different meanings. For humans, understanding the order of words is natural. But Transformers only see numerical vectors and they cannot understand the position of words automatically. Without positional information every token look independent and model lose the sentence structure. To [solve this problem](https://answers.mindstick.com/qa/95669/my-website-appears-different-in-firefox-and-different-in-chrome-how-do-i-solve-this-problem), Transformers use [Positional Encoding](https://answers.mindstick.com/qa/116779/why-is-positional-encoding-needed-in-transformers). Positional Encoding provide positional information of each token which help the model to understand the order of words while processing all tokens in parallel. As the [Transformer architecture](https://www.mindstick.com/forum/162056/how-does-the-transformer-architecture-work) evolved, researchers introduced various methods to represent the positional information. Some common approaches are: - Sinusoidal Positional Encoding - Learned Positional Embeddings - [Rotary Positional](https://answers.mindstick.com/qa/116797/compare-learned-positional-embeddings-sinusoidal-embeddings-and-rotary-positional-embeddings-rope) Embeddings (RoPE) Among all these approaches, RoPE become one of the most popular positional technique used in modern LLMs. It helps model to understand long context better and [improve the performance](https://www.mindstick.com/forum/160682/how-to-improve-the-performance-of-css-files) on longer sequences. Before learning about RoPE, first we have to understand why positional information is needed in Transformers and how traditional positional encoding works. ## Why Do Transformers Need Positional Information? Let's understand the problem, suppose we have any sentence: ```plaintext I love AI ``` After tokenization: ```plaintext ["I", "love", "AI"] ``` After embeddings: ```plaintext I → [0.2, 0.5, ...] love → [1.1, -0.4, ...] AI → [0.8, 0.7, ...] ``` The Transformer receives only vectors. It does not know: ```plaintext I comes first love comes second AI comes third ``` For the model: ```plaintext [0.2,0.5,...] [1.1,-0.4,...] [0.8,0.7,...] ``` are simply vectors but here no position information exists. ## What Problem Does This Create? Consider two sentences: ```plaintext sentence 1: Dog bites man sentence 2: Man bites dog ``` [Embeddings](https://www.mindstick.com/forum/162047/what-are-large-language-models-llms-and-how-do-they-work) only represent meaning, they do not represent order. Without positional information: ```plaintext Dog Bites Man and Man Bites Dog ``` look very similar to the Transformer. But the meanings are completely different. so the transformer Needs Position Information. ## Solution: Positional Encoding Researchers introduced Positional Encoding the idea of Positional Encoding is simple every token will receive Token Meaning and position information. Final Input: ```plaintext Final Embedding = Token Embedding + Positional Encoding ``` *Example:* ```plaintext Token Embedding = [0.2, 0.4, 0.6] Position 1 Encoding = [0.01, 0.02, 0.03] ``` Final: ```plaintext [0.21, 0.42, 0.63] ``` Now the model knows both the meaning and position. ## Types of Positional Encoding Over time researchers proposed different methods: 1. Sinusoidal Positional Encoding 2. Learned Positional Embeddings 3. Rotary Positional Embeddings (RoPE) ## 1. Sinusoidal Positional Encoding This was introduced in the Transformer research paper named Attention Is All You Need (2017). Instead of learning positions, researchers generated them using mathematical functions. In transformers the encoding uses Sine and Cosine functions. such that each position receives a unique pattern. *Example:* ```plaintext Position 1 → Pattern A Position 2 → Pattern B Position 3 → Pattern C ``` Every position becomes mathematically unique. ### Why Sine and Cosine? Because they provide Smooth Patterns, Periodic behaviour and Relative Distance Information. here nearby positions receive similar representations and the far positions receive different representations which helps Attention understand sequence order. - Limitations of Sinusoidal Encoding - Fixed Representation - Less Flexible - Long Context Issues ## Learned Positional Embeddings Here Instead of generating positions mathematically the model learns them. Example: ```plaintext Position 1 → Learned Vector Position 2 → Learned Vector Position 3 → Learned Vector ``` Similar to token embeddings. During training: ```plaintext Backpropagation ↓ Position Vectors Updated ``` The model learns optimal position information. ## Advantages - More flexible - Learns task-specific positions - [Better performance](https://www.mindstick.com/forum/156505/where-do-i-put-my-javascript-code-in-my-file-for-better-performance) in some tasks ## ## What is RoPE? RoPE stands for Rotary Positional Embeddings. RoPE was introduced to provide positional information directly inside the [Attention mechanism](https://answers.mindstick.com/blog/406/attention-mechanism-in-llms). Instead of: ```plaintext Embedding + Position ``` RoPE modifies: ```plaintext Query Key ``` vectors before Attention calculation which is a major difference. ## ## Why Modern LLMs Use RoPE RoPE provides several benefits like: - Better Long Context Understanding - Relative Position Awareness - Better Generalization - Efficient Computation Most modern LLMs use RoPE like: - Llama - Mistral - Qwen - DeepSeek - GPT-style architectures RoPE has become the [industry standard](https://answers.mindstick.com/qa/51696/what-is-thunderbolt-is-it-an-industry-standard-what-advantages-does-it-offer-are-there-any-disadvantages) positional encoding technique. ## Positional Encoding vs RoPE | Positional Encoding | RoPE | | --- | --- | | Added to embeddings | Applied to Q and K | | Absolute positions | Relative positions | | Limited long context | Better long context | | Less scalable | Highly scalable | | Original Transformer | Modern LLMs | --- Original Source: https://answers.mindstick.com/blog/410/positional-encoding-and-rope-in-llms Copyright © MindStick Software Pvt. Ltd. This Markdown version is provided for developers, AI systems, and offline reading.