DeepSeek fixes RoPE mismatch in the V3.2-Exp inference demo

DeepSeek API Docs Logo DeepSeek API Docs Logo # Introducing DeepSeek-V3.2-Exp 🚀 Introducing DeepSeek-V3.2-Exp — our latest experimental model! ✨ Built on V3.1-Terminus, it debuts DeepSeek Sparse Attention (DSA) for faster, more efficient training & inference on long context. 👉 Now live on App, Web, and API 💰 API prices cut by 50%+! ## ⚡️ Efficiency Gains​ 🤖 DSA achieves fine-grained sparse attention with minimal impact on output quality — boosting long-context performance & reducing compute cost. 📊 Benchmarks show V3.2-Exp performs on par with V3.1-Terminus. ## 🧑‍💻 API Update​ 🎉 Lower costs, same access! 💰 DeepSeek API prices drop 50%+, effective immediately.

The actual indexer implementation (see `inference/model.py`) uses a mathematically precise formula to compute importance scores. For each query token at position t and a preceding token at position s, the index score is: ``` I[t,s] = Σ(j=1 to H_I) w[t,j] · ReLU(q[t,j] · k[s]) ``` Where: `H_I = 64` indexer heads (half the 128 main attention heads) `q[t,j]` ∈ ℝ¹²⁸ is the query vector for head j at token t `k[s]` ∈ ℝ¹²⁸ is the key vector for token s `w[t,j]` ∈ ℝ is a learned weight for head j Design Choices: ReLU Instead of Softmax: The indexer uses ReLU activation rather than the typical softmax. This choice dramatically improves throughput because ReLU is computationally cheaper and maintains sparsity (negative scores become zero). [...] TileLang (research-oriented prototyping): Provides readable implementations of FP8 quantization and sparse indexing kernels (see `inference/kernel.py`) + `act_quant_kernel`: Block-wise FP8 quantization with 128-element blocks + `fp8_gemm_

## Update 2025.11.17: We have identified that previous versions of the inference demo code contained an implementation discrepancy in Rotary Position Embedding (RoPE) within the indexer module, potentially leading to degraded model performance. Specifically, the input tensor to RoPE in the indexer module requires a non-interleaved layout, whereas RoPE in the MLA module expects an interleaved layout. This issue has now been resolved. Please refer to the updated version of the inference demo code and take note of this implementation detail. ## Open-Source Kernels For TileLang kernels with better readability and research-purpose design, please refer to TileLang. [...] For high-performance CUDA kernels, indexer logit kernels (including paged versions) are available in DeepGEMM. Sparse attention kernels are released in FlashMLA. ## How to Run Locally ### HuggingFace We provide an updated inference demo code in the inference folder to help the community quickly get started with our mo

Rₚₒₛ(WᵀᵤₖA) ≠ WᵀᵤₖRₚₒₛ(A) The rotation is applied after the projection and varies with position. Therefore: Wᵀᵤₖ cannot be merged into W\_q. ### Consequence Without RoPE: W\_qWᵀᵤₖ is computed once and reused. With RoPE: Rₚₒₛ(Wᵀᵤₖ(XW\_dkv)ᵀ) must be reconstructed for every token position. ↓ We need to recompute the keys for all tokens during inference. ↓ More computation ↓ More memory traffic ↓ Lower inference throughput ↓ This significantly hinders inference efficiency. ## 𝙃𝙤𝙬 𝘿𝙚𝙚𝙥𝙎𝙚𝙚𝙠 𝙎𝙤𝙡𝙫𝙚𝙙 𝙏𝙝𝙞𝙨 𝙋𝙧𝙤𝙗𝙡𝙚𝙢 ? DeepSeek said if I can’t absorb Rₚₒₛ(Wᵀᵤₖ(XW\_dkv)ᵀ) while using RoPE why don’t I break down latent attention in 2 parts. Here, I break Query into 2 parts Qc and Qr where Qc is without RoPE and Qr is with RoPE and vice versa to Keys. [...] Sign up Sign in Sign up Sign in Unknown user # How DeepSeek exactly implemented Latent Attention | MLA + RoPE: Sujangyawali -- Listen Share A step-by-step explanation of Multi-Head Latent

As we have explained in the background part, this rotation matrix is position-dependent, meaning the rotation matrix for each position is different. As a result, W^{UK} can no longer be absorbed by W^Q. To address this conflict, the authors propose what they call "a decoupled RoPE", by introducing additional query vectors along with a shared key vector, and use these additional vectors in the RoPE process only, while keeping the original keys kind of isolated with the rotation matrix. The entire process of MLA can be summarized as below (equation numbers reused from Appendix C in ):  to d\_c (here D in the superscript stands for "down-proje

that degrades my performance by a huge amount because uh my value my different my content is shared across different heads. So the diversity is not there in the attention scores and my models just don't perform well on various evaluation benchmarks. Even GQA if you see they they are better than MQA because instead of sharing values or instead of replicating values across all heads they divide it into groups. So each group has the same or one group then second group third group fourth group etc. So each group has the same value but different groups have different value among one group all attention and share the same value. It's a bit better than MQA but still the performance is not quite good. So if you see actually uh in appendix D1 I think in the deepsek paper they have a comparison [...] are these different things? This h is my uh attention head dimension. N is the number of attention heads. S is the number of um S is the context length. B is the batch size. L is the number of trans