DeepSeek’s Engram Module: How 196 Billion Parameters of Conditional Memory Reshape LLM Architecture

The dominant architectural story of the large language model era has been Mixture-of-Experts (MoE). By sparsely activating parameters — DeepSeek V4.1 Flash activates just 8 billion of its 552 billion total parameters per token during prefill — MoE decouples model capacity from inference cost. But MoE solves only half of the efficiency problem. It optimizes how models compute. It does not optimize how models remember.

DeepSeek’s Engram module, introduced in the paper “Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models” and integrated into the V4.1 Flash architecture, addresses the other half. It gives the Transformer backbone a native knowledge lookup primitive — a mechanism that retrieves static facts through constant-time O(1) hash lookups rather than through expensive multi-layer attention computation.

In V4.1 Flash, the Engram module carries 196 billion parameters — roughly 26% of the model’s total parameter count — distributed across dedicated layers, sitting alongside the 552-billion-parameter MoE backbone. Its inclusion is not an incremental optimization. It is a structural rethinking of what a language model is.

The Problem Engram Solves

Standard Transformers lack a native mechanism for knowledge retrieval. When a model needs to resolve a common entity — say, “Paris” in “The capital of France is Paris” — it cannot simply look up the answer. Instead, it must consume multiple layers of attention and feed-forward computation to reconstruct the association from its weight matrix. The model is, in effect, running an expensive runtime simulation of a lookup table every time it encounters a fact it already knows.

This is wasteful for a simple reason: language is not uniformly complex. A substantial portion of text — named entities, formulaic patterns, common collocations — is local, static, and highly stereotyped. The N-gram models of the pre-deep-learning era captured these regularities efficiently through statistical lookup. Modern Transformers, despite their vastly greater reasoning capacity, are forced to simulate that same lookup through computation because they lack a lookup primitive.

DeepSeek’s research team quantified this cost. Resolving a common multi-token entity can consume the first several layers of attention and feed-forward networks — sequential depth that could otherwise be allocated to higher-level reasoning. The model’s early layers are effectively performing static reconstruction, a task that N-gram embedding lookup can accomplish in O(1) time.

How Engram Works

Engram modernizes N-gram embedding for the deep learning era. The module operates in two phases for each token position:

Phase 1: Retrieval

The module takes the local context — the most recent few tokens — compresses it, and maps it through multi-head hashing into a massive embedding table. The lookup is approximately O(1) in time complexity, and the module’s deterministic addressing enables runtime prefetching from host memory, incurring negligible latency overhead.

The hash-based addressing is the key engineering insight. Unlike attention, which computes relevance scores across the entire context, Engram’s lookup is a direct table access. The model does not need to compute which memory entries are relevant; the hash function determines the address deterministically from the local context.

Phase 2: Fusion

A context-aware gating mechanism merges the retrieved static embedding with the model’s dynamic hidden state. The gate learns how much to trust the lookup versus the computed representation at each position. For tokens that are part of well-known patterns, the gate opens and the lookup dominates. For tokens requiring novel reasoning, the gate closes and the computation path takes over.

This separation is the architectural parallel to the asymmetric compute split in V4.1 Flash’s CED architecture. Just as CED assigns 8 billion parameters to prefill (input-heavy, knowledge-retrieval-heavy) and 16 billion to decode (output-heavy, generation-heavy), Engram separates knowledge retrieval (cheap, static, lookup-based) from reasoning (expensive, dynamic, computation-based).

The U-Shaped Scaling Law

The most significant finding in the Engram paper is the discovery of a U-shaped scaling law that governs the allocation of parameters between MoE computation and Engram memory.

DeepSeek researchers ran controlled experiments where total parameter count and FLOPs were held constant, and the ratio of MoE parameters to Engram parameters was systematically varied. The results show:

• Pure MoE (no Engram): Good performance, but computation is wasted on static knowledge reconstruction
• Too much Engram (over-allocated memory): Performance degrades because the model lacks sufficient computational capacity for dynamic reasoning
• Optimal balance (20-25% Engram, 75-80% MoE): Peak performance across all evaluated benchmarks

The U-shape mirrors the bias-variance tradeoff in classical machine learning. Too little memory and the model overcomputes; too much memory and the model undercomputes. The optimum is a balance where static patterns are offloaded to lookup and dynamic reasoning receives the full computational budget.

V4.1 Flash’s architecture reflects this optimum. The 196-billion-parameter Engram module represents approximately 26% of the model’s total parameter budget — within the optimal range identified by the scaling law.

The Performance Gains Are Not Where You Would Expect

One might assume that a knowledge lookup module would primarily help with factual recall benchmarks — MMLU, CMMLU, and similar knowledge tests. The gains there are real: MMLU +3.4 points, CMMLU +4.0 points. But the larger gains appear in domains that are not obviously knowledge-retrieval tasks:

Benchmark	Gain from Engram
MMLU	+3.4
CMMLU	+4.0
BBH (general reasoning)	+5.0
ARC-Challenge	+3.7
HumanEval (code)	+3.0
MATH	+2.4
Multi-Query NIAH (long-context retrieval)	84.2 → 97.0

The explanation is mechanistic. By delegating local dependency resolution to Engram lookups, the model’s early layers are freed from static reconstruction. The network effectively becomes deeper for complex reasoning — the same number of layers, but more of them devoted to higher-level computation rather than pattern matching. The attention mechanism also benefits: by handling local dependencies through lookups, attention capacity is freed for global context, dramatically improving long-context retrieval.

The Multi-Query Needle-in-a-Haystack improvement from 84.2 to 97.0 is particularly striking. The module that was designed to help with factual recall turns out to be transformative for long-context retrieval — because the same offloading logic applies. Local pattern matching is handled by Engram; attention focuses on global information flow.

Infrastructure Implications: CXL and the Memory Hierarchy

Engram’s design has implications that extend beyond model architecture into hardware infrastructure. The module’s embedding tables are massive — 196 billion parameters in V4.1 Flash — but its access pattern is sparse and predictable. The deterministic hash addressing means that embeddings can be prefetched from host memory, overlapping with the computation of non-Engram layers.

This characteristic makes Engram an ideal candidate for memory disaggregation. A 2026 paper from Peking University and Alibaba Cloud, “Pooling Engram Conditional Memory in Large Language Models using CXL,” demonstrated that Engram parameters can be offloaded to CXL (Compute Express Link) memory pools with near-DRAM end-to-end performance. Unlike RDMA-based pooling, which incurs network stack overhead, CXL provides hardware-level load/store primitives that match Engram’s fine-grained, low-latency access requirements.

The implication is significant. As Engram scales to hundreds of gigabytes in future models, the memory can be pooled across compute nodes rather than replicated on each GPU. A shared Engram memory pool — accessed via CXL switches — could serve multiple inference nodes simultaneously, dramatically reducing the per-node memory cost of large-scale Engram deployment.

The Engram Ecosystem: PEFT, Adapters, and Community Adoption

Engram has already spawned a community ecosystem beyond DeepSeek’s own implementation. The open-source engram-peft package provides a PEFT-style interface for injecting conditional memory into any Transformer-based LLM. It supports LoRA+Engram hybrid fine-tuning, where the combined approach achieves better convergence than either method alone — 2.3% better than standalone LoRA, 5% better than standalone Engram, in published benchmarks.

The “Engram Adapter” paradigm extends the concept to domain specialization. Rather than injecting Engram at pretraining time, adapters use N-gram pattern matching as a conditional gate that determines when domain-specific residuals should be applied. The result is adaptation that improves in-domain performance while preserving 99.4%–100.1% of out-of-domain capabilities — a dramatic improvement over always-on PEFT methods that degrade general performance.

What Engram Means for the Next Generation

V4.1 Flash is the smallest model in DeepSeek’s new architecture family. The technical report explicitly states that the CED architecture, CSA2 attention, and Engram memory are designed to scale to larger models. When V4.1 Pro launches, the Engram module will likely grow proportionally — potentially exceeding 300 billion parameters.

The U-shaped scaling law suggests that this growth is not just additive but multiplicative. As Engram capacity increases, the backbone’s computational budget is freed for deeper reasoning. The model does not just know more facts; it reasons better about them, because the layers that were previously consumed by static reconstruction are now available for dynamic analysis.

This is the architectural bet that DeepSeek is making. The industry’s dominant strategy has been to scale MoE — more experts, larger expert capacity, finer-grained routing. DeepSeek is scaling along a second axis: conditional memory. The bet is that the combination of sparse computation and sparse retrieval will outperform pure sparse computation at any given parameter and FLOP budget.

The early evidence supports the thesis. V4.1 Flash, with its Engram-augmented architecture, outperforms V4 Pro — a model with a larger MoE backbone but no conditional memory — across most benchmarks. The architectural gap will widen as DeepSeek scales the Engram module in V4.1 Pro and beyond.

Conditional memory is not a plugin. It is a foundational modeling primitive — and the next generation of large language models will be built on it.