Excerpt: Mixture-of-Experts models are changing AI scale by activating only a fraction of their parameters per token. This article explains how Mixtral, DeepSeek, LatentMoE, and Kimi K3 made sparse LLMs larger, cheaper, and more stable. #moe #llmresearch #machinelearning #aiarchitecture #deeplearning #sparsemodels

Large language models are often described in terms of raw size, but parameter count alone no longer tells the full story. Some of today’s most important AI systems are massive not because every parameter works on every token, but because the model can choose a small, relevant subset of specialized components at runtime. That design pattern is known as Mixture-of-Experts, or MoE, and it has become one of the clearest paths to scaling modern AI without paying the full computational cost of dense models.

The evolution from Mixtral to Kimi K3 shows how quickly this idea matured. Early open-weight MoE models proved that sparse routing could expand model capacity. Newer systems then refined the design with smaller experts, shared pathways, compressed routing, and better stability controls. The result is a new class of sparse large language models that deliver much larger capacity per token than earlier architectures could realistically support.

For students, developers, and AI practitioners, this shift matters because it changes how we interpret model size, training cost, inference performance, and even infrastructure design. Understanding MoE is no longer optional if you want to follow where language model architecture is heading.

Why Mixture-of-Experts Models Matter

In a standard dense Transformer, every token passes through the same feed-forward network in each layer. Attention mixes information across the sequence, then the feed-forward block transforms each token independently. This feed-forward portion holds a large share of the model’s parameters, so making it wider increases capacity, but it also increases compute for every token.

MoE changes that trade-off. Instead of one feed-forward network, a layer contains many expert feed-forward networks. A router scores those experts for each token and selects only the top few. The chosen experts process the token, and their outputs are combined into the layer result.

That sounds simple, but the implications are huge:

• The model gains access to a much larger pool of parameters.
• Only a small subset becomes active per token.
• Compute grows far more slowly than total parameter count.
• Different tokens can trigger different transformations at different depths.

This is why a model can have hundreds of billions or even trillions of parameters without using all of them on every forward pass. Sparse activation is the core efficiency trick.

Mixtral Made Sparse LLMs Easier to Understand

One of the clearest mainstream examples of open-weight MoE was Mixtral 8x7B. It helped many readers finally see what sparse scaling looks like in practice. Each MoE layer in Mixtral contains eight experts, and the router chooses two experts for every token.

The key misunderstanding to avoid is this: Mixtral is not eight separate 7B models stacked together. Its attention layers, embeddings, normalization, and other shared components exist once. What repeats are the expert feed-forward blocks inside the MoE layers.

That distinction matters when people compare dense and sparse models. Mixtral has far more total parameters than a comparable dense model, but only a fraction are active for a given token. In practice, this gives the model access to more learned capacity without requiring all experts to run every time.

Mixtral established the basic MoE pattern:

• keep expert networks full-sized,
• route each token to only a few of them,
• combine the selected outputs with learned weights.

It was a strong proof point for sparse architecture, but it also exposed a limitation. If each selected expert is still large, activating more experts quickly becomes expensive. That set the stage for the next wave of MoE design.

Why Scaling MoE Is Harder Than It Looks

At a glance, sparse activation sounds like an easy efficiency win: use only a few experts and save compute. In reality, MoE introduces new bottlenecks that dense models do not face as sharply.

The first issue is memory bandwidth. Expert weights live in GPU high-bandwidth memory, and those weights still need to be read when an expert is activated. If only a small number of tokens hit a particular expert, the system may move a lot of weight data for relatively little arithmetic. That makes memory traffic, not raw compute, the limiting factor.

The second issue is communication. Large MoE systems spread experts across multiple GPUs because no single device can hold everything. When a token needs an expert hosted elsewhere, its representation must travel to that GPU, get processed, and come back. This all-to-all communication can become expensive, especially as expert counts rise.

So while MoE reduces dense computation, it creates a balancing act among three constraints:

• computation per token,
• weight movement from memory,
• token movement across devices.

The architectures that followed Mixtral can be understood as successive attempts to manage those costs more intelligently.

DeepSeekMoE Added Finer Experts and Shared Knowledge

DeepSeekMoE pushed the idea further by changing not just how many experts exist, but what kind of experts they are. Instead of relying on a smaller number of full-width routed experts, it divided expert work into finer-grained units.

An ordinary feed-forward block usually expands a token representation into a wider hidden space, performs transformations there, and projects it back to the model width. DeepSeek made each routed expert narrower internally. That means the architecture could use more experts and activate more of them without letting compute explode in the same way it would with full-sized experts.

This finer granularity matters because it gives the router more building blocks. A token is no longer limited to choosing from a small number of large transformations. It can instead receive contributions from several smaller experts, which may encourage more flexible specialization during training.

DeepSeekMoE also introduced shared experts. Unlike routed experts, which only process selected tokens, shared experts run on every token. The intuition is practical: some transformations are broadly useful across many contexts, so forcing routed experts to relearn them repeatedly wastes capacity. Shared experts can absorb that common work once, leaving routed experts free to focus on more context-specific behavior.

This was an important conceptual shift. MoE was no longer just about sparse expert selection. It became a blend of always-on shared knowledge and selectively activated specialization.

LatentMoE Compressed the Routed Path

Even with finer-grained experts, one major inefficiency remained. The token representation sent into routed experts was still full width. That meant every selected expert still needed a wide input and output interface, and tokens still had to be communicated across GPUs at full size.

LatentMoE addressed that problem directly. Before dispatching a token to routed experts, the architecture projects the full token representation into a smaller latent space. The routed experts operate entirely inside that compressed space. Their outputs are combined there, then projected back up to the original width.

This seemingly modest change solves two practical problems at once.

• Each routed expert becomes cheaper because its input and output matrices connect to a smaller dimensional space.
• Cross-GPU communication becomes lighter because the system sends shorter token vectors to remote experts.

That is why LatentMoE became such a meaningful advance. It did not merely rearrange expert computation; it reduced the cost of the interface around the expert path. In distributed inference and training, that interface can be just as important as the expert itself.

The design also shows that MoE optimization happens along multiple dimensions. DeepSeek’s fine-graining shrank the internal middle of each expert. LatentMoE shrank the route into and out of the expert. Those are different levers, and they can work together.

Kimi K3 Turned LatentMoE Into a Scalable System

Kimi K3 is a striking example of how far sparse architecture has advanced. The model reportedly reaches 2.8 trillion total parameters while activating only about 104 billion per token. That gap between total and active parameters captures the promise of MoE more clearly than any headline number could.

Its design is bold. In most MoE layers, a router selects 16 routed experts out of 896, while two shared experts process every token. The routed branch works in a compressed latent space rather than the full model width. This allows Kimi K3 to expand expert capacity dramatically without paying the communication and memory cost that a full-width routed path would impose.

At this scale, however, architecture is not just about efficiency. It is about keeping the model trainable and numerically stable. Sparse systems with hundreds of experts per layer can become fragile if routing becomes unbalanced or if activation values become too large inside the expert stack.

Kimi K3’s Stable LatentMoE addresses those risks with three targeted mechanisms.

1. RMSNorm stabilizes the combined routed signal

Because different tokens activate different experts with different weights, the magnitude of the aggregated expert output can vary substantially. Kimi K3 applies RMSNorm after the selected expert outputs are combined and before they are projected back to full width. This keeps the scale entering the up-projection more consistent, which helps the routed branch behave more predictably during training.

2. SiTU-GLU limits activation spikes

Inside expert feed-forward networks, multiplicative gating can produce extreme activation values, especially at large scale and low precision. Kimi K3 replaces standard SwiGLU behavior with SiTU-GLU, which smoothly caps the gate and value branches before they are multiplied together. The goal is not to remove expressiveness, but to prevent runaway internal values that can destabilize training or hurt quantized computation.

This is a subtle but important lesson in modern model design: scaling is often blocked not by high-level ideas, but by low-level numerical behavior that becomes dangerous only at very large size.

3. Quantile Balancing keeps expert usage healthier

Routing imbalance is one of the classic MoE problems. If a small set of experts attracts too many tokens, those experts become bottlenecks, while underused experts receive too little training signal to become useful. Earlier approaches often added a balancing loss, but that introduces another objective for the optimizer to manage.

Kimi K3 instead uses expert-specific routing biases adjusted through Quantile Balancing. Rather than applying a fixed correction, the system estimates how much each expert’s bias should move based on score distributions across the training step. In effect, the router is nudged toward a more even global load without directly distorting the contribution weights of selected experts.

That distinction matters. The balancing mechanism influences selection, but the final contribution strength still comes from the original scoring behavior. It is a cleaner separation between routing control and expert output weighting.

How to Read Active Parameters in Sparse Models

As MoE models become more common, readers need a better way to interpret model statistics. Total parameters describe the overall capacity available in the network. Active parameters describe how much of that capacity is actually used for one token.

For dense models, those numbers are effectively the same. For MoE systems, they can be dramatically different.

That difference affects how we compare models:

• Total parameters hint at the breadth of learned capacity.
• Active parameters are closer to the cost of a single token pass.
• Expert count and top-k routing explain how sparse the architecture really is.
• Communication design helps determine whether that sparsity pays off in real hardware environments.

So when a model advertises trillions of parameters, the right question is not only