FEPLB Technical Review: Nearly Free MoE Load Balancing with the NVLink Copy Engine

Paper: FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training
Authors: Shuyao Qi, Haoyuan Liu, Shizhen Zhao, Shanghai Jiao Tong University
arXiv: 2604.19654v1, 21 Apr 2026
Review draft date: 2026-04-29
Area: Mixture-of-Experts training, distributed systems, GPU communication, load balancing

1. One-Sentence Summary

FEPLB observes that Hopper GPUs have an NVLink Copy Engine that can move data between GPUs without consuming GPU SM cycles, then uses that otherwise-idle hardware to rebalance overloaded Mixture-of-Experts experts inside each NVLink node on every micro-batch.

The paper's pitch is not "we found a better all-to-all." It is more specific and more interesting:

Keep the normal expert-parallel communication path unchanged, but add a second, intra-node rebalancing path that runs on different hardware resources.

That resource separation is the main idea. If it holds, the load balancer becomes a new dynamic parallel dimension rather than another source of communication overhead.

2. What the Paper Does

Mixture-of-Experts (MoE) models scale parameter count by routing each token to only a small subset of expert networks. This is efficient when the router distributes tokens evenly. In practice, the router is learned and data-dependent, so some experts receive many more tokens than others. Under expert parallelism, experts are placed across GPUs, so uneven expert popularity becomes uneven GPU work.

In synchronous training, the step cannot finish until the slowest GPU finishes. The paper reports that for GLM-5 MoE layers with 128 routed experts, EP $=8$ , and no auxiliary load-balancing loss, this imbalance wastes 18.6% of GPU time per MoE layer on average.

FEPLB addresses this by:

keeping standard expert-parallel dispatch for inter-node traffic;
selecting a configurable subset of experts as dynamic experts;
after normal dispatch, moving dynamic experts' tokens and expert weights within the local NVLink domain;
doing that movement through the NVLink Copy Engine, not through SM-consuming kernels;
scheduling the movement on the CPU while static experts are already computing on the GPU.

The reported result is a large reduction in the straggler gap:

Token straggler: reduced by 51-70%.
GEMM straggler: reduced by 50-68%.
EP communication overhead: below measurable noise, reported as <1% for FEPLB.

The strongest setting is the high-EP case. At PP $=2$ , EP $=8$ , FEPLB reduces token straggler from 6,666 to 2,021, while FasterMoE reduces it to 4,036. That is the paper's clearest evidence that a reactive, per-micro-batch scheduler can beat predictive hot-expert replication as routing becomes sparse and volatile.

3. Prerequisites and Background

This section builds the minimum mental model needed to read the paper comfortably.

3.1 Mixture-of-Experts in Plain Terms

A dense transformer feed-forward layer applies the same feed-forward network to every token. MoE replaces that one dense feed-forward network with many expert networks. A small router chooses which expert or experts process each token.

A simplified MoE layer looks like this:

input tokens
    |
    v
router: choose top-k experts for each token
    |
    +--> token 1 -> expert 7, expert 42
    +--> token 2 -> expert 3, expert 7
    +--> token 3 -> expert 99, expert 12
    |
    v
expert computation
    |
    v
combine expert outputs back into token order

The benefit is sparse activation. A model may contain hundreds of experts, but each token only activates a few. This lets model parameter count grow faster than per-token compute.

The cost is routing irregularity. Since the router is learned, it may send many tokens to the same expert in a given micro-batch. That is a systems problem, not just a modeling detail.

3.2 Expert Parallelism

In expert parallelism (EP), experts are split across devices. For example, with 128 experts and EP $=8$ , each GPU owns 16 experts.

EP = 8, 128 experts

GPU 0: experts 0-15
GPU 1: experts 16-31
GPU 2: experts 32-47
...
GPU 7: experts 112-127

Every GPU may start with tokens that need experts owned by other GPUs. The system therefore performs a dispatch step: tokens are sent to the GPUs that own their selected experts. After expert computation, outputs are combined back.

Modern MoE training stacks often use specialized dispatch libraries such as DeepEP rather than generic all-to-all collectives. These libraries are fast because they use custom bulk-transfer protocols. A key point in this paper is that such libraries are not naturally friendly to fine-grained staged delivery, so "just pipeline the communication" is not free.

3.3 Why Load Imbalance Hurts

Let $T_d$ be the number of tokens assigned to device $d$ . A simple token imbalance metric is:

$S_T = \max_d T_d - \bar{T}$

where $\bar{T}$ is the average token count across devices. This is the token straggler used by the paper.

Likewise, let $G_d$ be the Grouped GEMM execution time on device $d$ . The paper defines the GEMM straggler as:

$S_G = \max_d G_d - \bar{G}$

These metrics matter because synchronous training waits for the maximum, not the average. If GPU 3 receives far more expert work than the others, the others finish early and idle at the synchronization point.

3.4 Grouped GEMM

Each expert contains matrix multiplications. Running many small matrix multiplications separately is inefficient, so MoE systems often use Grouped GEMM, which batches many expert GEMMs into one grouped operation.

This matters for FEPLB because the paper chooses to migrate whole experts rather than split one expert's tokens across many devices. Splitting tokens could make per-expert GEMM batches smaller and push computation into a less efficient memory-bound regime. Whole-expert movement is coarser, but it preserves GEMM efficiency.

3.5 Hopper NVLink Copy Engine

The paper relies on a Hopper-era hardware feature: the NVLink Copy Engine.

The important distinction is between:

SMs: GPU compute cores that run kernels such as GEMM.
Copy Engine: a dedicated data movement engine that can move memory without consuming SM cycles.

On the H100 SXM5 platform used in the paper, intra-node NVLink 4.0 bandwidth is reported as 900 GB/s bidirectional per GPU. FEPLB uses this path for local rebalancing while normal expert computation continues on SMs.

That is why the paper keeps saying "nearly free": not because bytes vanish, but because the bytes move on a hardware resource that would otherwise be idle during the relevant window.

4. The Problem FEPLB Targets

The core problem is dynamic, per-micro-batch load imbalance.

same model, same expert placement, different micro-batches

micro-batch A: GPU loads = [120, 118, 121, 119, 122, 117, 120, 123]
              almost balanced

micro-batch B: GPU loads = [80, 91, 165, 73, 88, 177, 70, 156]
              several GPUs wait for the hot ones

Static parallel planning assumes that tensor shapes and communication volumes are predictable. MoE breaks this assumption because routing depends on the actual data and on the learned router state.

The paper highlights two unsatisfactory options:

Force the router to be balanced. Auxiliary load-balancing losses can make routing more uniform, but the paper argues that they constrain router expressiveness and may hurt model quality.
Accept the imbalance. The result is wasted synchronous training time. In the paper's GLM-5 measurement, this is the reported 18.6% average GPU-time waste per MoE layer.

FEPLB tries a third path: preserve router freedom, then repair the resulting systems imbalance after routing decisions are known.

5. Why Prior Dynamic Scheduling Is Hard

The paper's critique of prior work is mainly about hidden overhead.

5.1 Tutel and SmartMoE

Tutel and SmartMoE switch among pre-defined parallel strategies or combine offline and online scheduling. The paper's concern is that these approaches partition or move expert weights across GPUs in ways that add communication. If the extra communication competes with the critical path, the load-balancing gain can be offset by the load-balancing cost.

5.2 FasterMoE

FasterMoE uses hot-expert replication and shadow experts. This can work when hot experts are predictable. But the paper identifies two issues:

Pipelining dispatch with compute can require staged delivery. On bulk-transfer backends such as DeepEP, staged delivery may add extra volume rather than hide latency.
Splitting work into smaller expert kernels can reduce Grouped GEMM efficiency.

In the evaluation, FasterMoE still helps, especially at low EP. But its reduction in token straggler weakens as EP grows: from 55% at EP $=2$ to 39% at EP $=8$ .

5.3 Triton Distributed

Triton Distributed fuses computation and communication for TP-parallel MoE. The problem is resource contention: fused communication uses SM resources, reducing the SM budget available for actual expert computation. In Table 2 of the paper, Triton Distributed is much slower than the baseline in these experiments.

5.4 The Common Compatibility Issue

DeepEP and FUSCO are designed for high-throughput bulk transfers. They are not described as supporting the staged delivery assumed by many overlap schemes. FEPLB's design avoids changing that inter-node path. It lets DeepEP do the normal EP dispatch, then performs local rebalancing through a separate copy-engine path.

6. FEPLB's Design Principle: Orthogonal Dynamic Parallelism

The main systems principle is resource-level separation.

Dimension	Scope in the paper	Communication resource	Compute/scheduling resource
Expert Parallelism (EP)	Cross-device / inter-node expert dispatch	RDMA / NVLink, via EP backend	GPU SMs for expert compute
Pipeline Parallelism (PP)	Cross-stage model pipeline	RDMA / NVLink	GPU SMs
FEPLB	Intra-node dynamic load balancing	NVLink Copy Engine	CPU scheduler

A rebalancer that uses the same NICs, same SMs, or same bulk-transfer path as EP is not orthogonal; it steals resources from the thing it is trying to improve. FEPLB's claim is that its rebalancer mostly uses otherwise-unused CPU and Copy Engine resources.

A useful mental model:

normal MoE training critical path:
    EP dispatch -> expert GEMM on SMs -> combine

FEPLB adds a side path:
    CPU decides copies + NVLink CE moves weights/tokens

The side path is useful only if it finishes before the GPU needs dynamic experts.

This is why FEPLB divides experts into static and dynamic sets. Static expert computation creates the time window that hides the copy-engine work.

7. Two-Phase Dispatch

Two-Phase Dispatch is the core algorithmic structure.

            
            flowchart TD
    A[Router chooses experts for each token] --> B[Phase 1: standard EP dispatch]
    B --> C1[Static expert tokens go to owning devices]
    B --> C2[Dynamic expert tokens are collected into local NVLink domain]
    C1 --> D1[Compute static experts on GPU SMs]
    C2 --> D2[CPU load balancer reads dynamic token counts]
    D2 --> E2[NVLink Copy Engine copies selected weights and tokens]
    D1 --> F[Barrier: static compute and CE copies complete]
    E2 --> F
    F --> G[Compute dynamic experts with rebalanced load]
    G --> H[Combine results back to source tokens]

7.1 Phase 1: Preserve the Existing EP Path

Phase 1 uses the normal EP backend, such as DeepEP.

Static expert tokens follow the unmodified EP route.
Dynamic expert tokens are routed to the corresponding NVLink domain.
The inter-node communication pattern is preserved.
No cross-node load balancing is attempted.

This is a conservative design. It does not ask the MoE framework to replace its high-performance dispatch layer.

7.2 Phase 2: Rebalance Inside the NVLink Domain

After Phase 1, devices inside the local NVLink domain know the token counts for dynamic experts. FEPLB then decides whether some dynamic expert should execute on a less-loaded device.

When it migrates an expert for the current micro-batch, it copies:

the dynamic expert's weights; and
the tokens assigned to that expert.

The copy happens through the NVLink Copy Engine. The expert's mathematical semantics are unchanged: each token is still processed by the same expert weights. Only the physical device executing that expert for this micro-batch changes.

7.3 Why Only Intra-Node?

The design is deliberately local. On the H100 cluster in the paper, cross-node communication goes through InfiniBand, while intra-node communication can use NVLink CE. If FEPLB tried to rebalance across nodes, it would interfere with the same inter-node resources used by EP and would lose the orthogonality argument.

The paper notes that future or different topologies, such as NVIDIA GB200 NVL72 with all-to-all NVLink across 72 GPUs, could enlarge the NVLink domain. In that case, the same idea could rebalance a much larger EP group without going through the inter-node NIC path.

8. Static and Dynamic Experts

FEPLB partitions each device's experts into two categories.

Expert type	What happens	Purpose
Static experts	Always execute on their assigned device through the normal EP path	Provide stable work and create an overlap window
Dynamic experts	Eligible to have weights/tokens copied within the NVLink domain for this micro-batch	Absorb routing imbalance reactively

The key knob is $dyn$ , the number of dynamic experts per device. The paper gives this example:

128 routed experts;
EP $=8$ ;
each device owns $128 / 8 = 16$ experts;
if $dyn = 4$ , then 4 experts are dynamic and 12 remain static per device.

A good way to see the design:

GPU d owns 16 experts

static:  12 experts -> compute locally, always
         create time window for CPU scheduling and CE copies

dynamic:  4 experts -> may be copied for this micro-batch
         used to reduce the current imbalance

This split is a trade-off. More dynamic experts give the scheduler more flexibility, but fewer static experts leave less guaranteed computation time for hiding CPU scheduling and copy-engine transfers.

The paper's sensitivity study finds diminishing returns. Moving from $dyn=2$ to $dyn=4$ improves token-straggler reduction by roughly 1-3 percentage points, and $dyn=4$ to $dyn=8$ adds another 1-3 points. The authors describe $dyn=4$ as a practical default.

9. The CPU Greedy Scheduler

FEPLB's scheduler is intentionally lightweight. It runs on a dedicated CPU thread, starts after router completion, and uses the actual per-micro-batch token counts rather than relying only on prediction.

The paper describes a greedy algorithm:

repeat:
    find the most overloaded device
    on that device, find the busiest dynamic expert
    if that expert has fewer than tau tokens:
        stop or skip it
    find the most underloaded device
    copy the whole expert's weights and its tokens to the underloaded device
    update the simulated per-device loads
until no useful migration remains or dynamic-copy budget is exhausted

The threshold $\tau$ prevents copying experts whose token count is too small to justify the movement. The paper does not provide a universal numeric $\tau$ in the extracted text, so reproductions should treat it as an implementation/configuration parameter.

9.1 Why Whole-Expert Migration?

A natural question is: why not split a hot expert's tokens across several GPUs?

FEPLB avoids token-level splitting for two reasons given by the paper:

Grouped GEMM efficiency. Smaller per-expert batches can reduce matrix multiplication efficiency.
Determinism and coordination simplicity. If the scheduler migrates whole experts, each device can independently derive the same copy plan from the same routing information, without extra coordination.

This is also the source of FEPLB's main granularity limitation. At low EP, each GPU owns many experts. Moving one whole expert changes load relatively coarsely, so FEPLB has less fine-grained control.

9.2 Scheduler Cost

The paper reports that the greedy scheduler completes in about 50 microseconds on a single CPU core, which is intended to fit within the static-expert computation window.

The important nuance: FEPLB is not claiming CPU scheduling is zero-cost. It is claiming that the CPU work is hidden behind GPU static-expert computation.

10. Timeline and Overlap Strategy

The overlap is the most important engineering trick. FEPLB needs three things to happen at once:

static experts compute on GPU SMs;
the CPU scheduler decides dynamic expert migrations;
the NVLink Copy Engine transfers selected weights and tokens.

A simplified timeline:

time --->

GPU SMs:       [Phase 1 dispatch] [static expert GEMM........] [dynamic expert GEMM] [combine]
CPU:                             [greedy load balancing plan]
NVLink CE:                         [copy weights + tokens.....]
critical path:  static work hides CPU + CE side work if side work finishes in time

The design succeeds when:

$T_{CPU\ scheduler} + T_{CE\ copies} \leq T_{static\ expert\ compute}$

This inequality is not written as a formal theorem in the paper, but it is the operational condition behind the design. If the copy plan takes longer than static expert computation, the dynamic experts must wait and FEPLB is no longer free.

11. System Architecture

FEPLB operates at two timescales.

11.1 Macro-Level Placement

At a slower timescale, a Router Predictor periodically optimizes expert-to-device assignment using historical routing statistics. The paper says this is executed at checkpoint time to spread out migration cost.

This part is not the per-micro-batch load balancer. It is closer to background placement hygiene: keep the default expert placement reasonable so the micro-level rebalancer does not start from a terrible layout.

11.2 Micro-Level Rebalancing

At every micro-batch, FEPLB reacts to the actual token distribution:

router decisions are produced;
Phase 1 dispatch sends static tokens normally and collects dynamic tokens;
static experts begin computing;
CPU scheduler computes a migration plan;
NVLink CE copies chosen dynamic weights/tokens;
dynamic experts compute on the rebalanced devices;
outputs are combined back.

            
            flowchart LR
    subgraph Slow_Path[Occasional macro path]
        RPred[Router Predictor] --> Placement[Expert placement update at checkpoint time]
    end

    subgraph Fast_Path[Every micro-batch]
        Router[Router] --> Dispatch[Phase 1 EP dispatch]
        Dispatch --> Static[Static expert compute]
        Dispatch --> Counts[Dynamic token counts]
        Counts --> CPU[CPU greedy scheduler]
        CPU --> CE[NVLink CE copies weights/tokens]
        Static --> Dyn[Dynamic expert compute]
        CE --> Dyn
        Dyn --> Combine[Combine]
    end

11.3 Memory Management

FEPLB allocates extra buffer space for copied dynamic expert weights. The paper states the overhead as:

$\text{extra buffer per device} = \text{max\_num\_dyn} \times W_{expert}$

For GLM-5, the paper gives:

expert size: 72 MiB;
$\text{max\_num\_dyn}=8$ ;
buffer size: 576 MiB per device;
less than 0.7% of an 80 GB HBM3 GPU;
reused across all MoE layers.

That is small for H100-class training, but it is still a real hardware assumption: the target system needs spare HBM for temporary expert copies.

12. Experiment Setup

The evaluation is per-MoE-layer, not full end-to-end model pretraining.

12.1 Hardware

The paper uses:

NVIDIA H100 SXM5 GPUs;
80 GB HBM3 per GPU;
NVLink 4.0 within each node;
reported 900 GB/s bidirectional bandwidth per GPU for intra-node NVLink;
400 Gbps InfiniBand between nodes.

12.2 Software

The baseline stack is:

Megatron-LM MoE framework;
DeepEP for expert dispatch;
NVIDIA Transformer Engine for mixed-precision training;
multi-stream cuBLAS-based Grouped GEMM.

FEPLB is implemented on top of this stack. The paper emphasizes that all configurations share the same communication and computation kernels, which is important for fair comparison.

12.3 Model

The model is a reduced-layer variant of GLM-5:

18 layers instead of the original 78;
original MoE layer architecture preserved;
128 routed experts;
top- $k$ routing;
no auxiliary load-balancing loss.

The authors argue that reducing the number of layers does not affect per-layer evaluation because FEPLB operates independently inside each MoE layer.

12.4 Parallel Configurations

The paper evaluates three PP/EP configurations:

PP	EP	GPUs	Experts per device
4	2	8	64
4	4	16	32
2	8	16	16

These settings are useful because they test different granularity regimes. At EP $=2$ , each device owns many experts, so whole-expert migration is coarse. At EP $=8$ , each device owns fewer experts, and imbalance across devices becomes more severe but more addressable by moving selected dynamic experts.

12.5 Baselines

The paper compares:

Before LB: standard expert parallelism with no dynamic load balancing.
FasterMoE: shadow-expert replication. The paper re-implements it with SM-free NVLink CE transfers and DeepEP dispatch; unless noted, reported results use pipe $=1$ .
Triton Distributed: TP-parallel MoE using fused compute-communication kernels.
Tutel: adaptive switching between EP and DP modes.
FEPLB: Two-Phase Dispatch with NVLink CE rebalancing.

This is a reasonably strong comparison set because it separates three strategies: predictive replication, fused communication/compute, and adaptive parallel-mode switching.

13. Main Results

13.1 Per-Layer Execution Time

The paper reports forward/backward execution time per MoE layer in milliseconds.

PP/EP	Before LB	FasterMoE	Triton Dist.	Tutel	FEPLB
4/2	8.2 / 14.9	7.9 / 14.0	13.1 / 22.8	8.0 / 17.1	7.9 / 14.4
4/4	7.3 / 13.2	6.9 / 12.2	15.3 / 24.0	7.2 / 15.2	6.8 / 12.1
2/8	6.9 / 12.5	6.3 / 11.1	22.8 / 30.0	6.8 / 14.5	6.0 / 10.6

Key readings:

Triton Distributed is much slower in this setup, especially as collective participation grows, because communication consumes SM resources.
Tutel can match or slightly improve forward time, but it adds backward overhead.
FEPLB is strongest at PP $=2$ , EP $=8$ : forward time improves from 6.9 ms to 6.0 ms and backward from 12.5 ms to 10.6 ms.
At EP $=2$ , FEPLB is not clearly better than FasterMoE on backward time. This matches the paper's explanation that whole-expert migration is less flexible when each device owns many experts.

13.2 EP Communication Overhead

The orthogonality claim is tested by measuring dispatch and combine communication time at EP $=8$ .

The paper reports:

FasterMoE with pipe $=2$ adds up to 46.8% dispatch overhead and 40.2% combine overhead.
FEPLB introduces no measurable overhead, reported as <1%.

This is one of the most important results because it verifies the design principle. FEPLB's load-balancing path does not perturb the EP communication path in the measured setting.

13.3 Token Straggler

PP/EP	Before LB	FasterMoE	FEPLB
4/2	2,278	1,014 (-55%)	1,107 (-51%)
4/4	4,649	2,471 (-47%)	1,697 (-63%)
2/8	6,666	4,036 (-39%)	2,021 (-70%)

This table shows the crossover:

At EP $=2$ , FasterMoE is slightly better on token count.
At EP $=4$ , FEPLB becomes clearly better.
At EP $=8$ , FEPLB's token straggler is roughly half of FasterMoE's.

The paper's interpretation is that prediction becomes harder as EP grows and routing distributions become sparser. FEPLB is reactive: it sees the actual token distribution for the current micro-batch before deciding what to move.

13.4 GEMM Straggler

PP/EP	Before LB	FasterMoE	FEPLB
4/2	0.316 ms	0.170 ms (-46%)	0.157 ms (-50%)
4/4	0.652 ms	0.380 ms (-42%)	0.247 ms (-62%)
2/8	1.110 ms	0.625 ms (-44%)	0.352 ms (-68%)

The GEMM straggler is more directly connected to wall-clock compute waste. Here FEPLB beats FasterMoE in all three settings, including EP $=2$ .

The high-EP case is again the strongest:

$\frac{0.625}{0.352} \approx 1.8$

So at EP $=8$ , FEPLB has about 1.8x lower GEMM straggler than FasterMoE.

13.5 Sensitivity to Dynamic Expert Count

The paper evaluates $dyn = 2, 4, 8$ .

The trend is intuitive:

$dyn=2$ already helps because imbalance is often caused by a small number of busy experts.
$dyn=4$ gives a little more flexibility.
$dyn=8$ adds only another small improvement.

The stated gain from increasing $dyn$ is modest: around 1-3 percentage points from 2 to 4, then another 1-3 from 4 to 8. This supports $dyn=4$ as a practical default.

14. Why FEPLB Improves with EP Degree

The paper's most interesting scaling behavior is that FEPLB improves as EP increases, while FasterMoE weakens.

There are two reasons.

14.1 Prediction Gets Harder

With higher EP, each device owns fewer experts. Token routing becomes sparser per device, and hot experts can shift more abruptly across micro-batches. A predictive scheme that guesses hot experts from history may choose the wrong replication plan.

FEPLB waits until after routing for the current micro-batch. That makes it less dependent on stable expert popularity.

14.2 Whole-Expert Migration Gets Finer Relative to Device Load

At EP $=2$ , each device owns 64 experts. Moving one whole expert is a coarse adjustment relative to the device's total work.

At EP $=8$ , each device owns 16 experts. Moving one dynamic expert can make a more meaningful correction, and the imbalance itself is larger. This makes the same mechanism more valuable.

The design therefore has a natural sweet spot: large expert-parallel groups inside fast NVLink domains.

15. Correctness and Training Semantics

FEPLB preserves forward semantics because it does not change which expert a token uses. It changes only where the expert computation runs for that micro-batch.

without FEPLB:
    token x -> expert e on owner GPU

with FEPLB:
    token x -> same expert e
    expert e's weights are copied to another GPU for this micro-batch
    output is combined as if expert e ran on its owner

This is an exact semantic transformation for the forward computation, assuming the copied weights are identical.

For training, the paper reports backward execution times, so the implementation must also preserve gradient/update consistency. The extracted paper text does not fully detail the gradient accumulation protocol. A faithful implementation needs to ensure that gradients produced by temporary copied expert computation are accumulated back to the owning expert state or otherwise kept exactly consistent with the optimizer's ownership model.

This is not a reason to distrust the result, but it is an important reproducibility detail. The blog reader should not walk away thinking that copying forward weights alone is a complete training implementation.

16. Limitations and Boundary Conditions

FEPLB is clever, but its assumptions are specific.

16.1 It Requires the Right Hardware Path

The method depends on an SM-free intra-node data movement path. The paper evaluates on H100 SXM5 GPUs with NVLink 4.0 and the NVLink Copy Engine. On hardware without comparable copy-engine behavior, the "nearly free" claim may not hold.

16.2 It Is Intra-Node on Current H100 Clusters

FEPLB does not rebalance across InfiniBand-connected nodes. That is intentional. Cross-node rebalancing would likely interfere with EP communication and break orthogonality.

This means FEPLB can only fix imbalance within the NVLink domain. If the dominant imbalance is across nodes rather than within a node, the benefit may be limited.

16.3 Whole-Expert Migration Limits Granularity

Moving whole experts preserves Grouped GEMM efficiency, but it is coarse. This is why FEPLB is less dominant at EP $=2$ .

A token-splitting design might improve load balance, but it would risk smaller GEMMs, more coordination, and less deterministic scheduling.

16.4 It Needs an Overlap Window

The CPU scheduler and Copy Engine transfers are hidden behind static expert computation. If static expert computation is too short, if too many experts are dynamic, or if copied expert weights are too large, the side path may become visible on the critical path.

16.5 It Consumes Extra HBM

The reported GLM-5 overhead is small: 576 MiB per device for $\text{max\_num\_dyn}=8$ . But larger experts, smaller-memory GPUs, or other memory-heavy optimizations could make the buffer more painful.

16.6 Evaluation Scope Is Focused

The evaluation uses reduced-layer GLM-5 MoE layers and up to 16 H100 GPUs. The per-layer focus is justified for isolating the mechanism, but full training runs would be useful for checking interactions with optimizer state, activation checkpointing, pipeline bubbles, data mixtures, and long-term router drift.

16.7 Paper Does Not Provide Every Reproduction Detail

The paper gives strong system-level detail, but some exact reproduction inputs are not visible from the extracted text:

source code or release status;
exact $\tau$ value;
exact router predictor implementation details;
routing traces or data mixture;
complete backward gradient-transfer protocol;
exact top- $k$ value for the evaluated GLM-5 variant.

A reimplementation is possible in principle, but exact result reproduction would require these missing details.

17. Reproducibility Notes

A practical reproduction would need the following components.

17.1 Required Platform

H100-class GPUs with NVLink Copy Engine behavior comparable to the paper.
Multi-GPU nodes with high-bandwidth NVLink.
Inter-node fabric such as 400 Gbps InfiniBand if reproducing multi-node EP configurations.
Enough HBM to reserve temporary copied expert buffers.

17.2 Required Software Stack

Megatron-LM MoE or a comparable MoE training framework.
DeepEP or an equivalent high-performance expert dispatch backend.
Transformer Engine or equivalent mixed-precision support.
Grouped GEMM implementation, such as cuBLAS grouped/multi-stream kernels.
CUDA stream control that can issue copy-engine transfers without consuming SMs.

17.3 Implementation Checklist

[ ] Partition experts into static and dynamic sets per device.
[ ] Run normal EP dispatch for static experts.
[ ] Collect dynamic expert tokens into the local NVLink domain.
[ ] Expose per-expert token counts to the CPU scheduler.
[ ] Implement deterministic greedy whole-expert migration.
[ ] Enforce a minimum token threshold tau.
[ ] Allocate max_num_dyn * W_expert temporary weight buffers.
[ ] Use dedicated CUDA streams for CE transfers.
[ ] Overlap CPU scheduling and CE copies with static expert GEMM.
[ ] Compute dynamic experts after side-path completion.
[ ] Combine outputs back to the original token owners.
[ ] Preserve backward gradient and optimizer-state semantics.
[ ] Measure token straggler, GEMM straggler, dispatch time, and combine time.

17.4 What to Measure First

Before attempting a full model run, I would validate four small gates:

Copy/compute overlap: prove NVLink CE transfers do not slow down a concurrent Grouped GEMM.
Semantic equivalence: compare dense outputs and gradients with FEPLB disabled/enabled on a fixed seed.
Scheduler latency: confirm the CPU plan is around the paper's reported 50 microseconds or at least hidden by static computation.
Communication isolation: verify DeepEP dispatch/combine time remains within noise, ideally below the paper's <1% overhead.

Only after those gates pass does the full straggler experiment become meaningful.

18. Conceptual Diagram: Why the Design Is Different

Many load-balancing systems try to hide communication under compute, but still use compute or communication resources that are on the critical path. FEPLB instead searches for an unused lane.

Prior overlap-style idea:

GPU SMs:      compute expert work + assist/fuse communication
Network:      EP traffic + extra balancing traffic
Result:       improvement depends on whether overhead is truly hidden

FEPLB idea:

GPU SMs:      static expert GEMM, then dynamic expert GEMM
EP network:   normal dispatch/combine, unchanged
NVLink CE:    local weight/token copies
CPU:          small greedy scheduling loop
Result:       balancing path is mostly orthogonal to EP and PP

This is the main reason I find the paper compelling. The method is not just a new heuristic; it is a resource allocation argument.

19. Relation to Broader ML Systems Trends

FEPLB fits a broader pattern in ML systems: performance gains increasingly come from using specialized hardware paths correctly, not only from reducing algorithmic FLOPs.

Examples of this pattern include:

overlapping communication with computation;
exploiting asynchronous copy units;
using specialized tensor cores effectively;
designing memory layouts around hardware transfer engines;
reducing synchronization stragglers rather than only optimizing average throughput.

MoE makes this especially important because conditional computation introduces runtime irregularity. Static parallel plans are not enough when the router changes the amount of work each device receives on every micro-batch.

FEPLB's lesson is general:

Dynamic model behavior should be matched with dynamic systems behavior, but the dynamic system must use resources that are not already on the critical path.

20. Final Assessment

FEPLB is a strong systems paper because the core idea is simple, hardware-grounded, and measured against relevant baselines.

What I Like

The design principle is clean: resource-level separation from EP and PP.
The method preserves the existing EP backend rather than replacing it.
The paper reports both token-level and GEMM-time straggler metrics.
The evaluation checks communication overhead directly, which is the key failure mode for this kind of work.
The high-EP results are convincing: FEPLB gets much better relative to FasterMoE as EP grows.

What I Would Treat Carefully

The method is tightly tied to Hopper/NVLink CE behavior.
It is currently intra-node on normal H100 clusters.
Whole-expert migration trades fine-grained balance for GEMM efficiency.
Exact reproduction needs more implementation detail than the paper text provides.
Full-training interactions are not as fully explored as per-layer performance.

Bottom Line

FEPLB is best understood as a reactive, intra-node MoE load balancer that turns the NVLink Copy Engine into a separate dynamic parallel dimension. Its value is largest when expert routing is volatile, EP degree is high, and the system has enough static expert work to hide the CPU scheduling and copy-engine transfers.

For beginners reading ML systems papers, the takeaway is: the paper is not only about MoE load balancing. It is also a good example of how modern training performance depends on matching the algorithm's irregularity to the hardware's unused execution paths.