1. Long-Horizon Coherence

Question: As rollout horizon grows, do predictions remain usable?

Signature failure: Compounding error. Small per-step deviations ($\epsilon$ per step) become $H\epsilon$ total error after $H$ steps, pushing trajectories into impossible regions.

How to measure:

• Plot task success rate vs. horizon
• Look for graceful degradation (success drops smoothly) vs. cliff (success drops suddenly)
• Example: Does a robot successfully grasp objects in 5-step rollouts? 10 steps? 50 steps?

Diagnostic findings:

• Dreamer-based models typically remain coherent out to 50-100 steps for robotic manipulation
• Video generation models (Sora, Genie) struggle beyond 10-20 seconds (severe compounding error)
• Code reasoning (SWE-bench) requires coherence over hundreds of steps when fixing multi-file bugs

2. Intervention Sensitivity

Question: Does changing the action sequence produce meaningfully different trajectories?

Signature failure: Controllability failure. Model outputs the same trajectory regardless of action, making it useless for planning.

How to measure:

• Counterfactual divergence: From same initial state, execute two different action sequences; measure how much resulting trajectories differ
• Action sensitivity ratio: What fraction of action perturbations produce a detectable outcome change?

Example:

• In web automation: Inject a pop-up interrupt; does the agent replan or continue clicking blindly?
• In dialogue: Change one agent's opening move; does negotiation outcome shift?
• In robotics: Perturb object placement; does manipulation strategy adapt?

Current gap: Most benchmarks measure output quality (success rate, fidelity) but don't explicitly test action sensitivity. Closing this gap requires new evaluation protocols.

3. Constraint Consistency

Question: Do rollouts satisfy the governing laws throughout the entire trajectory?

Why this matters: Violations are often invisible per-step but catastrophic for planning.

Examples:

• Physical: Object trajectories violate gravity or penetrate obstacles → imagined success is impossible
• Digital: Browser predicts page loads, but actual API contract would fail (type mismatch, null return)
• Social: Model predicts negotiation success assuming user is price-sensitive, but user is actually quality-frustrated → plan fails
• Scientific: Predicted phase doesn't satisfy thermodynamic stability constraints → synthesis fails

How to measure:

• Physics: Check penetration depth, energy conservation, support-relation consistency
• Code: Verify type-constraint satisfaction, API receipt matching, exception handling
• Social: Detect norm violations, commitment consistency, Theory of Mind accuracy
• Science: Validate conservation law satisfaction, causal ordering, evidence-chain validity

Core Contribution 3: Unified Evaluation Framework

Beyond Prediction-Centric Evaluation

Traditional metrics focus on prediction accuracy: "Does the model predict the next frame well?"

But the paper argues this misses the point. A model with perfect next-frame prediction might fail at planning because:

• It doesn't compose coherently over many steps
• It's insensitive to action changes
• It violates domain constraints

The alternative: Decision-centric evaluation. Ask: "Does the model enable good decisions for downstream agents?"

The Minimal Reproducible Evaluation Package (MREP)

The paper proposes a lightweight evaluation protocol with three tiers:

Tier 1: Basic Capability Check

• Does the model make predictions at all?
• Does it respect the correct input/output shapes?
• Does it run without crashing?

Tier 2: Boundary Condition Verification

• Long-horizon coherence: Plot success vs. horizon curve
• Intervention sensitivity: Run action perturbation tests
• Constraint consistency: Check domain-specific violations

Tier 3: Decision-Centric Performance

• Can the model improve downstream agent performance?
• Does fine-tuning on agent-relevant regions help more than improving overall prediction accuracy?
• What's the sample efficiency gain from using the model vs. pure environment interaction?

Benchmark Coverage Gaps

The paper catalogs existing benchmarks and identifies major gaps:

Well-covered:

• Physical robotics (RoboCasa, ManiSkill3, MetaWorld)
• Some video generation (VBench for Sora)
• Code agents (SWE-bench)
• Embodied AI (Minecraft, Crafter)

Under-evaluated:

• Social simulation (only Sotopia; needs more domains)
• Scientific discovery (few benchmarks beyond climate/drug discovery)
• Cross-regime transfer (when does knowledge from one regime help in another?)
• Safety and calibration under distribution shift

Architecture and Implementation Guidance

Building Blocks Across Regimes

The paper identifies common architectural patterns:

State Representation:

• Bottleneck architectures (learned latent codes): Compress observations to low-dim codes, predict codes, decode back to observations
• Hierarchical representations: Different levels of abstraction for different time scales (immediate pixel changes vs. object trajectories vs. goals)
• Modular representations: Separate channels for position, velocity, appearance, lighting

Dynamics Model:

• Autoregressive: Predict each future step conditioned on previous predictions (classic but suffers compounding error)
• Non-autoregressive: Predict full trajectory at once (faster but harder to condition on actions)
• Latent dynamics: Predict in learned latent space (can be more stable)

Action Conditioning:

• Concatenation: Append action to state before prediction
• Multiplicative gating: Learned interaction between state and action
• Hierarchical planning: Abstract high-level actions into low-level dynamics

Design Tradeoffs by Regime

Physical World

• Favor: Explicit physics priors (Lagrangian mechanics, contact constraints)
• Avoid: Pure learning from pixels (unless data abundant); insufficient for long-horizon planning
• Sweet spot: Hybrid—learn what physics doesn't capture (material properties, deformations) while enforcing conservation laws

Digital World

• Favor: Symbolic execution (compose known API behaviors); constraint solvers
• Avoid: Pure neural prediction (APIs are discrete and deterministic; neural models are brittle)
• Sweet spot: Neural models for understanding (parsing intent, inferring unobserved state) + symbolic engines for composition

Social World

• Favor: Language models for dialogue generation; explicit Theory of Mind models
• Avoid: Purely behavioral imitation (loses interpretability of agent models)
• Sweet spot: LLM-based rollout with learned social belief updating

Scientific World

• Favor: Physics-informed neural networks (PINN), operator learning (DeepONet), Bayesian surrogate models
• Avoid: Pure black-box learning (need interpretability and uncertainty quantification for hypothesis-driven experiments)
• Sweet spot: Surrogate models with uncertainty + active learning for new experiments

Failure Modes and Limitations

Beyond the boundary-condition failures (compounding error, controllability, constraint violation), the paper identifies broader challenges:

L1 Failures

• Mode averaging: Multiple plausible futures collapse into blurry average (partially addressed by VAEs, diffusion models)
• Stochasticity: True randomness hard to capture in deterministic neural models
• Long-tail events: Rare scenarios poorly represented in training data

L2 Failures

• Distribution shift: Model works on training regime but fails on slight variations
• Exploitation: Agent finds "cheats" that work in simulation but violate constraints (e.g., walking through walls, using impossible API calls)
• Insufficient compositionality: Single predictors don't combine smoothly; joint training required

L3 Failures

• Attribution ambiguity: Which component of the model failed? (friction? contact model? object representation?)
• Overcorrection: Updating model to fix one failure case creates new failures elsewhere
• Feedback loops: If model guides agent exploration, data becomes biased; agent avoids regions model is uncertain about

State-of-the-Art Systems

By Application Domain

Robotics: MuZero → Dreamer → LEXA

• MuZero learns abstract dynamics for value estimation
• Dreamer adds visual fidelity + RL from imagination
• LEXA adds long-horizon exploration guided by learned models

Code/Web Agents: TextRL → SWE-agent → OAC

• Early: Script-based simulators (limited to Bash, Python)
• Current: LLM-based trajectory sampling (more general but less constraint-aware)
• Next: Hybrid symbolic + neural for constraint satisfaction

Video Generation: Variational Video Autoencoders → Video Diffusion → Sora/Genie

• VAV: Learned latent dynamics (precise but low fidelity)
• Diffusion: High fidelity but slower inference, less action-conditioned
• Sora: Multimodal training (video + text), 1-2 minute generation

Scientific Discovery: Traditional Bayesian optimization → Neural surrogates → Active learning loops

• Bayesian: Principled uncertainty, expensive
• Neural: Fast inference, calibration challenging
• Active learning: Combines both for sample efficiency

Open Problems and Research Directions

Fundamental Challenges

1. Cross-regime transfer: Can a world model trained on one regime (e.g., physics) help in another (e.g., social)?

   • Tentative answer: Possibly, if learning hierarchical abstractions

2. Constraint generalization: How do models learn that constraints hold across domains they haven't seen?

   • Challenge: Physics holds everywhere, but social norms don't; models need to recognize this

3. Closed-loop L3 design: How do you design agents that safely revise their own models?

   • Requires: Interpretability, anomaly detection, version control for learned models, regression testing

4. Scalability: Current video generation (Sora) works for ~1 min; can we scale to hours?

   • Bottleneck: Compounding error, compute scaling, attention mechanisms for long sequences

Architectural Directions

1. Compositional learning: Can we build world models from modular pieces (object detectors, interaction rules) that compose reliably?

2. Uncertainty quantification: Current models give point predictions; better uncertainty estimates could reduce exploration waste and enable better planning

3. Adaptive latent spaces: Can models dynamically expand their state representation when encountering novel concepts?

4. Neuro-symbolic integration: Deep learning for perception + symbolic reasoning for constraint satisfaction

Reproducibility and Implementation Notes

Data Requirements

• Physical: Video + action annotations (millions of frames)
  • Example: Robotic manipulation datasets (RoboNet: 15M+ video clips)
• Digital: Browser traces + API logs
  • Example: OSWorld (912 tasks), macOSWorld
• Social: Dialogue corpora + metadata (speaker relationships, outcomes)
  • Example: Sotopia scenarios
• Scientific: Experimental logs + measurements
  • Example: Benchmark datasets from literature

Typical Training Procedure

1
2
3
4
5
6
7
8
9
10
11

1. Collect trajectory data D = {(s_t, a_t, s_{t+1})}
2. Train L1 predictor:
   - Loss: E[(s_{t+1} - f_θ(s_t, a_t))²] + KL divergence (for uncertainty)
   - Validate: Next-frame accuracy, distribution drift
3. Scale to L2:
   - Compose predictions over horizon H
   - Validate: Constraint consistency, action sensitivity
4. Deploy with closed-loop improvement (L3 potential):
   - Log environment vs. predicted divergences
   - Analyze failure patterns
   - Update model incrementally

Computational Cost

• Training L1: GPU-weeks for visual models (depends on data scale)
• Inference: Real-time for robotics (∼10ms per step), interactive for code/web (100s ms for multi-step reasoning)
• L3 updating: Continuous background process (efficient retraining on new examples)

Verdict and Impact

Strengths

1. Conceptual unification: The levels × laws framework aligns fragmented communities
2. Comprehensive scope: 400+ papers synthesized with clear organization
3. Practical guidance: Implementation roadmaps for each regime
4. Honest assessment: Open problems clearly stated; no false consensus

Limitations

1. Framework maturity: L3 exists mostly in theory; few deployed systems
2. Benchmark gaps: Evaluation infrastructure incomplete across regimes
3. Generalization unclear: How do insights from robotics transfer to code? To science?

Who Should Read This?

• Researchers building world models (RL, vision, agents) → essential unification framework
• ML engineers deploying agentic systems → architectural guidance and failure mode catalogue
• Science administrators → roadmap for AI-driven discovery
• Policy makers → understanding agent capabilities and limitations

Future Impact

This paper may become the standard taxonomy for world models across AI—similar to how transformer papers unified NLP architectures. The levels × laws framework provides the conceptual foundation for:

• Comparing progress across domains
• Identifying and plugging research gaps
• Building safer, more interpretable agents that revise their own models

The move from L1 → L2 → L3 reflects an implicit progression: from passive prediction to active simulation to autonomous adaptation. L3 remains largely open; papers that crack reliable L3 systems (robotics with online model updating, AI-driven science with closed-loop discovery) will define the next era of agentic AI.

Key Takeaways

1. World models are not one thing: The same term applies to different capabilities (L1/L2/L3) and constraints (physical/digital/social/scientific)

2. Capability levels matter more than prediction accuracy: A model that perfectly predicts next frames but can't compose or respond to actions is useless for planning

3. Domain laws are non-negotiable: Constraint violations (penetrations, type errors, norm breaches, causal inversions) make simulated plans irrealizable

4. Evaluation must be decision-centric: Judge models by whether they improve downstream agent performance, not by prediction loss alone

5. L3 is the frontier: Moving from L1/L2 (passive) to L3 (adaptive) requires solving interpretability, anomaly detection, and safe model revision—open challenges with major implications for AI safety

6. Cross-regime insights exist: Robotics teaches us about compounding error; code teaches us about constraint checking; science teaches us about uncertainty quantification

Extended Resources

• Homepage: https://agentic-world-modeling.xyz
• GitHub: https://github.com/matrix-agent/awesome-agentic-world-modeling
• Citation: Chu et al., "Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond," arXiv:2604.22748, 2026

1. Executive Summary

OGER (Offline-Guided Exploration Reward) introduces a novel framework for enhancing Large Language Model (LLM) reasoning by seamlessly integrating offline teacher trajectories with online reinforcement learning. The key innovation lies in positioning offline data as a semantic reference point for computing auxiliary exploration rewards, rather than treating it as additional training samples.

The framework addresses critical limitations in current RLVR (Reinforcement Learning with Verifiable Rewards) approaches: the "echo chamber" effect where models converge to dominant pre-existing distributions, and entropy collapse that prevents novel solution discovery. By computing divergence-based exploration rewards and refining them through entropy-aware modulation, OGER achieves 4-7.9% improvements across mathematical and general reasoning benchmarks.

1. Mixture of Experts (MoE)

• Each token routes to a small subset of expert networks (e.g., top-k routing)
• Only the routed experts are computed; others are silent
• Enables scaling to trillion-parameter models without proportional compute growth

2. Expert Parallelism (EP)

• Distribute experts across devices: each device holds a disjoint subset
• Token load per device depends on router decisions, which vary per micro-batch
• Uneven load = some GPUs idle while waiting for others

3. Hardware: NVLink & Hopper GPUs

• NVLink 4.0: 900 GB/s bidirectional bandwidth between intra-node GPUs
• Hopper GPU: includes a dedicated Copy Engine (DMA-like unit)
• Copy Engine operates independently from SMs—zero compute interference
• Key capability: can move data while compute kernels run

4. Pipeline Parallelism (PP) & Synchronous Training

• Multiple pipeline stages, all synchronized by slowest device
• Imbalance directly impacts wall-clock time

5. Grouped GEMM

• Specialized batched matrix multiply for MoE dispatch
• Sensitive to batch size distribution; splitting tokens can degrade perf

The Problem: Load Imbalance in MoE

Root Cause

In expert parallelism, the router is learned end-to-end without constraints. It assigns tokens based on learned affinities to experts, not fairness. Result: per-device token counts vary randomly across micro-batches, even during stable training. This variation is data-dependent—different training corpora produce different routing distributions.

Why This Matters

Figure 1(b) from the paper shows the waste:

• Token straggler: The slowest device gets more tokens than average.
• GEMM straggler: Wall-clock time difference between slowest and average device.
• Quantified waste: 18.6% of GPU time per MoE layer is lost to synchronization overhead.

At scale (128 experts, up to 16 H100s), this is enormous—hours of wasted compute per day.

Why Prior Work Fails

Three main approaches tried to fix this:

1. Coarse-grained mitigation (auxiliary losses)

   • Force the router to produce balanced assignments
   • Constraints degrade model quality and expressiveness
   • Doesn't fully eliminate imbalance

2. Dynamic scheduling with overhead (FasterMoE, Tutel, SmartMoE)

   • FasterMoE: replicate "hot" experts (shadow experts) and pipeline dispatch
   • Problem: splitting communication into stages adds volume, not just reduces latency
   • When routers change unpredictably, prediction degrades
   • Tutel: switch between parallel modes, but partition weights → extra communication

3. SM-based communication overlap (Triton Distributed)

   • Fuse computation and communication kernels
   • Problem: kernels consume SM resources, reducing available compute
   • Reduces efficiency instead of improving it

The deeper issue: Specialized MoE backends (DeepEP, FUSCO) do bulk transfers without staged delivery. You can't split their communication into fine-grained pipelined stages without paying extra volume penalty.

The Solution: Two-Phase Dispatch with NVLink CE

Design Principle: Orthogonal Parallelism

FEPLB's central insight is resource-level separation:

• EP & PP use: RDMA NICs (inter-node) + GPU SMs (compute)
• FEPLB uses: NVLink Copy Engine (intra-node, no SMs) + CPU scheduler

Because these resource sets don't overlap, FEPLB is a true new parallel dimension, not a scheduling trick. It coexists with existing parallelism without reconfiguration.

Two-Phase Dispatch

Phase 1 (EP routing):

• Tokens route to their assigned devices via standard EP backend (e.g., DeepEP)
• Static experts process normally on assigned devices
• Dynamic expert tokens are collected into the NVLink domain (local node) for rebalancing
• This phase uses standard EP communication (~50 GB/s over NVLink inter-node links)

Phase 2 (NVLink CE rebalancing):

• CPU load balancer (running on dedicated thread) analyzes actual token distribution
• Decides which dynamic expert weights to copy and where
• NVLink Copy Engine redistributes both tokens and expert weights intra-node only
• Operates at 900 GB/s, completely SM-free (no compute interference)
• Happens concurrently with static expert computation on SMs

The Timeline Trick

Static experts serve dual purpose:

1. Contribute to model output
2. Provide a time window (their computation) during which CPU scheduler and weight copying finish

This window is usually sufficient—CPU load balancer runs in ~50 µs on a single core, well hidden.

Load Balancing Algorithm

Greedy, per-micro-batch:

• Repeatedly select busiest dynamic expert on most overloaded device
• Copy entire expert weights (not token-level splitting) to most underloaded device
• Threshold τ prevents copying experts with too few tokens
• Migrating whole experts preserves Grouped GEMM efficiency (batch size sensitivity)
• Fully deterministic: same routing → same copy plan across all devices (no coordination needed)

Memory Overhead

Minimal:

• Allocates buffer for max_num_dyn × expert weights per device
• For GLM-5 (72 MiB per expert, dyn=8): 576 MiB per device
• <0.7% of 80 GB HBM3—negligible

Key Results: From 18.6% Waste to 51-70% Improvement

Experimental Setup

• Hardware: Up to 16 NVIDIA H100 SXM5 GPUs (80 GB HBM3), NVLink 4.0 (900 GB/s)
• Model: Reduced GLM-5 (18 layers, preserving MoE architecture: 128 routed experts, top-k routing, no auxiliary loss)
• Configurations: Three PP/EP settings:
  • PP=4, EP=2 (8 GPUs, 64 experts per device)
  • PP=4, EP=4 (16 GPUs, 32 experts per device)
  • PP=2, EP=8 (16 GPUs, 16 experts per device)

Performance Metrics

Two stragglers directly measure load imbalance:

1. Token straggler: max(tokens_per_device) - mean(tokens)
   • Measures excess token count on slowest device
2. GEMM straggler: max(GEMM_time) - mean(GEMM_time)
   • Measures wall-clock wasted time in compute

Per-Layer Execution Time (Table 2)

Key observations:

• Triton Distributed is 1.6-3.3× slower (fused kernels consume SMs)
• Tutel adds 15-16% backward overhead (weight partitioning)
• FEPLB consistently matches or outperforms all baselines
• At high EP (2/8), FEPLB achieves strongest speedup

Load Balance Quality (Figures 5 & 6)

Token straggler reduction:

• PP=4, EP=2: 51% reduction (FEPLB) vs 55% (FasterMoE)
• PP=4, EP=4: 63% reduction (FEPLB) vs 47% (FasterMoE)
• PP=2, EP=8: 70% reduction (FEPLB) vs 39% (FasterMoE)

GEMM straggler reduction:

• PP=4, EP=2: 50% (FEPLB)
• PP=4, EP=4: 62% (FEPLB)
• PP=2, EP=8: 68% (FEPLB)

Critical insight: FasterMoE's advantage decreases with EP degree because prediction accuracy degrades under sparse, unpredictable routing. FEPLB's reactive approach improves—at EP=8, achieves 2× lower token straggler than FasterMoE.

Orthogonality Verification: EP Communication Overhead (Figure 4)

Does FEPLB interfere with EP communication?

• Before LB: baseline (100%)
• FEPLB: <1% overhead ✓
• FasterMoE pipe=1: ~0% (matches baseline)
• FasterMoE pipe=2: +46.8% dispatch, +40.2% combine (breaks orthogonality!)

This validates the paper's claim: staged pipelining on DeepEP adds volume. FEPLB avoids this by operating on separate hardware path.

Sensitivity to Dynamic Expert Count (Figure 6)

Parameter dyn controls how many experts per device are eligible for rebalancing.

• dyn=2: substantial reduction already
• dyn=2→4: +1-3 pp improvement
• dyn=4→8: +1-3 pp more (diminishing returns)
• Practical default: dyn=4

Technical Deep Dive: Why This Works

1. Hardware Resource Separation

FEPLB's elegance is achieving orthogonality by construction:

No overlap = no interference. FEPLB doesn't compete with EP for NICs or with PP/EP for compute.

2. Why Whole-Expert Migration?

You might ask: why copy entire experts instead of moving individual tokens?

Answer: Grouped GEMM is sensitive to per-expert batch size. Under the roofline model:

• Small batch → memory-bound, lower flops/cycle
• Splitting tokens into smaller batches degrades efficiency
• Migrating whole experts preserves batch size → maintains high efficiency

Trade-off: coarser granularity at low EP (e.g., 64 experts per device), but still wins overall.

3. Deterministic Load Balancing

The greedy algorithm is run independently on each device:

• Input: routing decisions (same on all devices)
• Output: same weight copy plan (no inter-device coordination)

This is crucial—avoids synchronization barriers during the critical micro-batch path.

4. Scaling with EP Degree

FasterMoE assumes stable, predictable routing. But at high EP:

• Each device sees sparser token distribution
• Router behavior becomes less predictable per device
• Prediction-based approach degrades

FEPLB is reactive, not predictive:

• Observes actual token distribution each micro-batch
• Adapts immediately
• No prediction, no degradation
• Improves with more EP degrees

Related Work & Comparisons

MoE Frameworks

• GShard, Switch Transformer: First large-scale MoE models; used auxiliary losses
• DeepSeek-V3, GLM-5: Trillion-parameter MoE systems; no auxiliary loss, live with imbalance
• Megatron-LM, DeepSpeed-MoE: Industry infrastructure

Dynamic Scheduling

1. Tutel: Adaptive EP/DP switching

   • Pro: simple
   • Con: partitions weights, adds communication

2. FasterMoE: Shadow experts + pipelined dispatch

   • Pro: good at low EP
   • Con: degrades at high EP, prediction-based

3. SmartMoE: Pre-computed strategy menu

   • Pro: offline planning
   • Con: not reactive, less flexible

4. Triton Distributed: TP-parallel MoE with fused kernels

   • Pro: explores communication-compute overlap
   • Con: consumes SMs, reduces efficiency

Communication Libraries

• DeepEP, FUSCO: Specialized bulk transfer
  • Assumption: don't split communication into stages (adds volume, not reduces latency)
  • FEPLB is compatible with both

Deployment Guide: How to Use FEPLB

1. Hardware Requirements

• NVIDIA Hopper GPUs (H100, H200) with NVLink 4.0
• Intra-node connectivity (all-to-all NVLink)
• Supporting frameworks: Megatron-LM with DeepEP dispatch

2. Configuration

Essential parameters:

1
2
3

dyn = 4  # default: 4 dynamic experts per device
τ = 0    # minimum token threshold (0 = no threshold)
max_num_dyn = 8  # max dynamic experts per device

Set these once, they're stable across runs.

3. Integration Points

• Router predictor: Periodically optimize expert-to-device assignment (at checkpoints)
• Per-micro-batch dispatch:
  • Phase 1: Standard EP routing (unmodified)
  • Phase 2: CPU scheduler analyzes tokens, issues NVLink CE copy commands
  • Combine: Return results to source devices

4. Implementation Checklist

• [ ] Baseline: measure 18.6% waste on your model
• [ ] Add CPU load balancer thread
• [ ] Integrate NVLink CE copy commands (CUDA streams)
• [ ] Tune dyn parameter for your EP degree
• [ ] Verify EP communication overhead <1%
• [ ] Validate load balance improvement (50%+ reduction expected)

5. No Auxiliary Loss Required

Major advantage: FEPLB works without auxiliary balancing losses. This means:

• Better model quality (router isn't constrained)
• Simpler training code
• Works with any existing MoE architecture

Open Questions & Limitations

1. Coarse Granularity at Low EP

• Whole-expert migration limits rebalancing flexibility when each device has many experts
• At EP=2 (64 experts per device): improvement is smaller (~51% vs ~70% at EP=8)
• Mitigated by tuning dyn, but fundamental trade-off remains

2. Scope Limited by NVLink Topology

• Current implementation: intra-node rebalancing only
• Future: With all-to-all NVLink (e.g., NVIDIA SuperPod GB200 NVL72), Phase 2 could rebalance across 72 GPUs
• Question: How much further can improvements go?

3. Router Predictor Adaptation

• Paper mentions a "Router Predictor" for periodic expert-to-device assignment
• Limited details on how this adapts to changing routing patterns
• How often does it need to run? What's the overhead?

4. Interaction with Other Optimizations

• How does FEPLB interact with speculative decoding, KV cache optimizations, or other recent MoE improvements?
• Can improvements stack?

5. Generalization Beyond Top-k Routing

• Paper focuses on top-k routing with learned router
• What about other routing schemes (random, expert choice, etc.)?
• Intuition: FEPLB should work, but not explicitly validated

6. Cross-Node Imbalance

• FEPLB addresses intra-node imbalance only
• What about imbalance across nodes (EP domain)?
• Is this a separate problem requiring a different approach?

Why This Matters

For LLM Training Teams

1. Instant 50-70% efficiency gain without model changes
2. No auxiliary losses → better model quality
3. Drop-in replacement for existing EP infrastructure
4. Scales with EP degree → better at larger clusters

For Chip Designers

• Copy Engine on Hopper was previously underutilized for MoE workloads
• This paper shows one valuable use case
• Suggests future hardware optimizations specifically for dynamic compute patterns

For Systems Researchers

• Demonstrates that resource-level separation is a powerful design principle
• Shows how to exploit idle hardware resources
• Provides a blueprint for future orthogonal optimizations

Reproducibility Notes

Code Availability

• Paper doesn't mention open-source release (yet)
• Implementation details sufficient for reproduction:
  • CPU load balancer: greedy algorithm, ~50 µs per micro-batch
  • NVLink CE commands: standard CUDA stream API
  • Integration: on top of Megatron-LM + DeepEP

Experiment Reproducibility

• Hardware clearly specified (H100 SXM5 + NVLink 4.0)
• Model: reduced GLM-5 (18 layers)
• Metrics: token straggler, GEMM straggler (easy to implement)
• Baselines: FasterMoE, Triton Distributed, Tutel (all publicly available or reimplemented fairly)

Concern: Paper uses a reduced-layer GLM-5 for efficiency. Results on full-scale 78-layer model would be valuable.

Final Thoughts

FEPLB is a rare systems paper: it identifies a real problem (18.6% waste), explains why prior work fails, and proposes an elegant solution that exploits overlooked hardware.

The key insight—use NVLink Copy Engine as a new parallel dimension—is simple but consequential. By maintaining orthogonality through resource-level separation, FEPLB coexists peacefully with existing parallelism and provides immediate, measurable gains.

The results are compelling:

• 51-70% token straggler reduction
• 50-68% GEMM straggler reduction
• Zero EP communication overhead
• Works at scale (16 H100s)
• No model changes required
• No auxiliary losses needed

For any team training trillion-parameter MoE models on Hopper-class hardware, FEPLB is a must-implement optimization. It's the kind of work that becomes industry standard quickly.

Publication timing is notable: April 2026, during the era of DeepSeek-V3 and GLM-5 scaling. MoE load balancing is an active area, and this solution is timely and impactful.

Implementation Details & Engineering Insights

1. CPU Load Balancer Implementation

The CPU scheduler is surprisingly simple yet effective:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

for each micro-batch:
  1. Wait for routing decisions from all GPUs
  2. For each device d:
     - Calculate per-expert token counts
     - Identify dynamic experts eligible for copy
  3. Greedy selection loop:
     while devices_unbalanced():
       - Find busiest dynamic expert on most-loaded device
       - Find least-loaded device
       - If token_count(expert) > threshold τ:
         - Issue NVLink CE copy command
         - Update bookkeeping
  4. Wait for all copies to complete
  5. Signal GPU to begin dynamic expert computation

Runtime: ~50 µs on single CPU core—completely hidden in static expert computation window (~2-5 ms).

2. Memory Management Strategy

Pre-allocation + reuse pattern:

1
2
3
4
5
6

Per-device buffer allocation:
  weight_buffer = [max_num_dyn experts] × [expert_size]
  For GLM-5: 8 × 72 MiB = 576 MiB
  
  Reuse across all MoE layers (no per-layer allocation)
  Freed only at training end

This is key—avoids allocation/deallocation overhead per layer.

3. Determinism & Reproducibility

One subtle strength: FEPLB's greedy algorithm is fully deterministic.

Given identical routing decisions, every GPU independently derives the same weight copy plan. No consensus protocol, no synchronization barriers—just local computation. This means:

• Same training run produces identical results every time
• Easy debugging
• Reproducible experiments across sites

Compare to stochastic load balancing (random expert selection)—much harder to debug.

4. Integration with Megatron-LM

Minimum changes required:

• Add CPU thread in training loop
• Hook into DeepEP's dispatch path (Phase 1 unmodified)
• Issue NVLink CE commands on GPU after routing
• Synchronize before dynamic expert kernel launch

No changes to:

• Router network
• Loss computation
• Backward pass
• Gradient aggregation

Performance Analysis: Why FEPLB Wins at Scale

Why FEPLB Scales Better with EP

FasterMoE's prediction problem (detailed):

At low EP (e.g., EP=2):

• Each device sees 64 experts
• Routing is relatively stable per-device across micro-batches
• Historical routing statistics are predictive
• FasterMoE shadow expert replication works well

At high EP (e.g., EP=8):

• Each device sees only 16 experts
• Token assignment per expert becomes sparser and noisier
• Historical statistics become less predictive
• Prediction-based expert selection fails

FEPLB's reactive advantage:

• Doesn't predict—observes actual distribution each micro-batch
• Adapts immediately to changing patterns
• Gets better at high EP because more expert diversity per node
• At EP=8: 2× advantage over FasterMoE

This is a fundamental insight: for sparse, unpredictable distributions, reactive algorithms win.

Communication Efficiency Deep Dive

Why does FEPLB's <1% EP overhead matter so much?

Typical MoE layer on 16 GPUs:

• Static expert dispatch: 100-200 µs
• Dynamic token redistribution (FEPLB Phase 2): 50-100 µs (hidden in Phase 1)
• Combine phase: 50-100 µs

Total: ~300 µs overhead added. Compare to:

• FasterMoE pipelined dispatch: +46.8% = additional 50-100 µs

In a model with 50 MoE layers, FEPLB saves:

• 50 × 50 µs = 2.5 ms per forward pass
• Per training run (1M steps): 2.5 ms × 1M = 2,500 seconds = 42 minutes saved

Failure Cases & When FEPLB Doesn't Help

1. When Load Imbalance Isn't the Bottleneck

FEPLB is orthogonal to other bottlenecks:

• Memory-bound compute: If Grouped GEMM is already memory-bound, load imbalance doesn't matter much
• Communication-bound training: If inter-node EP communication dominates, intra-node FEPLB doesn't help
• Pipeline bubble: If pipeline imbalance is worse than MoE routing imbalance, FEPLB is secondary

2. Low EP Degrees

At EP=2 (16 GPUs, 64 experts/device):

• FEPLB still wins but advantage is smaller (~51% vs ~55%)
• Coarser granularity: fewer experts to migrate
• Other optimizations might be better

3. Perfect Routing (Hypothetical)

If you had perfect router (all devices get same token count):

• FEPLB reduces to no-op (zero stragglers to fix)
• But in practice, routing is never perfectly balanced due to:
  • Learning dynamics (router isn't optimized for balance)
  • Data diversity (different batches → different routing patterns)
  • Numerical stability (no perfect tie-breaking in routing)

4. Non-Hopper Hardware

• Older GPUs lack efficient Copy Engine
• Newer but different-architecture GPUs may have different bottlenecks
• Results probably don't transfer directly

Cross-System Comparison Table

Interesting Edge Cases

Case 1: Router Collapse

What happens if the router learns to send all tokens to one expert?

FEPLB handles gracefully:

• Greedy algorithm detects this expert as busiest
• Copies its weights to all other devices (up to dyn limit)
• Routes remaining tokens across all devices
• Immediate load balancing without retraining

Compare to FasterMoE: prediction-based approach would be completely off.

Case 2: Data-Dependent Routing Changes

Training data changes → routing pattern changes.

Example: Switch from English to Chinese mid-training (e.g., in a multilingual model).

• Token affinities shift per expert
• Expert load distribution changes
• FasterMoE's historical statistics become stale
• FEPLB re-adapts within one micro-batch

This is realistic in continuous-training scenarios (e.g., online model updates).

Case 3: Micro-Batch Size Variation

What if batch size changes?

FEPLB remains effective:

• Larger batch = more tokens per expert
• Small imbalances remain
• FEPLB rebalances at new scale

Token straggler reduction likely stays similar because imbalance is typically data-dependent, not scale-dependent.

Future Research Directions

1. Token-Level vs. Expert-Level Migration

Current: whole-expert migration for Grouped GEMM efficiency.

Future: hybrid approach—split tokens for a few high-frequency experts, keep others whole. Requires:

• Grouped GEMM variant that handles variable batch sizes
• More sophisticated load balancer algorithm
• Potential for even finer-grained balancing

2. Cross-Node Balancing

Current scope: intra-node (limited by NVLink topology).

When NVIDIA releases all-to-all NVLink hardware (e.g., 72-GPU SuperPod):

• Phase 2 could rebalance across entire cluster
• Remove the constraint on bounding box
• Potentially reach near-perfect balance

Question: Would this improve beyond 70% GEMM straggler reduction?

3. Interaction with TP-MoE

Tensor parallelism within MoE experts (combination of TP + EP).

FEPLB currently focuses on EP. How to optimize TP-MoE load balance?

• May require cooperative scheduling between TP and EP dimensions
• Interesting systems research question

4. Learned Load Balancer

Rather than greedy algorithm, could train a small neural network to predict optimal copy assignments.

Trade-offs:

• More complex
• Potentially better decisions in complex regimes
• But loses determinism, reproducibility

Broader Context: Hardware Trends

This paper highlights an important trend: specialized hardware (Copy Engine) goes unused in general-purpose frameworks.

Similar examples:

• Tensor cores (before cuBLAS optimization)
• Async memory copy engines
• Hardware accelerators for collective operations

FEPLB is one of many papers that will likely unlock previously-idle hardware for better performance. This suggests:

1. For practitioners: Check your favorite hardware specs—there might be untapped resources
2. For chip designers: Document hidden capabilities; they may unlock surprising optimizations
3. For systems researchers: Hardware-software codesign is increasingly valuable

Reproducibility Roadmap

If you want to implement FEPLB:

Phase 0: Setup (1 day)

• Obtain 4+ Hopper GPUs with NVLink
• Install Megatron-LM with DeepEP
• Establish baseline metrics on your model

Phase 1: Static Framework (1 week)

• Implement CPU load balancer
• Add NVLink CE copy commands
• Verify EP communication overhead <1%

Phase 2: Dynamic Tuning (1 week)

• Tune dyn parameter for your EP configuration
• Measure token/GEMM straggler reduction
• Validate no accuracy degradation

Phase 3: Production Deployment (2 weeks)

• Integrate Router Predictor for periodic rebalancing
• Run long-duration training (100K+ steps)
• Compare wall-clock time vs. baseline

Total effort: ~4-5 weeks for experienced systems engineers.

Word count: ~5,800 | Pages (single-spaced): ~13-14

1. What This Paper Does

Core Problem

The edge of stability phenomenon, discovered by Cohen et al. (2021), presents a theoretical puzzle: when training with sufficiently large learning rates η, the largest Hessian eigenvalue λ₁ frequently exceeds the stability threshold 2/η, implying the system should diverge according to classical optimization theory. Yet empirically:

• Training loss continues to decrease
• Model generalization often improves in this regime
• The optimizer doesn't settle at a point but explores a bounded, chaotic set

Prior explanations relying on pointwise properties (Hessian trace, spectral norm) fail to capture this phenomenon because they ignore the ensemble behavior of the attractor set.

Main Contribution

The paper's central insight: characterize generalization through the geometric properties of the random attractor itself, not individual solutions.

They prove that:

1. Sharpness Dimension (SD) < ambient dimension d with high probability at EoS
2. Worst-case generalization error depends on SD, not parameter count d
3. The complete Hessian spectrum structure matters, not just the trace or largest eigenvalue
4. The attractor forms a fractal set with intrinsic dimension strictly smaller than the parameter space

This explains why overparameterized models generalize: the training dynamics naturally compress into a lower-dimensional manifold despite the high-dimensional parameter space.

SAGE: Training-Free Semantic Evidence Composition for Edge-Cloud Inference

Paper: Choi & Park, arXiv:2604.19623 (April 2026)
Focus: Efficient inference in edge-cloud hybrid systems through optimal evidence composition
Key Contribution: Demonstrates that coverage-aware patch selection outperforms importance-only methods under hard bandwidth constraints

What This Paper Does

This paper addresses a practical but underexplored problem in edge-cloud inference systems: how should the edge device select which image patches to transmit to the server when the uplink channel strictly limits the number of patches per request?

The standard approach—selecting patches by importance (attention score)—turns out to be fundamentally limited. The paper shows that this creates "coverage gaps": high-attention patches cluster in the same semantic region, wasting budget on overlapping information. SAGE proposes a simple but effective alternative that combines importance filtering with diversity-maximizing sampling, achieving 93% of the server's full-transmission accuracy while sending fewer than half the patches.

The insight is elegant: under hard budgets, every transmitted patch must count, so we should prioritize information coverage alongside importance.

Prerequisites: What You Need to Know

Edge-Cloud Hybrid Inference

In a typical edge-cloud system:

• A lightweight edge model (e.g., DeiT-Tiny) runs on resource-constrained devices
• When the edge is uncertain, it offloads to a powerful server (e.g., DeiT-Base)
• The uplink channel has hard constraints: bandwidth caps, latency deadlines, energy budgets

For image classification, this means selecting which image information to transmit is critical.

Vision Transformers and Patch Tokens

ViTs break images into discrete patch tokens (e.g., 196 patches for a 14×14 grid). This discrete structure is crucial:

• Early approaches relied on split computing: transmit entire feature maps (fixed size)
• ViTs enable selective transmission: each patch is independent, so we can choose a subset
• This transforms the problem from "compress the feature map" to "select which patches matter"

Attention-Based Importance

In prior work (Im et al., 2024), patches are ranked by the model's attention scores and the top-B are selected. This makes intuitive sense: high-attention patches are "important" to the model. However, this strategy assumes that individual patch importance translates directly to accuracy gains, which turns out not to be true under hard budgets.

Coverage and Redundancy in Deep Learning

Recent work on efficient ViTs (token pruning, token merging) has identified a key insight: importance-only selection is redundant. Methods like DivPrune and BAT show that diversity among retained tokens matters. However, this insight hasn't been applied to the communication setting, where the server has no access to discarded patches and cannot recover from redundancy.

The Problem: Why Importance Alone Fails

The Hard Budget Constraint

The paper formalizes a critical distinction: average-cost optimization vs. hard per-request budgets.

In prior work, the metric is average communication cost:

$$E[C] = \text{Pr(offload)} \times E[C | \text{offload}]$$

This can be misleading. Low average cost doesn't guarantee that individual offloaded requests fit within the uplink constraint. In their experiments, even with budget B=64 (one-third of patches), over 99% of offloaded images exceed the budget under standard attention-based selection.

Why? Because images offloaded to the server are precisely the hard ones. Their attention distributions are flat and diffuse (high entropy), so importance-based selection retains 140-150 patches before reaching reasonable importance thresholds—far exceeding practical budgets.

The hard-budget formulation forces realistic deployability: every offloaded request must satisfy B, no exceptions.

Empirical Evidence Against Importance-Only

The paper provides two compelling experiments:

Evidence 1: Individual importance doesn't predict value

• Compare SAGE's selected patches against Attention Prefix
• Patches SAGE adds have 3× lower server attention than patches SAGE drops
• Yet SAGE improves accuracy by +2-4 percentage points
• Interpretation: Value isn't individual importance; it's marginal contribution to information coverage

Evidence 2: Coverage has independent value

• Four strategies compared: Random (no info), Uniform Grid (coverage only), Attention Prefix (importance only), SAGE (importance + coverage)
• Uniform Grid (spatially uniform, no content awareness) outperforms Random by +6 pp at B=64
• SAGE consistently achieves highest accuracy by combining both

This cleanly separates importance and coverage as distinct signals.

The Solution: SAGE Method

Design Principle

Importance filtering first, then coverage maximization.

The method has two stages:

1. Prefilter by importance: Retain the top-2B patches by attention (candidates)
2. Select by diversity: Among candidates, greedily choose patches that maximize coverage

Algorithm Details

1
2
3
4
5
6
7
8
9
10
11
12

Input: Attention vector a, Patch embeddings Z, Budget B
Output: Selected set S (|S| = B)

1. Prefilter: C ← top-2B patches by attention score
2. Normalize: Z ← L2-normalize embedding vectors
3. Seed: s₁ ← argmax(a[C])  [highest attention]
4. Greedy diversity selection:
   For t = 2 to B:
     For each candidate i in C \ S:
       similarity[i] ← max(cosine_sim(z_i, z_j) for j in S)
     Select: s_t ← argmin(similarity)
5. Return S

The key insight: After prefiltering, don't use attention scores anymore. Only embedding similarity matters. This ensures all selected patches are reasonably important while maximizing their diversity.

Why This Works

Intuition: If two patches have similar embeddings, they encode similar semantic features. By selecting patches with low maximum similarity to the current set, we ensure the evidence covers diverse aspects of the image.

Computational efficiency: No training, no fine-tuning. Requires only:

• One forward pass to extract embeddings and attention
• O(B² × D) operations for greedy selection (negligible compared to inference)

Flexibility: Works with any frozen pretrained ViT; no model modifications.

Hyperparameter: Prefilter Ratio

The only hyperparameter is the prefilter size (top-kB where k is typically 2):

• k=1: Reduces to Attention Prefix (no diversity room)
• k=2: Default; provides candidates while constraining importance
• k≥3: Admits low-importance patches; noise offsets diversity gains

Their ablation shows k=2 is robust.

Experimental Setup and Results

System Configuration

Models:

• Edge: DeiT-Tiny (5.7M parameters)
• Server: DeiT-Base (86M parameters)
• Both pretrained on ImageNet-1K, frozen (no fine-tuning)
• Images produce 196 patches (14×14 grid, 16×16 patch size)

Dataset: ImageNet-1K validation (50,000 images, 1,000 classes)

Offloading: Controlled by confidence gate parameter η

• Lower η: more offloading, lower local accuracy
• Higher η: less offloading, higher local accuracy
• Under η=1.0, approximately 35.7% of images (17,829) are offloaded

Metrics:

• Offloaded accuracy: accuracy on images sent to server (direct measure of evidence quality)
• Overall accuracy: system-level accuracy (local + offloaded images)

Main Results (Table III)

Server ceiling (all 196 patches): 64.4% offloaded, 80.4% overall

Key finding: SAGE achieves 93% of the server ceiling at B=96 (just under half the patches) with +2-3 pp gains across tight budgets.

Ablation Studies

Effect of confidence gate (η):

• SAGE advantage widens as budget decreases
• At B=32, gain exceeds +2 pp across all η values
• Gain largest for hardest images (highest η), reaching +3.0 pp

Prefilter size (k ratio):

• k=2 is robust; k=1 collapses to Attention Prefix
• k≥3 admits low-importance noise
• Confirms importance filtering is essential

Where SAGE helps most:

• Partitioned offloaded images by attention entropy
• High-entropy images (flat attention): +5.7 pp gain at B=48
• Low-entropy images (concentrated attention): +2.8 pp gain
• Intuition: Hard cases with diffuse attention benefit most from coverage diversity

Spatial Coverage Quantification (Table II)

Measured coverage on 7×7 coarse grid (fraction of spatial cells with ≥1 patch):

Observation: SAGE consistently achieves broader spatial coverage, with largest gaps at tight budgets where it matters most.

Qualitative Analysis (Figure 4)

Visual inspection confirms:

• Attention Prefix: Concentrates selections around the single most salient region (red)
• SAGE: Distributes patches across complementary image areas (blue)
• Example: 53% → 78% coverage, enabling the server to "see" multiple object parts

Key Insights and Limitations

What We Learn

1. Hard budget constraints are fundamentally different. Average-cost optimization provides no deployability guarantee; individual requests can exceed the uplink limit.

2. Importance and coverage are orthogonal. Value doesn't come from individual patch importance but from marginal contribution to information diversity. This bridges computational efficiency and communication efficiency literature.

3. Coverage carries independent value. Uniform Grid (no content awareness) achieves 96% of Attention Prefix accuracy at B=64, proving spatial coverage alone is substantial.

4. Training-free methods can outperform learned baselines. No fine-tuning needed; frozen embeddings suffice.

Limitations and Boundary Conditions

Limited scope:

• Evaluated only on ImageNet-1K with DeiT models
• Unclear how well this transfers to other domains (COCO, medical imaging, etc.)
• Only Vision Transformers tested; CNN-based edge models unexplored

Scalability questions:

• Assumes relatively small patch vocabularies (196)
• Edge devices store full embeddings; memory footprint on ultra-constrained devices (IoT) not discussed
• Computational overhead of iterative FPS on-device not deeply analyzed

Optimality gap:

• SAGE is greedy; no proof of optimality
• Could covering-aware selection with learned importance outperform SAGE? (Not tested)
• The 2× prefilter ratio is fixed; adaptive ratios based on input difficulty not explored

Communication assumptions:

• Assumes all patches have equal transmission cost
• Real systems have variable overhead per patch (packet headers, compression)
• Doesn't address cases where spatial locality helps (sparse transmission)

Generalization:

• Confidence gate (η) is task-specific; unclear how sensitive SAGE is to gate tuning
• All experiments use frozen models; fine-tuned edge models might exhibit different attention patterns

Reproducibility and Practical Deployment

Code and Data Availability

The paper is from KAIST (Korea Advanced Institute of Science and Technology). Standard ImageNet-1K is public. Implementation requires:

• PyTorch with timm library (for pretrained ViTs)
• Basic numpy/scipy for embeddings and cosine similarity
• Approximately 50-100 lines of Python for the core SAGE algorithm

No learned parameters to train, so reproduction should be straightforward.

Implementation Considerations

On the edge device:

1. Load pretrained DeiT-Tiny, compute local prediction and embeddings
2. Extract attention scores from CLS token → patch attention
3. Run SAGE prefilter + greedy selection (Algorithm 1)
4. Transmit selected patches to server

Server side:

• Load full patch embeddings into the transmitted subset
• Standard ViT inference

Latency considerations:

• Edge inference (DeiT-Tiny): ~10-50ms
• Embedding extraction + SAGE selection: ~1-5ms
• Server inference: ~200-500ms (depends on network latency)
• Total overhead for offloaded requests: minimal

Real-World Deployment

Figure 9 in the paper plots operating points across different devices (Orin Nano, RPi 5) and channels (NB-IoT, LTE-M, 5G, Wi-Fi). Key deployments:

• IoT devices + NB-IoT: Tight budget (B=32), latency ~1s, 60-65% overall accuracy
• Raspberry Pi + 5G: Moderate budget (B=64), latency ~0.1s, 75% overall accuracy
• Edge GPU + Wi-Fi: Can achieve server ceiling with B≥96

The method is practical and deployable today.

Comparison to Prior Work

Attention-Based Selection (Im et al., 2024)

• Limitation: No hard budget guarantee; easy images dominate average-cost metrics
• SAGE advantage: +2-3 pp under deployable hard budgets

Token Merging (ToMe)

• Merges similar tokens; SAGE outperforms by +0.5-1.5 pp
• ToMe works within single-device context; SAGE accounts for zero-context server reception

BAT (Beyond Attentive Tokens)

• SOTA importance-diversity balance for computational pruning
• SAGE beats BAT by +0.4-3.6 pp when applied to communication setting
• Key difference: BAT optimizes diversity among selected tokens; SAGE optimizes diversity across the image

Semantic Communication Approaches

• DeepJSCC and others learn joint source-channel coding
• SAGE is training-free; no fine-tuning overhead, immediate deployment
• Trade-off: learned methods might achieve higher accuracy with model-specific optimization

Technical Depth: Embedding Diversity and Farthest-Point Sampling

Why Cosine Similarity on Embeddings?

The greedy selection (Algorithm 1) uses farthest-point sampling (FPS) on normalized patch embeddings:

$$s_t = \arg\min_{i \in C \setminus S} \max_{j \in S} \hat{z}_i^T \hat{z}_j$$

This maximizes the minimum distance to already-selected patches, ensuring coverage. The intuition:

• Patch embeddings encode semantic features (color, texture, shape)
• Low cosine similarity → different semantic content
• Greedy FPS iteratively adds the most different patches

Why greedy works: Under soft budget constraints, greedy FPS provides ~log(N) approximation ratio for the maximum spread problem. Here, with N=196 and B≤96, the gap to optimal is small.

Relation to Information Theory

While not explicitly framed as such, SAGE implicitly maximizes information coverage:

• Each patch carries information about different image regions
• Selecting maximally-diverse patches ensures we don't "repeat" information
• Under hard budget B, this is equivalent to maximizing conditional entropy given the transmission constraint

Deep Technical Analysis: Why Greedy FPS Suffices

The Farthest-Point Sampling Algorithm

Algorithm 1 uses a greedy approach to maximize coverage. At each iteration, it selects the patch that is most diverse from the already-selected set:

$$s_t = \arg\min_{i \in C \setminus S} \max_{j \in S} \cos(z_i, z_j)$$

This is the farthest-point sampling (FPS) algorithm, a classical technique in computational geometry.

Theoretical properties:

• Greedy FPS provides a ~log(N) approximation ratio for the maximum spread problem
• With N=196 patches, the gap to optimal is at most ~5-6%, negligible in practice
• Computational complexity: O(B² × D) where D is embedding dimension
• For D=768 (typical ViT) and B≤96, this is ~7M operations—marginal compared to inference (~1B operations)

Practical advantages:

• Deterministic (no random sampling needed)
• No learned parameters (no training required)
• Robust across different image types
• Parameter-free (after prefilter ratio selection)

Why Greedy Beats Exact Optimization Here

One might ask: could integer programming find a better patch set? The answer is likely yes, but:

1. Computational cost: IP solvers require exponential time; ~O(N choose B) = astronomical
2. Marginal gains: Exact solution might improve by 0.2-0.3 pp at most
3. Deployment friction: Learning an IP solver or heuristic requires domain expertise
4. Generalization: Learned optimization may overfit to ImageNet

SAGE's greedy approach achieves the 95%+ efficiency level with O(B²) cost—a sweet spot for deployment.

Case Study: When Coverage Matters Most

Analyzing High-Entropy Images

The paper shows SAGE's largest gains (+5.7 pp) come from high-entropy images at B=48. Let's understand why:

Example scenario: Image with multiple objects

• Attention Prefix selects top-48 patches by attention
• These cluster around the largest/most salient object
• The server "sees": object A in high detail, but no context about objects B, C, D
• Server inference: "This is object A" (often confident but wrong)

SAGE's approach:

• Prefilter: retain top-96 by attention (includes patches from all objects)
• FPS: iteratively spread selections across objects A, B, C, D
• Server inference: "Multiple objects present"—better grounding for decision

Quantified improvement:

• Attention Prefix achieves 32% offloaded accuracy
• SAGE achieves 38.4% offloaded accuracy
• The +6.4 pp (20% relative improvement) reflects the value of diverse evidence

This pattern holds across all vision tasks where multiple semantic regions matter.

Low-Entropy Images: Where Coverage Helps Less

In contrast, for images with concentrated attention (e.g., a centered object):

• Attention Prefix naturally concentrates selections on the relevant region
• Random blocks outside the region add noise
• SAGE's diversity gain is modest (+2.8 pp)
• But SAGE still wins: even 2-3 pp improvements are significant at scale

Practical Deployment Guide

Real-World Latency Breakdown

Figure 9 provides latency estimates across device-channel pairs. Here's the detailed breakdown:

Edge device (DeiT-Tiny inference):

• Orin Nano: 10-15 ms
• Raspberry Pi 5: 40-60 ms
• Inference + embedding extraction: +5 ms
• SAGE selection (B=48): +3 ms
• Total edge latency: 15-70 ms

Transmission (depends on channel):

• Wi-Fi: 10-20 ms for B=48 patches
• 5G: 5-10 ms
• LTE-M: 50-100 ms
• NB-IoT: 200-500 ms

Server inference (DeiT-Base):

• T4 GPU: 100-150 ms
• Multiple CPUs: 500-1000 ms

Total per-request latency:

• Wi-Fi + GPU: 130-200 ms (excellent)
• 5G + GPU: 110-170 ms (excellent)
• LTE-M + GPU: 160-270 ms (good)
• NB-IoT + CPU: 700-1700 ms (acceptable for non-real-time)

Memory footprint:

• DeiT-Tiny weights: 22 MB
• Cached embeddings (196×768): 600 KB
• Total per image: ~1 MB (manageable on IoT devices)

Deployment Checklist

Before deploying SAGE:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

✅ Availability
  - [ ] Pretrained ViT models available (DeiT, ViT, ImageNet pretrained)
  - [ ] Network bandwidth >= 50 KB/s (for B=48 at ~3 KB/patch)
  - [ ] Edge device has ≥200 MB RAM (for model + embeddings)

✅ Configuration
  - [ ] Set confidence gate η based on local accuracy tolerance
  - [ ] Choose budget B based on network SLA
  - [ ] Test on representative data (10-20 sample images)

✅ Validation
  - [ ] Verify offloaded accuracy meets target (≥50% for B=48)
  - [ ] Check end-to-end latency (should match Figure 9)
  - [ ] Monitor per-request variance (should be low with SAGE)

✅ Monitoring
  - [ ] Track offloading rate (should be stable with good confidence gate)
  - [ ] Log per-request latency distribution
  - [ ] Alert if accuracy drops >2 pp (may indicate model drift)

Comparison with State-of-the-Art Alternatives

vs. Split Computing (Traditional)

Traditional split computing partitions the model at a fixed layer, transmitting intermediate feature maps.

When split computing wins:

• CNNs on edge (ViT advantages don't apply)
• Extremely tight budgets where even selective transmission fails
• Custom model architectures without discrete tokens

When SAGE wins:

• Any ViT-based system
• Dynamic budget constraints (network conditions vary)
• Need for per-instance optimization

vs. Learned Feature Compression (JSCC)

Joint source-channel coding methods (DeepJSCC) learn end-to-end compression.

Trajectory:

• For research/offline applications: DeepJSCC likely wins on peak accuracy
• For production systems: SAGE wins on time-to-deployment and flexibility

A hybrid approach—SAGE for rapid prototyping, then learning for production optimization—is a viable path.

Limitations and Open Questions

Fundamental Limitations

1. Greedy suboptimality: FPS provides log(N) approximation. Could we do better?

   • Answer: Unlikely without exponential search
   • Practical relevance: ~5% gap is negligible for 2-3 pp accuracy improvement

2. Fixed embedding space: Assumes the frozen ViT's embeddings capture task-relevant semantics.

   • When this breaks: Task-specific data (medical imaging) where generic embeddings fail
   • Solution: Fine-tune embeddings or learn a task-specific distance metric (adds complexity)

3. Prefilter ratio (k=2) is heuristic: Why 2× and not 1.5× or 3×?

   • Answer: Empirically optimal for ImageNet
   • Concern: May not generalize to all domains

Open Research Questions

1. Cross-domain generalization: Does SAGE work equally well on:

   • Medical imaging (lower variation, higher importance on details)
   • Surveillance video (motion cues)
   • Satellite imagery (spatial patterns, less object-centric)

2. Adaptive prefiltering: Could we set k based on attention entropy?

   • Entropy high → increase k (more candidates for diversity)
   • Entropy low → decrease k (importance sufficient)

3. Learned importance weighting: Can we improve on attention-based importance with learned gates?

   • Trade-off: adds parameters, loses training-free advantage
   • Potential gain: might adapt better to specific edge models

4. Theoretical analysis: Can we prove SAGE is optimal under specific conditions?

   • E.g., if embeddings are uniformly distributed, does greedy achieve optimality?

Conclusion

SAGE elegantly solves a practical problem often glossed over: when edge devices must offload under strict bandwidth limits, how should they choose what to send? The paper's key insight—that coverage matters as much as importance—is simple but was missing from prior work.

The method is:

• Simple: Two-stage algorithm, no learned parameters
• Effective: +2-3 pp improvements over attention-based methods under tight budgets
• Deployable: Training-free, works with frozen models, minimal computational overhead
• Well-analyzed: Clear ablations, coverage quantification, and real-world latency estimates
• Generalizable: Applicable to any ViT-based edge-cloud system

For practitioners building edge-cloud inference systems, SAGE provides a principled, ready-to-deploy approach that works today. The paper makes an important contribution by formalizing the hard-budget constraint and demonstrating the value of coverage-aware evidence composition—insights that will likely influence future work in efficient collaborative inference.

References

[1] Choi, I., & Park, H. (2026). SAGE: Training-free semantic evidence composition for edge-cloud inference under hard uplink budgets. arXiv:2604.19623

[2] Im, J., et al. (2024). Attention-aware semantic communications for collaborative inference. IEEE Internet of Things Journal, 11(22).

[3] Long, S., et al. (2023). Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers. CVPR.

[4] Bolya, D., et al. (2023). Token merging: Your ViT but faster. ICLR.

[5] Ranjbar Alvar, S., et al. (2025). DivPrune: Diversity-based visual token pruning for large multimodal models. CVPR.

[6] Kang, Y., et al. (2017). Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. ASPLOS.

[7] Shao, J., & Zhang, J. (2020). BottleNet++: End-to-end feature compression for device-edge co-inference. ICC Workshops.

1. Error Propagation in Multi-Step Tasks

When a draft makes a subtle mistake, standard SD's token-level verification doesn't catch it:

1
2
3

Draft Step 1: "The sum of 3 and 4 is 7"      p_target = 0.8  ✓ Accepted
Draft Step 2: "Multiply by 2 to get 15"      p_target = 0.7  ✓ Accepted
Draft Step 3: "The answer is 15"             p_target = 0.6  ✓ Accepted

Each individual token has reasonable probability, but the chain violates arithmetic. An external reward model would catch this immediately, but SD cannot.

2. Latency & Overhead of External Verifiers

PRMs typically require:

• Separate forward pass through another model
• Memory overhead to store PRM weights
• Serialization overhead (can't parallelize PRM calls)
• 30-50% additional latency

For real-time applications (interactive AI, live coding), this defeats the purpose of speculative decoding.

3. Limited Generalization

A PRM trained on math problems doesn't work well on code reasoning. Each new task domain requires retraining or fine-tuning.

Core Contribution: SpecGuard Framework

SpecGuard proposes a radical idea: use model-internal signals for verification instead of external models.

The key insight is that a language model already encodes trustworthiness indicators:

1. Attention patterns show whether the model is paying attention to relevant context
2. Log-probabilities indicate the model's own confidence

High-Level Architecture

1
2
3
4
5
6
7
8
9

For each reasoning step i:
├─ Draft Model samples k candidates: {ŷ_i^(1), ..., ŷ_i^(k)}
├─ Self-Consistency Selector picks the most coherent candidate
├─ Ensemble Verifier checks two signals:
│  ├─ Attention-Based Grounding (ABGV): Is this grounded in input?
│  └─ Log-Probability-Based (LPBV): Is the model confident?
└─ Decision:
   ├─ If both signals strong: Accept draft (fast path)
   └─ If either signal weak: Invoke target model (accurate path)

Key Innovation: Self-Consistency Selector

Instead of accepting the first draft output, SpecGuard samples k candidates and picks the one that appears most self-consistent.

This is inspired by "self-consistency prompting"—the idea that if you sample multiple reasoning paths from an LLM and pick the most common answer, you get better accuracy.

SpecGuard applies this at inference time, not just as a sampling heuristic.

Technical Deep Dive: Verification Mechanisms

Mechanism 1: Attention-Based Grounding Verification (ABGV)

Problem it solves: Detect hallucinations—tokens that sound plausible but aren't actually connected to the input.

How it works:

1. Attention Rollout: For each output token, we compute cumulative attention weights across all layers using matrix multiplication:

   1	Rollout = A^(L) × A^(L-1) × ... × A^(1)

   This tells us: "How much influence does each input token have on this output token?"

2. Grounding Score: Sum the attention weights that point back to the original input or previously validated steps:

   1	G(y_t) = Σ_{j ∈ Input} R_{y_t}[j]

   A score of 1.0 means "this output is 100% attributed to input context." A score of 0.1 means "this output is only 10% grounded—mostly made up."

3. Step-Level Threshold: We take the minimum grounding score across all tokens in a step:

   1	G_min-step = min_t G(y_{i,t})

   This prevents a few grounded tokens from masking several hallucinating tokens.

Why this works: Genuine reasoning requires paying attention to prior context. Hallucinated content tends to have low attention to the input.

Memory optimization:

• Store only the last 3 layers' attention (sufficient for grounding quality)
• Sparsify attention weights < 0.01 (negligible impact, significant memory savings)

Mechanism 2: Log-Probability-Based Verification (LPBV)

Problem it solves: Detect low-confidence predictions that might be wrong.

How it works:

1. Log-Probability per Token: After generating each token, the model assigns a probability. We take the log:

   1	L(y_{i,t}) = log p(y_{i,t} | input, y_{i,<t})

   High log-prob (-0.5 to 0) = model is confident Low log-prob (-5.0 to -2.0) = model is uncertain

2. Step-Level Minimum: Again, we take the minimum across tokens:

   1	L_min-step = min_t L(y_{i,t})

   Even one very low-probability token indicates the model was unsure about this step.

Why this works: Erroneous or hallucinated steps often involve tokens the model generates with low confidence. The model "knows" it's making something up.

Connection to uncertainty quantification: This is similar to Bayesian uncertainty—the model's entropy over predictions indicates how uncertain it is about the answer.

Mechanism 3: Ensemble Verification & Adaptive Acceptance

Neither ABGV nor LPBV alone is sufficient. They're complementary:

• ABGV detects hallucinations (high confidence but ungrounded)
• LPBV detects uncertainty (low confidence, possibly grounded)

SpecGuard combines them with a weighted ensemble:

1
2
3

Score = β × LPBV_normalized + (1-β) × ABGV_normalized
Threshold: Score ≥ τ → Accept draft
           Score < τ → Invoke target model

The paper finds that β ≈ 0.5 (equal weighting) works best, suggesting both signals are equally important.

Concrete Example of Ensemble Decision:

Consider a reasoning step: "Therefore, we multiply both sides by 2 to get 14."

Contrast with a hallucinating step: "The answer is 42 because quantum mechanics."

The hallucinated step looks good locally (high confidence) but scores low in ensemble because it lacks grounding in the problem context. This is precisely the failure mode standard SD exhibits.

Self-Consistency Selector Algorithm

The self-consistency selector operates as follows:

1. Sample Phase: Draft model generates k candidate continuations, each starting fresh from the same context
2. Similarity Scoring: Compute pairwise semantic similarity (e.g., using embedding distances or token overlap)
3. Selection: Choose the candidate that maximizes average similarity to all other candidates
4. Rationale: The most "central" candidate is most likely to represent the true distribution

This differs from simple "temperature sampling":

• Temperature-based methods increase diversity but may sample implausible candidates
• Self-consistency selector filters implausible outliers while preserving diversity

Why this helps SD: Standard SD without sampling commits to the first draft token. If that token is implausible but high-probability (due to dataset bias), it gets locked in. The selector avoids this by comparing multiple paths.

Experimental Evaluation

Benchmarks & Setup

SpecGuard is evaluated on 4 major reasoning benchmarks:

1. MATH (500 competition math problems)

   • Requires step-by-step symbolic reasoning
   • Ground truth: final numerical answer

2. GSM8K (8,500 grade-school math problems)

   • More tractable than MATH
   • Tests arithmetic and logical consistency

3. MBPP (Mostly Basic Python Programming)

   • Code reasoning
   • Tests algorithmic thinking

4. TabMWP (Table-based math word problems)

   • Requires grounding in table context
   • Tests context attribution (perfect for ABGV)

Main Results

Key findings:

1. SpecGuard achieves 3.6% average accuracy improvement over baseline SD
2. Performance exceeds reward-guided SD while being faster (RSD incurs latency)
3. Latency improvement is consistent across domains (~11%)
4. Speedup is slightly worse than theoretical maximum (due to extra verification overhead), but practical

Ablation Studies

The paper ablates each component:

Interpretation:

• LPBV provides modest gains (confidence filtering works)
• ABGV provides larger gains (grounding is more important for reasoning)
• Together they're synergistic (better than additive)

Sensitivity Analysis

1. Number of draft samples (k):

   • k=1: Standard SD behavior
   • k=2: Marginal improvement (~0.5% accuracy gain)
   • k=4: Best trade-off (most papers use this, ~2% gain)
   • k=8: Diminishing returns (~2.2% gain, 2x computation)
   • Interpretation: After k=4, the additional samples are highly correlated with earlier ones, providing minimal new information

2. Layer subset for ABGV:

   • Last 1 layer: Insufficient (captures shallow attention only, loses ~1.2% accuracy)
   • Last 2 layers: Moderate (loses ~0.5% vs. last 3)
   • Last 3 layers: Sweet spot (Figure 3 in paper)
   • Last 6 layers: Minimal improvement (~+0.1%), higher memory (3x)
   • Interpretation: Middle layers capture semantic grounding; very deep layers (near output) are too specific to token choices

3. Acceptance threshold τ:

   • Very strict (τ=0.9): Accuracy +4.2%, speedup 1.02x (rarely invokes target)
   • Slightly strict (τ=0.7): Accuracy +3.8%, speedup 1.08x
   • Balanced (τ=0.5): Accuracy +3.6%, speedup 1.11x (paper's choice)
   • Slightly permissive (τ=0.3): Accuracy +2.1%, speedup 1.14x
   • Very permissive (τ=0.1): Accuracy +0.8%, speedup 1.15x (mostly relies on target)
   • Interpretation: Sweet spot is τ ≈ 0.5 for most tasks; can be tuned per domain

4. Weight parameter β:

   • β=0 (ABGV only): Accuracy +2.8%, speedup 1.10x
   • β=0.3 (ABGV-heavy): Accuracy +3.2%, speedup 1.11x
   • β=0.5 (balanced): Accuracy +3.6%, speedup 1.11x (paper's choice)
   • β=0.7 (LPBV-heavy): Accuracy +3.1%, speedup 1.10x
   • β=1 (LPBV only): Accuracy +2.2%, speedup 1.08x
   • Interpretation: Equal weighting works best; neither signal dominates

Practical Implications

1. Inference Cost Reduction

For typical deployed LLMs (using LLaMA 2 70B as target, 7B as draft):

Per-Token Latency Breakdown:

The ~10% latency overhead vs. standard SD (1.5ms vs. 1.3ms) comes from:

• Attention rollout computation: ~0.4ms
• Self-consistency sampling: ~0.3ms
• Ensemble scoring: ~0.2ms

But this is more than compensated by:

• 3.6% accuracy improvement (fewer rejected draft tokens)
• Better error recovery (fewer error cascades)

For a 1000-token response:

• Before: 5000ms (target model only)
• Standard SD: 1300ms (3.8x speedup)
• SpecGuard: 1500ms (3.3x speedup, but 3.6% better accuracy)
• Cost reduction: 5000ms → 1500ms (70% faster overall)
• Quality improvement: +3.6% accuracy (reasoning quality significantly up)

Real-world scenario: Math problem requiring 50 tokens of reasoning

• Target-only: 250ms + computation for verification
• SpecGuard: 75ms + better correctness (fewer downstream errors)
• User perceives: Much faster AND more reliable answers

2. Scalability Without External Models

Unlike reward-guided approaches, SpecGuard:

• Uses only the models already deployed (draft + target)
• Requires no fine-tuning or task-specific models
• Works across different reasoning domains
• Can be applied to any reasoning task without retraining

3. Memory-Efficient Verification

Attention-based verification with sparsification and layer subset selection means:

• Memory overhead: ~50-100MB (negligible compared to model weights)
• No model loading: Don't need to load additional verifier models
• Parallelizable: Can be computed during target model's verification pass

Limitations & Future Directions

Known Limitations

1. Grounding Score Limitations

   • Attention rollout is known to conflate attention with attribution (Serrano & Smith 2019)
     • Attention pattern A→B doesn't guarantee A causally influenced the decision about B
     • May reflect information flow rather than reasoning dependency
   • Some spurious correlations may register as high grounding scores
     • Example: A token about "Apple" might attend to "fruit" in the input, appearing grounded even if reasoning about the company
   • Doesn't distinguish between copying context vs. reasoning with it
     • A step that directly copies from the input gets perfect grounding even if uncreative or irrelevant
   • Mitigation in paper: Uses minimum grounding across tokens, but doesn't fully resolve this
   • Research direction: Combine with gradient-based attribution methods (integrated gradients, etc.)

2. Log-Probability Biases

   • Log-probability is heavily influenced by training data frequency
     • Common but incorrect tokens may still have high probability ("Apple is a fruit" has high prob even in company context)
   • Doesn't directly measure correctness, only confidence
     • Model can be very confident about wrong answers if trained on misleading data
   • Calibration issues across domains
     • Math problems vs. code generation have different probability distributions
   • Why it still works: Erroneous steps often involve rare tokens (backtracking, corrections), which have low probability

3. Limited to Step-Level Reasoning

   • Requires that reasoning decomposes into clear "steps" separated by line breaks
   • May not apply well to tasks with continuous reasoning (story generation, dialogue)
   • Doesn't help if the draft fails at the token level within a step
     • SpecGuard accepts/rejects entire steps, not individual tokens
   • Breaks down for tasks without clear step structure
     • Creative writing, conversation, open-ended generation

4. Parameter Tuning

   • The thresholds τ and weight β require calibration per model/domain
   • Paper doesn't provide clear guidance on how to set these
     • Just recommends τ=0.5, β=0.5 without systematic analysis
   • No meta-learning approach to automatically tune thresholds
   • Cross-domain transfer unclear
     • Can we use thresholds tuned on MATH for GSM8K? Paper doesn't say

5. Computational Overhead

   • Sampling k candidates adds overhead (though minimal)
     • k=4 means 4 draft forward passes instead of 1
     • Mitigated by using smaller draft model, but still real cost
   • Attention rollout computation is non-zero
     • Requires storing attention matrices and performing matrix multiplications
     • Memory-optimized version uses 3 layers, but still not free
   • Best speedup is lower than theoretical maximum
     • Standard SD: ~3.8x speedup possible
     • SpecGuard: ~3.3x speedup achieved (13% tax for 3.6% accuracy gain)
   • Trade-off calculation: Is 0.5ms latency overhead worth 3.6% accuracy improvement?
     • Depends on application (interactive vs. batch), user tolerance, SLA requirements

6. Generalization Concerns

   • All experiments use LLaMA 2 family (except one Qwen experiment)
   • Unclear if results generalize to other architectures (GPT, PaLM, etc.)
   • Does ABGV work for models with different attention mechanisms?
   • What about sparse attention, grouped-query attention, MLA (DeepSeek)? Not tested

Future Research Directions

1. Hybrid Approaches: Combine SpecGuard with lightweight PRMs for high-stakes tasks
2. Adaptive Thresholds: Learn τ and β from data rather than tuning manually
3. Extended Verification: Use other internal signals (gradient magnitudes, hidden state norms)
4. Cross-Model Verification: Can a different target model's attention patterns help verify draft outputs?
5. Theoretical Analysis: Formal guarantees on error propagation under SpecGuard

Reproducibility & Implementation Notes

Key Implementation Details

1. Attention Rollout Implementation

   • Use matrix multiplication with layer-wise averaging
   • Normalize to probability distribution
   • Batch process for efficiency

2. Draft Sampling Strategy

   • Sample k=4 candidates (paper shows this is optimal)
   • Use temperature T=0.7 for diversity without excessive noise
   • Select candidate with highest self-consistency score

3. Ensemble Combination

   • Normalize ABGV and LPBV to [0,1] independently
   • Weighted average with β=0.5
   • Apply sigmoid if needed for smoother thresholding

4. Integration with Production SD

   • Should work with existing SD implementations
   • Minimal changes to draft/target pipeline
   • Can be toggled on/off for A/B testing

Computational Complexity

• ABGV: O(L × H × N²) for N tokens, L layers, H heads (use sparse version: O(L × H × sN²) where s << 1)
• LPBV: O(N) (just extract log-probabilities)
• Total overhead: ~5-10% of target model inference time

Code & Resources

The authors should provide:

• Reference implementation in PyTorch
• Pre-computed ABGV statistics for standard models
• Threshold calibration scripts
• Benchmark scripts for MATH, GSM8K, MBPP

Conclusion

SpecGuard makes a compelling contribution to LLM inference efficiency by:

1. Identifying a real problem in existing SD: token-level verification doesn't work for reasoning
2. Proposing an elegant solution using model-internal signals: no external models needed
3. Demonstrating consistent improvements across multiple benchmarks and reasoning domains
4. Showing practical speedups that maintain or improve quality

The key insight—that models' own attention and confidence patterns can serve as verification signals—is intuitive yet powerful. This opens new directions for inference-time optimization without the overhead of external verifiers.

For practitioners:

• If your LLMs handle reasoning tasks (math, code, planning), SpecGuard is worth trying
• Implementation should be straightforward given standard SD infrastructure
• Expected gains: 10-15% latency reduction + 3-4% accuracy improvement

For researchers:

• The ensemble verification framework could extend beyond speculative decoding
• The self-consistency selector at inference time is a neat idea worth exploring further
• The attention-grounding insight could improve other verification tasks

References & Further Reading

1. Leviathan et al. (2023) - Original Speculative Decoding paper
2. Liao et al. (2025) - Reward-Guided Speculative Decoding (RSD)
3. Wang et al. (2023) - Self-Consistency Prompting
4. Serrano & Smith (2019) - Attention is Not Explanation (important counterpoint)
5. Lightman et al. (2023) - Process Reward Models for Verification

1. 多步任务中的错误传播

当草稿犯了微妙的错误时，标准 SD 的 token 级验证无法捕捉到它：

1
2
3

草稿第 1 步："3 和 4 的和是 7"      p_target = 0.8  ✓ 接受
草稿第 2 步："乘以 2 得到 15"      p_target = 0.7  ✓ 接受
草稿第 3 步："答案是 15"            p_target = 0.6  ✓ 接受

每个单独的 token 都有合理的概率，但这个链条违反了算术规则。外部奖励模型会立即捕捉到这一点，但 SD 不能。

2. 外部验证器的延迟和开销

PRM 通常需要：

• 通过另一个模型的单独前向传递
• 存储 PRM 权重的内存开销
• 序列化开销（无法并行化 PRM 调用）
• 额外 30-50% 的延迟

对于实时应用（交互式 AI、实时编码），这违背了推测解码的初衷。

3. 泛化能力有限

在数学问题上训练的 PRM 在代码推理上表现不佳。每个新的任务域都需要重新训练或微调。

核心贡献：SpecGuard 框架

SpecGuard 提出了一个激进的想法：使用模型内部信号进行验证，而不是外部模型。

关键洞察是，语言模型已经编码了可信度指标：

1. 注意力模式显示模型是否关注相关上下文
2. 对数概率表示模型自身的置信度

高层架构

1
2
3
4
5
6
7
8
9

对于每个推理步骤 i：
├─ 草稿模型采样 k 个候选：{ŷ_i^(1), ..., ŷ_i^(k)}
├─ 自洽性选择器挑选最连贯的候选
├─ 集成验证器检查两个信号：
│  ├─ 基于注意力的接地验证（ABGV）：这是否接地于输入？
│  └─ 基于对数概率的验证（LPBV）：模型置信吗？
└─ 决策：
   ├─ 如果两个信号都强：接受草稿（快速路径）
   └─ 如果任一信号弱：调用目标模型（准确路径）

关键创新：自洽性选择器

与接受第一个草稿输出不同，SpecGuard 采样 k 个候选并选择看起来最自洽的。

这受到"自洽性提示"的启发——即如果从 LLM 采样多个推理路径并选择最常见的答案，你会获得更好的准确性。

SpecGuard 在推理时应用这个，而不仅仅是采样启发式。

技术深入：验证机制详解

机制 1：基于注意力的接地验证（ABGV）

解决的问题： 检测幻觉——听起来似乎合理但实际上与输入无关的 token。

工作原理：

1. 注意力展开（Attention Rollout）： 对于每个输出 token，我们使用矩阵乘法计算跨所有层的累积注意力权重：

   1	展开矩阵 = A^(L) × A^(L-1) × ... × A^(1)

   这告诉我们："每个输入 token 对这个输出 token 有多大影响？"

2. 接地分数： 将指向原始输入或之前验证步骤的注意力权重相加：

   1	G(y_t) = Σ_{j ∈ 输入} R_{y_t}[j]

   分数 1.0 意味着"这个输出 100% 来自输入上下文"。 分数 0.1 意味着"这个输出只有 10% 接地——大多是编造的"。

3. 步骤级阈值： 我们取步骤中所有 token 的最小接地分数：

   1	G_最小步骤 = min_t G(y_{i,t})

   这防止了几个接地的 token 掩盖几个幻觉 token。

为什么有效： 真正的推理需要关注先前的上下文。幻觉内容往往对输入的注意力很低。

内存优化：

• 仅存储最后 3 层的注意力（足以用于接地质量）
• 稀疏化注意力权重 < 0.01（忽略可能影响，显著内存节省）

机制 2：基于对数概率的验证（LPBV）

解决的问题： 检测可能错误的低置信预测。

工作原理：

1. 每 Token 的对数概率： 生成每个 token 后，模型分配一个概率。我们取对数：

   1	L(y_{i,t}) = log p(y_{i,t} | 输入, y_{i,<t})

   高对数概率（-0.5 到 0）= 模型置信 低对数概率（-5.0 到 -2.0）= 模型不确定

2. 步骤级最小值： 同样，我们取跨 token 的最小值：

   1	L_最小步骤 = min_t L(y_{i,t})

   即使一个概率非常低的 token 也表示模型对这个步骤不确定。

为什么有效： 错误或幻觉步骤通常涉及模型以低置信度生成的 token。模型"知道"它在编造东西。

与不确定性量化的连接： 这类似于贝叶斯不确定性——模型对预测的熵表示它对答案的不确定程度。

机制 3：集成验证与自适应接受

ABGV 或 LPBV 单独都不充分。它们互为补充：

• ABGV 检测幻觉（高置信但无接地）
• LPBV 检测不确定性（低置信，可能有接地）

SpecGuard 用加权集成组合它们：

1
2
3

分数 = β × LPBV_归一化 + (1-β) × ABGV_归一化
阈值：分数 ≥ τ → 接受草稿
      分数 < τ → 调用目标模型

论文发现 β ≈ 0.5（等权重）效果最好，表明两个信号同样重要。

集成决策的具体示例：

考虑一个推理步骤："因此，我们将两边都乘以 2 得到 14。"

与幻觉步骤对比："答案是 42，因为量子力学。"

幻觉步骤在局部看起来很好（高置信），但在集成中评分很低，因为它缺乏在问题上下文中的接地。这正是标准 SD 表现出的失效模式。

自洽性选择器算法

自洽性选择器的工作流程如下：

1. 采样阶段： 草稿模型生成 k 个候选延续，每个都从相同上下文新鲜开始
2. 相似度评分： 计算成对语义相似度（例如，使用嵌入距离或 token 重叠）
3. 选择： 选择与所有其他候选平均相似度最大的候选
4. 理由： 最"中心"的候选最可能代表真实分布

这不同于简单的"温度采样"：

• 基于温度的方法增加多样性，但可能采样不合理的候选
• 自洽性选择器在保留多样性的同时过滤不合理的异常值

为什么这对 SD 有帮助： 标准 SD 无需采样提交到第一个草稿 token。如果该 token 不合理但高概率（由于数据集偏差），它就被锁定。选择器通过比较多个路径避免了这一点。

实验评估

基准与设置

SpecGuard 在 4 个主要推理基准上进行评估：

1. MATH (500 个竞赛数学问题)

   • 需要逐步符号推理
   • 基准真值：最终数值答案

2. GSM8K (8,500 个小学数学问题)

   • 比 MATH 更容易处理
   • 测试算术和逻辑一致性

3. MBPP (主要是基础 Python 编程)

   • 代码推理
   • 测试算法思维

4. TabMWP (基于表格的数学词问题)

   • 需要在表格上下文中接地
   • 测试上下文归因（ABGV 的完美测试）

主要结果

关键发现：

1. SpecGuard 相对基础 SD 实现平均 3.6% 准确度改进
2. 性能超过奖励引导 SD，同时更快（RSD 产生延迟）
3. 延迟改进在各个域一致（~11%）
4. 加速略低于理论最大值（由于额外验证开销），但实用

消融研究

论文对每个组件进行消融：

解释：

• LPBV 提供适度收益（置信度过滤有效）
• ABGV 提供更大收益（接地对推理更重要）
• 一起它们有协同效应（比相加更好）

敏感性分析

1. 草稿样本数量 (k)：

   • k=1：标准 SD 行为
   • k=2：边际改进（约 0.5% 准确度增益）
   • k=4：最佳权衡（大多数论文使用这个，约 2% 增益）
   • k=8：边际收益递减（约 2.2% 增益，但计算翻倍）
   • 解释： 在 k=4 后，额外样本与早期样本高度相关，提供最少新信息

2. ABGV 的层子集：

   • 最后 1 层：不足（仅捕获浅层注意力，准确度降低 1.2%）
   • 最后 2 层：中等（相对最后 3 层降低 0.5%）
   • 最后 3 层：甜蜜点（论文中的图 3）
   • 最后 6 层：最小改进（约 +0.1%），更高内存（3 倍）
   • 解释： 中间层捕获语义接地；非常深的层（接近输出）对 token 选择太具体

3. 接受阈值 τ：

   • 非常严格 (τ=0.9)：准确度 +4.2%，加速 1.02x（很少调用目标模型）
   • 稍严格 (τ=0.7)：准确度 +3.8%，加速 1.08x
   • 均衡 (τ=0.5)：准确度 +3.6%，加速 1.11x（论文选择）
   • 稍宽松 (τ=0.3)：准确度 +2.1%，加速 1.14x
   • 非常宽松 (τ=0.1)：准确度 +0.8%，加速 1.15x（主要依赖目标模型）
   • 解释： 甜蜜点为 τ ≈ 0.5；可按域调优

4. 权重参数 β：

   • β=0（仅 ABGV）：准确度 +2.8%，加速 1.10x
   • β=0.3（ABGV 为主）：准确度 +3.2%，加速 1.11x
   • β=0.5（均衡）：准确度 +3.6%，加速 1.11x（论文选择）
   • β=0.7（LPBV 为主）：准确度 +3.1%，加速 1.10x
   • β=1（仅 LPBV）：准确度 +2.2%，加速 1.08x
   • 解释： 等权重效果最好；无单个信号占主导

实际应用意义

1. 推理成本降低

对于典型部署的 LLM（以 LLaMA 2 70B 为目标，7B 为草稿）：

每 Token 延迟分解：

相对标准 SD 的 ~10% 延迟开销（1.5ms vs 1.3ms）来自：

• 注意力展开计算：约 0.4ms
• 自洽性采样：约 0.3ms
• 集成评分：约 0.2ms

但这充分被以下补偿：

• 3.6% 准确度改进（更少被拒绝的草稿 token）
• 更好的错误恢复（更少错误级联）

对于 1000 token 响应：

• 之前： 5000ms（仅限目标模型）
• 标准 SD： 1300ms（3.8 倍加速）
• SpecGuard： 1500ms（3.3 倍加速，但准确度高 3.6%）
• 成本减少： 5000ms → 1500ms (70% 更快 整体)
• 质量改进： +3.6% 准确度（推理质量显著提升）

真实场景： 需要 50 个 token 推理的数学问题

• 仅目标模型：250ms + 验证计算
• SpecGuard：75ms + 更好的正确性（更少下游错误）
• 用户感受：快得多而且答案更可靠

2. 无需外部模型的可扩展性

与奖励引导方法不同，SpecGuard：

• 仅使用已部署的模型（草稿 + 目标）
• 不需要微调或特定任务模型
• 跨不同推理域有效
• 可以应用于任何推理任务而无需重新训练

3. 内存高效的验证

基于注意力的验证配合稀疏化和层子集选择意味着：

• 内存开销： 约 50-100MB（相对于模型权重可忽略）
• 无需加载模型： 不需要加载额外验证器模型
• 可并行化： 可在目标模型验证阶段计算

局限性与未来方向

已知局限

1. 接地分数局限

   • 已知注意力展开将注意力和归因混淆（Serrano & Smith 2019）
     • 注意力模式 A→B 不保证 A 在因果上影响了对 B 的决定
     • 可能反映信息流而非推理依赖
   • 一些虚假相关可能注册为高接地分数
     • 示例："Apple"的 token 可能关注输入中的"fruit"，显得接地，即使在推理公司时无关
   • 无法区分复制上下文与推理它
     • 直接从输入复制的步骤获得完美接地，即使创意不足或无关

2. 对数概率偏差

   • 对数概率深受训练数据频率影响
     • 常见但不正确的 token 仍可能有高概率（即使公司背景下"Apple 是水果"也有高概率）
   • 不直接测量正确性，仅置信度
     • 模型可以对错答非常置信（如果在误导数据上训练）
   • 跨域校准问题
     • 数学问题 vs 代码生成有不同的概率分布

3. 仅限于步骤级推理

   • 需要推理分解为清晰的"步骤"（以换行分隔）
   • 可能不适用于连续推理的任务（故事生成、对话）
   • 如果草稿在步骤内 token 级失效，无法帮助
     • SpecGuard 接受/拒绝整个步骤，而非单个 token
   • 对无清晰步骤结构的任务失效
     • 创意写作、对话、开放式生成

4. 参数调优

   • 阈值 τ 和权重 β 需要按模型/域校准
   • 论文未提供如何设置这些的清晰指导
     • 仅推荐 τ=0.5、β=0.5，未进行系统分析
   • 无元学习方法自动调优阈值
   • 跨域转移不清楚
     • 我们能否使用在 MATH 上调优的阈值用于 GSM8K？论文未说明

5. 计算开销

   • 采样 k 个候选增加开销（尽管最小）
     • k=4 意味着 4 个草稿前向传递而非 1 个
     • 通过使用较小草稿模型缓解，但仍有实际成本
   • 注意力展开计算非零
     • 需要存储注意力矩阵并执行矩阵乘法
     • 内存优化版本使用 3 层，但仍非免费
   • 最佳加速低于理论最大值
     • 标准 SD：可能达到 3.8 倍加速
     • SpecGuard：实现 3.3 倍加速（相比 3.6% 准确度增益有 13% 税收）

6. 泛化问题

   • 所有实验使用 LLaMA 2 系列（除了一个 Qwen 实验）
   • 不清楚结果是否推广到其他架构（GPT、PaLM 等）
   • ABGV 对不同注意力机制的模型有效吗？
   • 稀疏注意、分组查询注意力、MLA (DeepSeek) 呢？未测试

未来研究方向

1. 混合方法： 为高风险任务组合 SpecGuard 与轻量级 PRM

   • 对于数学竞赛问题，可能值得在最终步骤上运行轻量 PRM
   • 成本：增加 ~5% 延迟，但获得额外可靠性
   • 权衡分析：何时混合胜过纯 SpecGuard？

2. 自适应阈值： 从数据学习 τ 和 β 而不是手动调优

   • 使用贝叶斯优化为每个模型/任务自动调参
   • 元学习方法：从多个任务学习超参数初始化
   • 在线调整：根据错误率动态调整 τ

3. 扩展验证： 使用其他内部信号

   • 梯度大小：大梯度可能表示模型不确定
   • 隐藏状态范数：异常高/低范数指示异常
   • 残差连接活动：跳过连接中有多少信息？
   • 熵分布：token 概率的熵如何分布？

4. 跨模型验证： 不同目标模型的注意力模式能否帮助验证草稿输出？

   • 多目标设置中的混合信号
   • 集成多个目标的验证信号
   • 草稿模型自我验证（不需要目标）

5. 理论分析： SpecGuard 下错误传播的形式保证

   • 什么情况下 ABGV + LPBV 充分？
   • 错误传播的上界与下界
   • 最优阈值的闭式解

6. 更广泛的任务覆盖：

   • 代码生成中的步骤识别
   • 创意写作的动态步骤
   • 多模态推理（涉及图像的推理）

7. 部署优化：

   • 在推理硬件上优化注意力展开（CUDA kernel）
   • 缓存展开矩阵以加快重复调用
   • 量化接地分数（int8 vs float32）

可复现性与实现细节

关键实现细节

1. 注意力展开实现

   • 使用矩阵乘法与层级平均
   • 归一化为概率分布
   • 批量处理以提高效率

2. 草稿采样策略

   • 采样 k=4 个候选（论文显示这是最优的）
   • 使用温度 T=0.7 获得多样性而无过度噪声
   • 选择自洽性分数最高的候选

3. 集成组合

   • 独立将 ABGV 和 LPBV 归一化为 [0,1]
   • 加权平均，β=0.5
   • 如需平滑阈值化可应用 sigmoid

4. 与生产 SD 集成

   • 应与现有 SD 实现兼容
   • 对草稿/目标管道的最小改变
   • 可切换用于 A/B 测试

计算复杂度

• ABGV: O(L × H × N²) 对于 N 个 token、L 层、H 头（使用稀疏版本：O(L × H × sN²) 其中 s << 1）
• LPBV: O(N)（仅提取对数概率）
• 总开销： 约占目标模型推理时间的 5-10%

代码与资源

作者应提供：

• PyTorch 中的参考实现
• 标准模型的预计算 ABGV 统计
• 阈值校准脚本
• MATH、GSM8K、MBPP 的基准脚本

总结

SpecGuard 通过以下方式对 LLM 推理效率做出了有说服力的贡献：

1. 识别真实问题：在现有 SD 中，token 级验证对推理不起作用
2. 提出优雅解决方案：使用模型内部信号进行验证，不需要外部模型
3. 展示一致改进：跨多个基准和推理域
4. 展示实际加速：维持或改进质量

关键洞察——模型自身的注意力和置信度模式可以作为验证信号——既直观又强大。这为推理时优化开辟了新方向，而无需外部验证器的开销。

对于从业者：

• 如果你的 LLM 处理推理任务（数学、代码、规划），值得尝试 SpecGuard
• 实现应该简明，给定标准 SD 基础设施
• 预期收益：10-15% 延迟减少 + 3-4% 准确度改进

对于研究人员：

• 集成验证框架可以扩展到推测解码以外
• 推理时的自洽性选择器是值得进一步探索的巧妙想法
• 注意力接地洞察可以改进其他验证任务

参考资料与深入阅读

1. Leviathan et al. (2023) - 原始推测解码论文
2. Liao et al. (2025) - 奖励引导推测解码 (RSD)
3. Wang et al. (2023) - 自洽性提示
4. Serrano & Smith (2019) - 注意力不是解释（重要反点）
5. Lightman et al. (2023) - 用于验证的过程奖励模型

1. Why this paper still matters in 2026

I think PipeDream is one of those papers that is easier to appreciate after the field has moved on.

If I explain it in one sentence, I would say:

PipeDream turned pipeline parallelism from a vague idea into a system-level recipe: profile the model, partition it automatically, keep multiple minibatches in flight, and repair the optimization semantics enough that training still converges.

That sounds modest today because pipeline parallelism is now normal vocabulary in large-model training. But in 2018, this was an important systems step.

The paper is historically important for at least four reasons.

• It clearly shows that data parallelism is not always the right default. When models become large, or when interconnects are weak relative to GPU speed, weight synchronization becomes a real bottleneck.
• It reframes pipeline parallelism as a joint scheduling and optimization problem, not just a diagram where layers are placed on different GPUs.
• It identifies the subtle but crucial issue of parameter-version mismatch between forward and backward passes. That is the kind of detail that separates a classroom concept from a production system.
• It anticipates a lot of the design space that later became standard in large-scale training stacks: stage partitioning, pipeline schedules, weight-version policies, stage replication, and runtime-managed buffer reuse.

I also think the paper is still useful for modern readers because it teaches a systems mindset that remains valid:

1. first find the actual bottleneck,
2. then pick the right parallelization dimension,
3. then ask what semantic damage the optimization introduces,
4. then engineer around that damage carefully.

That sequence is still exactly how good ML systems work today.

1. 为什么这篇论文到 2026 年仍然值得读

如果让我用一句话概括这篇论文，我会说：

PipeDream 的价值，不只是“把模型切成几段在不同 GPU 上跑”，而是把 pipeline parallelism 真正做成了一个完整训练系统：先 profile，后 partition，再 schedule，同时处理参数版本一致性问题，最后用 time-to-accuracy 来衡量系统价值。

今天大家谈大模型训练，已经很习惯使用 pipeline、tensor parallel、ZeRO、FSDP、activation checkpointing 这些术语，所以回头看 PipeDream，好像会觉得它只是早期工作之一。

但如果放回 2018 年的语境，这篇论文做了几件非常关键的事：

• 它明确说明了：数据并行不是永远正确的默认解。
• 它把 pipeline parallelism 从“概念图”推进到了可实现、可验证、可比较的系统设计。
• 它抓住了一个非常本质的问题：同一个 minibatch 的 forward 和 backward 如果看到的不是同一版参数，会不会把训练语义搞坏？
• 它让后来很多大模型训练系统里的概念变得更容易表达，比如 stage 划分、1F1B 调度、weight version、stage replication 等等。

我觉得它到今天仍然值得认真读，原因不是“它还能直接拿来训练最新 LLM”，而是它教会了我们一个很重要的系统思路：

1. 先找真正的瓶颈；
2. 再决定用哪一种并行方式；
3. 再追问这种并行方式会不会破坏训练语义；
4. 最后才是运行时与实现层面的工程落地。

这个思路今天一点都不过时。

1. Why this paper matters

If I had to explain this paper to a non-specialist in one sentence, I would say:

The paper teaches a large language model to make decent predictions from earlier layers, then uses the remaining layers as a built-in checker so that inference becomes faster without needing a second draft model.

That sounds simple, but it addresses a very real systems bottleneck.

Modern LLM inference is expensive because each generated token usually pays for the full depth of the model. If a model has 32 or 40 transformer layers, then every next token runs through essentially all of them. That is painful for three reasons:

• latency is high,
• GPU cost is high,
• memory pressure becomes a serious deployment constraint.

A lot of acceleration work tries to reduce one of these costs by quantization, sparsity, pruning, or a separate draft model. Those are useful directions. But they all come with trade-offs:

• quantization can hurt quality or require hardware-aware kernels,
• sparsity often needs special kernels to pay off,
• separate-model speculative decoding doubles some engineering complexity and increases memory footprint.

What LayerSkip tries to do is elegant in a systems sense:

1. train one model so its intermediate layers are more predictive,
2. let those early layers draft tokens,
3. let the later layers verify and correct them,
4. reuse shared computation and cache because draft and verification come from the same network.

I like this paper because it sits exactly at the boundary of model training design and serving systems design. It is not merely “here is a trick that is 3% better on one benchmark.” It is asking a deeper question:

Can we train the model so that its internal depth becomes more usable at inference time?

That is a powerful framing. Instead of treating inference optimization as something that happens only after training, the authors redesign training so that faster inference becomes natural.

The headline results justify paying attention:

• up to 2.16× speedup on CNN/DM summarization,
• up to 1.82× speedup on coding,
• 2.0× speedup on TOPv2 semantic parsing,
• and code/checkpoints are open sourced.

For an inference paper, that is already respectable. But the deeper contribution is conceptual: the paper turns one deep model into an ensemble of sub-models of different depths plus a built-in verifier.

1. 为什么这篇论文值得认真读

如果要我用一句最朴素的话概括这篇论文：

它让同一个大模型“先用前几层快速猜，再用后几层批量核对修正”，从而在不引入第二个草稿模型的情况下实现明显加速。

这件事看起来像“推理技巧”，但本质上是训练与部署的联合设计。

今天大模型推理的核心痛点是：

• 每生成一个 token，通常都要走完整网络深度；
• 自回归导致串行瓶颈，无法像训练时那样大规模并行；
• 延迟和成本都很高；
• 多模型 speculative decoding 虽然有效，但显存与工程复杂度上去了。

LayerSkip 的价值在于它不是简单“后处理加速补丁”，而是三步联动：

1. 训练阶段让模型早层更有预测能力；
2. 推理阶段允许早退层先草拟 token；
3. 用同一模型剩余深层做校验修正，并复用缓存减少额外开销。

论文给出的代表性速度收益是：

• CNN/DM 最高 2.16×；
• coding 最高 1.82×；
• TOPv2 最高 2.0×。

如果你是系统工程师，这篇论文最重要的不是“2.16×这个数字”，而是它提出了一个更有长期价值的问题：

我们能不能在训练时就把“可加速推理”写进模型能力结构里，而不是等部署时硬抠？

这是一个方向性问题。LayerSkip 给出了一个可行答案。

1. 为什么这篇论文值得认真读

如果让我用一句很直白的话来描述本文：

它不是在“再造一个更大的奖励模型”，而是在尝试把奖励模型从黑箱打分器，改造成“可分解、可检查、可调权重”的偏好评审系统。

这件事在 RLHF 里非常关键。

因为在很多对齐流水线里，真正最有“隐形权力”的组件不是 PPO 也不是 DPO，而是奖励模型：

• 它决定什么样的回答会被判定为“好”；
• 它的偏差会被后续策略优化放大；
• 一旦它错了，模型会“稳定地朝错误方向更努力”。

最典型的错误就是 冗长偏置（verbosity bias）：

• 奖励模型潜意识里更偏爱长回答；
• 策略模型学到“越长越安全”；
• 最终用户得到的不是更好答案，而是更啰嗦、更绕、甚至信息密度更低的答案。

所以本文真正的问题不是“奖励模型能不能做”。这个问题早就有答案。

它要回答的是更深一层的问题：

能不能把奖励模型做成“多维、可解释、可按场景动态调节”的结构，减少黑箱偏差和 reward hacking 风险？

我认为，这个问题抓得非常准。