Alpa: Automating Inter- and Intra-Operator Parallelism — In-Depth Technical Review

Abstract

Training extremely large deep learning models — with billions or even hundreds of billions of parameters — on distributed GPU clusters is one of the most challenging engineering problems in modern machine learning. Today, achieving high throughput requires manually combining multiple forms of parallelism (data, tensor/operator, pipeline) in ways that are specific to both the model architecture and the cluster topology. This manual process demands deep systems expertise and does not generalize across different models or hardware setups. Alpa is a compiler system that automates this entire process. It introduces a new hierarchical view of parallelism — distinguishing between intra-operator and inter-operator parallelism — and uses a combination of Integer Linear Programming (ILP) and Dynamic Programming (DP) to automatically generate near-optimal execution plans. Evaluated on GPT-3, GShard MoE, and Wide-ResNet at up to 64 GPUs, Alpa matches or outperforms hand-tuned systems like Megatron-LM and DeepSpeed, and generalizes to models that have no manually-designed parallelization strategies at all.

1. Prerequisites: What to Know Before Reading This Paper

Before diving into Alpa's design, you need to understand several foundational concepts. This section will bring you up to speed, even if you have only a general familiarity with deep learning.

1.1 Deep Learning Computation as a Dataflow Graph

When we write a deep learning model in frameworks like PyTorch or TensorFlow, the computation is internally represented as a dataflow graph. In this graph:

Nodes represent computational operations (operators), such as matrix multiplication, convolution, element-wise addition, layer normalization, etc.
Edges represent multi-dimensional arrays called tensors that flow between operations.

Training one iteration involves three phases: (1) a forward pass that pushes a batch of data through the graph to compute a loss value, (2) a backward pass that computes gradients via backpropagation, and (3) a weight update step that adjusts model parameters using the computed gradients (typically via an optimizer like Adam or SGD).

The key insight is that everything — forward computation, backward computation, and weight updates — can be expressed as operators working on tensors. This graph-level representation is what Alpa's compiler operates on.

1.2 Why We Need Parallelism: Memory and Compute Constraints

A single GPU has limited memory (e.g., 16 GB on an NVIDIA V100) and limited compute throughput. Modern large models have parameter counts that far exceed what one device can hold:

Model	Parameters	Approximate Memory (FP16)
GPT-2	1.5B	~3 GB (params only)
GPT-3	175B	~350 GB (params only)
GPT-3 + optimizer states	175B	~1.4 TB

Even when a model fits on one GPU, training can be excruciatingly slow because GPUs can only process one batch at a time. Parallelism across multiple GPUs lets us split the work — either the data, the model, or both — to achieve reasonable training times.

1.3 The Three Classical Forms of Parallelism

Prior to Alpa, the ML systems community recognized three main parallelization strategies:

Data Parallelism (DP): Each GPU holds a complete copy of the model. Different GPUs process different portions of the training batch. After computing gradients, all GPUs synchronize their gradients (usually via all-reduce) so that every copy stays identical. Data parallelism is simple and scales well, but requires the entire model to fit on each GPU.

Operator (Tensor Model) Parallelism (TMP/OP): Instead of replicating the model, we split individual operators across GPUs. For example, a large matrix multiplication $C = A \times B$ can be split by partitioning the weight matrix $B$ column-wise across GPUs, so each GPU computes a portion of $C$ . This reduces memory per GPU but requires heavy inter-GPU communication at every operator boundary. Systems like Megatron-LM use hand-designed tensor parallelism strategies for transformer layers.

Pipeline Parallelism (PP): We divide the model into sequential stages and assign each stage to a different GPU (or group of GPUs). To overlap computation, we split the batch into smaller microbatches and pipeline them through the stages. This reduces communication volume (only activations between stages are sent point-to-point) but introduces pipeline bubbles — periods where some GPUs sit idle waiting for data.

1.4 The Combinatorial Explosion Problem

The real challenge is that these three parallelism forms must be combined to train the largest models. Megatron-LM famously uses "3D parallelism" — DP + TMP + PP together. But the number of possible configurations is enormous:

How many data-parallel replicas?
Which axes to partition each operator along?
How to split the model into pipeline stages?
How to assign devices to stages?
What mesh shape (e.g., 2×4 vs. 1×8) for each group of devices?

These decisions are deeply interdependent. Changing the pipeline stage boundaries changes the memory requirements, which changes whether a particular tensor parallelism plan is feasible, which in turn affects data parallelism degree. Manually navigating this space requires strong ML and systems expertise and produces solutions that are specific to one model and one cluster.

1.5 Integer Linear Programming (ILP) and Dynamic Programming (DP)

Alpa uses two optimization algorithms that come from operations research and computer science:

Integer Linear Programming (ILP): An optimization framework where the objective function and constraints are linear, but variables are restricted to integers. ILP solvers (like the CBC solver Alpa uses) can find globally optimal solutions, though worst-case runtime is exponential. In practice, the ILP problems Alpa constructs are tractable (solvable in seconds to minutes).

Dynamic Programming (DP): A technique for solving problems with optimal substructure — the optimal solution to the whole problem can be built from optimal solutions to subproblems. DP works by building a table of subproblem solutions bottom-up, avoiding redundant computation. Alpa uses DP for the inter-operator parallelism decisions.

1.6 Cluster Topology and Communication Bandwidth Hierarchy

Modern GPU clusters have a hierarchical communication structure:

Within a node (intra-node): GPUs are connected via NVLink or NVSwitch, providing 100–600 GB/s of bandwidth.
Between nodes (inter-node): Nodes communicate over Ethernet or InfiniBand, typically 10–100 Gbps (1–12.5 GB/s).

There is often a 10–50× difference in bandwidth between intra-node and inter-node links. This asymmetry is crucial to Alpa's design: communication-heavy parallelism (intra-operator) should be mapped to fast intra-node links, while communication-light parallelism (inter-operator/pipeline) should be mapped across slower inter-node links.

2. What This Paper Does (The Core Idea)

2.1 A New View: Intra-Operator vs. Inter-Operator Parallelism

Alpa's central conceptual contribution is to re-categorize all parallelism approaches into two orthogonal types, based on whether they partition individual operators:

Intra-Operator Parallelism: Any approach that partitions the computation of a single operator across devices. This includes:

Data parallelism (partitioning along the batch dimension)
Operator/tensor parallelism (partitioning along weight dimensions, as in Megatron-LM)
ZeRO optimizer (partitioning optimizer states and weight updates)

All of these share a common trait: they involve splitting tensors along some axis and using collective communication (all-reduce, all-gather, all-to-all) to coordinate.

Inter-Operator Parallelism: Any approach that assigns different operators to different devices without partitioning individual operators. This includes:

Pipeline parallelism (splitting the model into stages and pipelining microbatches)
Device placement (assigning different model parts to different GPUs)

Inter-operator parallelism uses only point-to-point communication between adjacent stages, which can be lightweight if sliced properly. However, it introduces pipeline bubbles (idle time).

This reclassification is more than cosmetic. The two types have fundamentally different communication characteristics that map naturally onto the hierarchical cluster topology:

Property	Intra-Operator	Inter-Operator
Communication pattern	Collective (all-reduce, all-gather)	Point-to-point (send/recv)
Communication volume	High (every operator boundary)	Low (only between stages)
Device utilization	High (all devices work simultaneously)	Lower (pipeline bubbles)
Best suited for	Fast intra-node links	Slower inter-node links

2.2 Hierarchical Optimization

Based on this reclassification, Alpa constructs a hierarchical execution plan:

At the inter-operator level: The compiler slices the model into pipeline stages and the cluster into device meshes. It assigns each stage to a mesh.
At the intra-operator level: For each stage-mesh pair, the compiler finds the best way to partition individual operators within that mesh.

The hierarchical decomposition is what makes the problem tractable. Instead of searching through a joint space that grows exponentially, Alpa solves two smaller problems:

The intra-op problem (ILP): Given a fixed subgraph and a fixed device mesh, find the best operator partitioning plan.
The inter-op problem (DP): Given the costs from the intra-op optimizer, find the best way to slice the model and cluster.

The inter-op optimizer calls the intra-op optimizer as a subroutine for each candidate stage-mesh pair, creating a two-level optimization loop.

2.3 How Alpa Differs from Prior Work

vs. Megatron-LM (manual 3D parallelism): Megatron-LM requires domain experts to manually design tensor parallelism strategies specific to transformer architectures. It assumes all pipeline stages are identical and uses the same TP/DP configuration uniformly. Alpa automates everything and allows heterogeneous stage sizes and per-stage parallelism plans.

vs. FlexFlow/Tofu (limited auto-parallelism): FlexFlow uses MCMC random search over a combined data+operator parallelism space but does not include pipeline parallelism. Tofu handles only linear graphs on a single node. Alpa covers the full space.

vs. PipeDream/DAPPLE (pipeline-focused): These systems automate pipeline + data parallelism but do not include operator parallelism within stages. Alpa includes all three.

3. Method Details

3.1 Intra-Operator Parallelism: The ILP Formulation

The intra-op pass takes a computational subgraph and a device mesh as input, and outputs the optimal parallel algorithm for every operator. The key abstractions are:

3.1.1 Device Mesh

A device mesh is a 2-dimensional logical view of physical devices. For example, 16 GPUs across 2 nodes could be viewed as a 2×8 mesh (2 rows = nodes, 8 columns = GPUs per node) or a 4×4 mesh, etc. The two mesh dimensions can have different communication bandwidths.

3.1.2 Sharding Specs

Each tensor is described by a sharding spec that says how it is distributed across the mesh. For an N-dimensional tensor on a 2D mesh, each tensor dimension can be:

S (Sharded): Partitioned along that axis, with a superscript indicating which mesh dimension (e.g., $S^0$ = split along mesh dimension 0)
R (Replicated): Fully copied across all devices

For example, for a matrix on a 2×2 mesh: $S^0R$ means row-partitioned (split along mesh dim 0, replicated along mesh dim 1). $S^0S^1$ means both row- and column-partitioned. $RR$ means fully replicated on all 4 devices.

The paper's Table 1 exhaustively lists all sharding specs for a 2D tensor on a 2×2 mesh — there are 10 possible specs.

3.1.3 Parallel Algorithms for an Operator

Each operator (e.g., matrix multiplication $C_{b,i,j} = \sum_k A_{b,i,k} B_{b,k,j}$ ) has multiple possible parallel algorithms, each corresponding to a different choice of which loop dimensions to parallelize across which mesh dimensions.

For a batched matmul, Table 3 in the paper lists 7 algorithms. For example:

Algorithm #1: Map loop $i \to$ mesh dim 0, loop $j \to$ mesh dim 1 → Output spec $RS^0S^1$ , zero communication cost.
Algorithm #2: Map loop $i \to$ mesh dim 0, loop $k \to$ mesh dim 1 → Output spec $RS^0R$ , requires all-reduce of $M/n_1$ bytes along mesh dimension 1.

3.1.4 Resharding Costs

When the output sharding spec of one operator does not match the input spec required by the next operator, a resharding operation is needed. This involves communication (all-gather, all-to-all, etc.). Table 2 in the paper lists resharding costs — for example, going from $S^0R$ to $RR$ requires an all-gather of $M$ bytes along mesh dim 0.

3.1.5 The ILP Objective

Given computational graph $G = (V, E)$ , for each node $v$ with $k_v$ possible algorithms, define:

$s_v \in \{0,1\}^{k_v}$ : one-hot decision vector (which algorithm to pick)
$c_v \in \mathbb{R}^{k_v}$ : communication cost vector
$d_v \in \mathbb{R}^{k_v}$ : compute cost vector
$R_{vu} \in \mathbb{R}^{k_v \times k_u}$ : resharding cost matrix between connected nodes

The objective is:

$\min_s \sum_{v \in V} s_v^T (c_v + d_v) + \sum_{(v,u) \in E} s_v^T R_{vu} s_u$

The first term minimizes compute and communication costs within each operator. The second term minimizes resharding costs between operators. The quadratic term $s_v^T R_{vu} s_u$ is linearized by introducing auxiliary variables, making it a standard ILP solvable by off-the-shelf solvers (CBC).

Practical simplifications:

Compute costs $d_v$ are set to zero for most operators (heavy ops like matmul always partition work evenly; lightweight ops have negligible cost).
Communication costs are estimated analytically (bytes / bandwidth) rather than profiled.
Computationally trivial operators are merged into their heavy operands to reduce graph size.
Post-ILP optimization replaces all-reduce with reduce-scatter + all-gather to achieve weight-update sharding (ZeRO-style) automatically.

3.2 Inter-Operator Parallelism: The DP Formulation

The inter-op pass takes the full computational graph and the cluster configuration, and decides:

How to slice operators into pipeline stages
How to slice the cluster into device meshes
How to assign stages to meshes

3.2.1 The Objective

Given operators $o_1, \ldots, o_K$ in topological order, sliced into $S$ stages $s_1, \ldots, s_S$ , with each stage $s_i$ assigned to a submesh of size $n_i \times m_i$ , and $B$ microbatches for pipelining, the total latency is:

$T^* = \min_{s_1,...,s_S; (n_1,m_1),...,(n_S,m_S)} \left\{ \sum_{i=1}^{S} t_i + (B-1) \cdot \max_{1 \leq j \leq S}\{t_j\} \right\}$

where $t_i = t_{\text{intra}}(s_i, \text{Mesh}(n_i, m_i))$ is the execution latency of stage $i$ on its assigned mesh (determined by the intra-op pass).

The first term is the latency of the first microbatch traversing the entire pipeline. The second term is the time for the remaining $B-1$ microbatches, bounded by the slowest stage (the pipeline bottleneck). This captures the classic pipeline parallelism insight: the ideal pipeline balances all stages to minimize the bottleneck.

3.2.2 Constraints

Two key constraints:

Colocation: Forward and backward operators of the same layer are assigned to the same submesh (avoiding expensive cross-mesh activation transfer).
Full coverage: All submeshes must exactly cover the entire cluster — no wasted devices: $\sum_{i=1}^{S} n_i \cdot m_i = N \cdot M$ .

3.2.3 Submesh Shapes

To make the DP tractable, Alpa restricts submesh shapes to two types:

1D submeshes: $(1,1), (1,2), (1,4), \ldots, (1,M)$ — subsets of GPUs within a single node
2D submeshes: $(2,M), (3,M), \ldots, (N,M)$ — using all GPUs per node across multiple nodes

The paper proves (Theorem 1 in Appendix A) that any combination of these shapes can always fully cover an $N \times M$ cluster mesh. This restriction also aligns with the bandwidth hierarchy: 2D submeshes that span the full second dimension maximize the use of fast intra-node connections.

3.2.4 The DP Recurrence

Define $F(s, k, d; t_{\max})$ = minimum total latency when slicing operators $o_k$ through $o_K$ into $s$ stages using $d$ devices, subject to no stage exceeding latency $t_{\max}$ .

Base case: $F(0, K+1, 0; t_{\max}) = 0$

Recurrence:

$F(s, k, d; t_{\max}) = \min_{\substack{k \leq i \leq K \\ n_s \cdot m_s \leq d \\ t_{\text{intra}}(\ldots) \leq t_{\max}}} \left\{ t_{\text{intra}}((o_k, \ldots, o_i), \text{Mesh}(n_s, m_s), s) + F(s-1, i+1, d - n_s \cdot m_s; t_{\max}) \right\}$

The optimal latency is: $T^*(t_{\max}) = \min_s \{F(s, 0, N \cdot M; t_{\max})\} + (B-1) \cdot t_{\max}$

The DP enumerates $t_{\max}$ from small to large, with early termination when $B \cdot t_{\max}$ exceeds the current best $T^*$ .

3.2.5 Operator Clustering

Real computation graphs can have tens of thousands of operators, making the DP infeasible ( $O(K^5)$ complexity with $K$ operators). Alpa clusters operators into $L$ "layers" ( $L \ll K$ ) using another DP that minimizes the maximum cross-layer communication while balancing computation per layer.

The clustering DP uses function $G(k, r)$ = minimum of the maximal data received by any single layer when clustering operators $o_1, \ldots, o_k$ into $r$ layers, with a FLOP balance constraint ensuring each layer's FLOP is within $(1+\delta)$ times the average.

3.3 Runtime Orchestration

After the compilation passes produce the execution plan, Alpa's runtime handles:

3.3.1 Cross-Mesh Resharding

When adjacent pipeline stages live on different device meshes with different shapes and sharding specs, Alpa must transfer tensors between them. This is a many-to-many multicast problem — not a simple send/recv.

Alpa uses a two-pass approach:

First pass: Compute which source devices hold which tiles needed by which destination devices. Generate point-to-point send/recv operations.
Second pass: Identify opportunities where the destination sharding has replicated dimensions. In these cases, send the data once to the destination mesh and use fast local all-gather within the mesh, rather than sending redundant copies across the slow inter-mesh link.

As shown in Figure 6 of the paper, this "local all-gather" optimization provides up to 2× speedup on 32 GPUs for Wide-ResNet by moving communication from slow inter-node links to fast intra-node NVLink.

3.3.2 MPMD Execution

Unlike SPMD systems (where all devices run the same program), Alpa uses Multiple Program Multiple Data (MPMD) at the inter-op level — each device mesh executes different instructions. Alpa generates static instruction lists for each mesh (including compute, communicate, allocate/deallocate memory, synchronize), dispatches them before execution begins, and avoids runtime coordination overhead.

3.4 The API

Alpa's user-facing API is remarkably simple (see Figure 4 in the paper). Developers just add a @parallelize decorator to their Jax training function. On first call, Alpa traces the function to obtain the Jax IR, runs both compilation passes, and replaces the original function with the parallel version. No manual parallelism annotations needed.

4. Experiment Setup

4.1 Hardware

All experiments run on an Amazon EC2 cluster of 8 p3.16xlarge instances with:

64 GPUs total (8 NVIDIA V100 16GB per node)
NVLink intra-node (high bandwidth, ~160 GB/s between pairs)
25 Gbps Ethernet inter-node (~3.1 GB/s)
64 vCPUs and 488 GB RAM per node

This creates a clear two-level bandwidth hierarchy — approximately 50× faster communication within a node vs. between nodes.

4.2 Models (Table 4)

Model	Task	Batch Size	Parameter Range	Precision
GPT-3	Language modeling	1024	0.35B–39B	FP16
GShard MoE	Language modeling	1024	0.38B–70B	FP16
Wide-ResNet	Image classification	1536	0.25B–13B	FP32

Models are scaled alongside GPUs (weak scaling) — larger models use more GPUs. Detailed specifications are in Appendix B (Tables 6-8).

4.3 Baselines

GPT-3: Megatron-LM v2 — state-of-the-art hand-tuned system for transformer LMs combining DP + TMP + PP with grid-searched parameters.
MoE: DeepSpeed — hand-crafted expert parallelism + ZeRO data parallelism. No pipeline parallelism support.
Wide-ResNet: PP-DP baseline (Alpa with only pipeline + data parallelism, mimicking PipeDream/DAPPLE).
For all models: Alpa with only intra-op parallelism ("Intra-op only") and only inter-op parallelism ("Inter-op only").

4.4 Metric

Aggregated PFLOPS (peta floating-point operations per second) across the cluster. Since models scale with GPU count (weak scaling), PFLOPS is the correct metric for comparing scaling efficiency.

5. Results & Analysis

5.1 GPT-3 (Figure 7a)

Alpa matches or slightly outperforms Megatron-LM on all GPU counts (1 to 64). Both achieve near-linear weak scaling. Key observations:

"Intra-op only" degrades above 16 GPUs because cross-node collective communication becomes a bottleneck.
"Inter-op only" surprisingly maintains linear scaling to 64 GPUs.
Alpa achieves slight improvements over Megatron-LM by automatically discovering weight update sharding (ZeRO-style), which Megatron-LM does not support.

Investigation of the auto-generated plans reveals they closely resemble Megatron-LM's best hand-tuned configurations: evenly-sized stages, batch-dimension partitioning when memory allows, and non-batch partitioning under memory pressure.

5.2 GShard MoE (Figure 7b)

Alpa dramatically outperforms DeepSpeed:

3.5× speedup on 2 nodes (16 GPUs)
9.7× speedup on 4 nodes (32 GPUs)

DeepSpeed collapses beyond 1 node because its intra-op-only strategy cannot handle the low inter-node bandwidth. "Intra-op only" fails similarly. "Inter-op only" runs out of memory on 32+ GPUs due to imbalanced stage slicing for MoE models. Only Alpa's full hierarchical approach scales effectively.

5.3 Wide-ResNet (Figure 7c)

Wide-ResNet is heterogeneous — activation tensors shrink while weight tensors grow through the network, creating imbalanced memory and compute profiles across layers. No manually-designed parallelization strategy exists.

"PP-DP" and "Inter-op only" run out of memory for large models (cannot partition weights).
"Intra-op only" fails across nodes (communication bottleneck).
Alpa achieves 80% linear scaling efficiency on 32 GPUs and successfully trains models up to 13B parameters.

The case study (Figure 12) reveals Alpa's non-trivial strategy: 3 stages with 4, 4, and 8 GPUs respectively. Data parallelism dominates early stages (large activations), while the last stage uses a complex convolution partitioning scheme (large weights). This strategy would be nearly impossible to design manually.

5.4 Intra-Op Ablation (Figure 8)

Comparing Alpa's ILP solver against alternatives on a single node (8 GPUs):

Method	Behavior
Data parallelism	Runs out of memory quickly
ZeRO-2	Solves memory but doesn't optimize communication
ZeRO-3	Same issue — always communicates all gradients
Heuristic (GSPMD-style)	Partitions everything but can increase communication
ILP (Alpa)	Near-linear scaling; minimizes communication overhead

The ILP consistently finds the best plan because it globally optimizes the partitioning of every operator to minimize total communication.

5.5 Inter-Op Ablation (Figure 9)

Comparing Alpa's DP algorithm against rule-based slicing:

Equal operator: Assigns equal numbers of operators per stage. Alpa's DP outperforms it because communication-aware clustering groups related operators together.
Equal layer: Forces equal numbers of model layers per stage. Matches Alpa's DP on homogeneous models (GPT) but loses on heterogeneous models. For Wide-ResNet on 32 GPUs, Alpa's DP outperforms "Equal layer" by 1.6× and "Equal operator" by 2.6×.

5.6 Compilation Time (Figure 10, Table 5)

For the largest model (GPT-3 39B on 64 GPUs):

Phase	Time
Compilation (parallelized)	1582.66 s
Profiling	804.48 s
Stage Construction DP	1.65 s
Other	4.47 s
Total	2393.26 s (~40 min)

Without optimizations (operator clustering + early pruning), compilation would exceed 40 hours. The 40-minute compilation time is acceptable since actual training runs for weeks.

5.7 Cross-Mesh Resharding (Figure 11)

The local all-gather optimization provides a 2× speedup on 32 GPUs for Wide-ResNet over naive send/recv, by moving repeated data transfers from slow inter-node links to fast intra-node NVLink.

6. Limitations & Boundary Conditions

6.1 Acknowledged Limitations

The authors explicitly identify several limitations:

No cross-stage communication cost modeling: The DP does not account for data transfer costs between pipeline stages. While typically small, this could matter in pathological cases.
Number of microbatches is a hyperparameter: The DP optimizes stage assignment for a given $B$ but does not jointly optimize $B$ . Users must search over $B$ separately.
Static linear pipeline schedule only: Alpa assumes a linear pipeline (stages execute sequentially). It cannot handle dynamic schedules or parallel branches in the computational graph.
No computation-communication overlap optimization: Alpa does not optimize for overlapping compute with communication, which is an important practical optimization in systems like Megatron-LM.
Static graphs only: All tensor shapes must be known at compile time. Dynamic shapes (e.g., variable-length sequences) are not supported.

6.2 Additional Observations

Suboptimality of hierarchical decomposition: Alpa's two-level approach is not guaranteed to find the globally optimal plan. The ILP within each stage is optimal, but the decomposition into stages may miss plans that are globally better.
Scaling to very large clusters: The evaluation maxes out at 64 GPUs. Modern training setups use thousands of GPUs. While the compilation time scales linearly, the quality of the auto-generated plans at massive scale is not evaluated.
MoE-specific optimizations: DeepSpeed's poor performance in the evaluation is partly because it lacks pipeline parallelism for MoE. A fairer comparison would be against a system that combines all three parallelism types for MoE models.
Submesh shape restriction: The restriction to 1D submeshes within a node and full-width 2D submeshes excludes shapes like $(2, 4)$ on an 8-GPU node. While the paper argues these are inferior, this is not formally proven for all cases.

7. Reproducibility & Practical Notes

7.1 Code Availability

Alpa is open-source at https://github.com/alpa-projects/alpa. The implementation consists of ~16K lines of Python and ~6K lines of C++, built on top of Jax (frontend) and XLA (backend). Communication uses NCCL, and the distributed runtime uses Ray actors.

7.2 Reproducibility

The experimental setup is well-documented: specific EC2 instance types, model configurations (Appendix B), and parameter search procedures. The evaluation uses standard models (GPT-3, MoE, Wide-ResNet) with public architectures. However, reproducing the exact results requires access to 8× p3.16xlarge instances, which costs approximately $196/hour on AWS.

7.3 Practical Considerations

Compilation cost: The 40-minute compilation time for a 39B model is a one-time cost, acceptable for week-long training runs. But for rapid experimentation with different model architectures, this could be a bottleneck.

Integration: Alpa requires models to be written in Jax. PyTorch users (the majority of the ML community) cannot directly use it. This has limited adoption despite the strong technical contributions.

Cost model accuracy: Alpa's ILP uses analytical cost estimates (bytes/bandwidth) rather than profiled measurements. This works well for the tested configurations but may be less accurate on hardware with non-uniform bandwidth characteristics or when communication and compute overlap significantly.

Impact and legacy: Alpa's first author (Lianmin Zheng) later led the development of vLLM, the dominant open-source LLM inference system. Many ideas from Alpa — particularly the hierarchical parallelism view and automatic plan generation — have influenced subsequent systems like DeepSpeed's auto-parallelism features and Unity (the successor to FlexFlow). The paper received the OSDI 2022 Jay Lepreau Best Paper Award.

References

Narayanan et al. "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM." SC 2021.
Rajbhandari et al. "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models." SC 2020.
Huang et al. "GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism." NeurIPS 2019.
Narayanan et al. "PipeDream: Generalized Pipeline Parallelism for DNN Training." SOSP 2019.
Jia et al. "Beyond Data and Model Parallelism for Deep Neural Networks" (FlexFlow). SysML 2019.
Lepikhin et al. "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding." 2020.
Xu et al. "GSPMD: General and Scalable Parallelization for ML Computation Graphs." 2021.
Li et al. "TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models." 2021.

Review written on 2026-03-26.