From Throughput to Goodput: Measuring LLM Training Efficiency at Scale

Beyond Tokens Per Second: The Rise of ‘Goodput’ in LLM Training

For years, the race to train larger and more capable large language models (LLMs) has been largely defined by a single metric: tokens per second. But a growing understanding of the complexities involved in large-scale pretraining is shifting the focus towards a more holistic measure – “goodput.” Goodput, as formalized by Google, represents the fraction of theoretical training capacity that translates into actual training progress. This isn’t just about speed; it’s about efficient speed.

Why Throughput Isn’t Enough

Raw throughput, although important, is easily misleading. A system churning out a high number of tokens per second can still be remarkably inefficient if it’s constantly restarting, struggling with data recovery, or underutilizing its compute resources. Consider a scenario where a training run is interrupted every few hours due to infrastructure failures. Even if it achieves a high tokens/second rate when running, the frequent disruptions significantly erode overall progress.

Goodput addresses this by factoring in reliability, recovery speed, and compute efficiency. It forces teams to acknowledge and address the sources of lost time and wasted resources – what Google terms “badput.”

Deconstructing Goodput: A Three-Layer Approach

To effectively measure and improve goodput, it’s helpful to view the training stack as three distinct layers:

1. Infra Layer: This encompasses the underlying cluster, orchestration, runtimes, and fault handling mechanisms. Infra goodput measures how often the job is actually in a healthy training state.
2. Framework Layer: This layer handles distributed training, checkpointing, state management, and initialization. Framework goodput assesses how much progress is lost during failures and the efficiency of recovery processes.
3. Program/Model Layer: This focuses on the parallelism strategy, kernels, precision, and batch/sequence regime – essentially, how efficiently the model’s math maps to the available hardware.

Each layer contributes to the overall training goodput, and identifying bottlenecks within each layer is crucial for optimization.

Infra Goodput: Minimizing Downtime

The formula for infra goodput highlights the importance of minimizing disruptions. A simple operational definition is: Infra Goodput = (W – ΣTDT_i) / W, where W is the measurement window and TDT_i represents the downtime from each disruption. Reliability becomes paramount as job sizes increase, demanding engineered solutions rather than relying solely on hardware improvements.

Framework Goodput: Efficient Recovery

Even with a reliable infrastructure, checkpointing and recovery processes can introduce significant overhead. Framework goodput measures the fraction of time not lost to these processes. Infrequent checkpointing risks losing substantial progress during failures, while overly frequent checkpointing consumes valuable resources. Finding the optimal balance is key.

Model Goodput: Maximizing Compute Utilization

Model goodput focuses on how effectively GPUs are utilized during training. Model FLOPs Utilization (MFU) is a common metric, measuring the observed compute rate against the theoretical peak. Low MFU often indicates issues with communication overhead, parallelism configuration, or memory bandwidth limitations.

The Future of LLM Training: Stack-Level Efficiency

The shift towards goodput signals a broader trend in LLM training: a move away from isolated optimizations towards holistic, stack-level efficiency. Future advancements will likely focus on:

• Automated Fault Tolerance: Systems that can automatically detect, isolate, and remediate failures with minimal downtime.
• Adaptive Checkpointing: Frameworks that dynamically adjust checkpointing frequency based on failure rates and recovery costs.
• Hardware-Aware Model Design: Models designed to maximize utilization of specific hardware architectures, minimizing communication overhead and memory bottlenecks.
• Integrated Monitoring and Diagnostics: Tools that provide real-time visibility into goodput across all three layers, enabling rapid identification and resolution of performance issues.

Did you know?

Google’s Goodput metric isn’t just a theoretical concept. They’ve developed an API-driven approach to compute goodput and diagnose badput sources, providing actionable insights for improving training efficiency.

FAQ: Understanding Goodput

• What is the difference between throughput and goodput? Throughput measures the raw rate of token processing, while goodput measures the fraction of potential training capacity that is actually converted into progress.
• Why is goodput important? Goodput provides a more accurate and actionable measure of training efficiency, helping teams identify and address bottlenecks across the entire training stack.
• How can I improve goodput? Focus on minimizing downtime, optimizing recovery processes, and maximizing compute utilization.

As LLMs continue to grow in size and complexity, the importance of efficient training will only increase. Goodput provides a valuable framework for navigating these challenges and unlocking the full potential of these powerful models.

References:

1. Google Cloud: Goodput metric as measure of ML productivity
2. AWS: Checkpointless training on Amazon SageMaker HyperPod
3. Kokolis et al.: Revisiting Reliability in Large-Scale Machine Learning Research Clusters (arXiv)