Why FLOP-based time estimates fail for LLM fine-tuning jobs
Hugging Face researchers show that standard FLOP calculations poorly predict fine-tuning duration. A CatBoost model using configuration data reduces error from 12 hours to under two.
Hugging Face researchers analyzed 272 real-world supervised fine-tuning jobs to understand why theoretical compute estimates often miss the mark. Published in November 2026, their findings reveal that standard floating-point operation counts explain very little of the actual wall-clock time required for training large language models on shared infrastructure.
What happened
The study focused on production workloads running on NVIDIA H200 GPUs, totaling approximately 7,382 GPU-hours. The team compared traditional analytical estimates against a machine learning approach to predict how long a job would take from start to finish. The goal was to improve cluster scheduling, reducing idle time when jobs finish early and preventing delays when they run late.
The results showed a significant gap between theory and practice. The standard FLOP-based baseline had a mean absolute error of 12.41 hours and an R² value of only 0.31. In contrast, a CatBoost regressor trained on job configuration data achieved an R² of 0.98. This new model reduced the mean absolute error to just 1.71 hours, with a median error of 0.51 hours across five-fold cross-validation.
The researchers noted that even more complex analytical models, such as Calculon, failed to capture the full picture. While Calculon accounts for memory traffic and communication at the step level, it still estimated only about 11% of the observed wall-clock time for production jobs. This highlights that much of the runtime is consumed by overheads outside the core optimization loop.
How it works
Theoretical estimates typically calculate training time by dividing the total floating-point operations by the peak throughput of the hardware, adjusted by a Model FLOPs Utilization factor. This formula assumes the job is purely computational. However, real-world fine-tuning involves environment setup, model loading, tokenization, periodic evaluations, and checkpoint uploads. These tasks compete for network and storage I/O in multi-tenant clusters and do not scale linearly with compute power.
To address this, the researchers treated end-to-end duration as a supervised regression problem. They built a feature vector using only information available before dispatch, such as hyperparameters, dataset sizes, and model identifiers. Crucially, they included derived features that expose overheads, such as the total number of evaluation samples and the volume of checkpoint data saved. The theoretical FLOP estimate was included as one feature among many, allowing the model to learn corrections based on historical data.
The team tested several algorithms, including Linear Regression, Ridge, XGBoost, and CatBoost. After feature selection and grid search, CatBoost emerged as the top performer. It effectively captured the nonlinear interactions between configuration choices and runtime, such as how evaluation frequency impacts total duration independently of model size.
Key details
- Dataset: 272 supervised fine-tuning runs on NVIDIA H200 GPUs, ranging from 1.6 to 172.4 hours in duration.
- Baseline performance: The best FLOP-based estimate (assuming 30% MFU) had a mean absolute error of 12.41 hours.
- Model performance: The CatBoost regressor achieved a mean absolute error of 1.71 hours and an R² of 0.98.
- Top features: Evaluation data volume, trainable parameter count, and checkpoint save volume were the strongest predictors of runtime.
- Error distribution: For jobs under 60 hours, the median absolute error remained below one hour.
- Limitations: The model is specific to H200 hardware and supervised fine-tuning, with less accuracy for jobs exceeding 60 hours due to sparse data.
Why it matters
For engineering leads managing shared GPU clusters, accurate duration estimates are critical for efficient resource allocation. When predictions are off by half a day, schedulers cannot reliably pack jobs together or promise delivery dates. This leads to fragmented GPU usage, where expensive hardware sits idle while other teams wait for resources. Moving from guesswork to data-driven planning allows for tighter scheduling and higher utilization rates.
The study also highlights that configuration choices have outsized impacts on runtime that are not obvious from compute metrics alone. Features like evaluation cadence and checkpoint policy drive significant overhead. Teams often set these parameters by habit rather than necessity. Understanding their impact on wall-clock time enables engineers to make trade-offs between experimental rigor and iteration speed, optimizing both cost and developer productivity.
What you can do
- Track evaluation frequency and checkpoint size in your training logs, as these are major drivers of runtime overhead.
- Move beyond simple FLOP calculations for scheduling; use historical run data to build regression models for your specific infrastructure.
- Include pre-dispatch features like dataset size and evaluation steps in your duration predictions rather than relying solely on model parameters.
- Be cautious with estimates for very long-running jobs, as errors tend to increase significantly beyond the 60-hour mark.
- Retrain prediction models when changing hardware generations, as utilization characteristics and I/O bottlenecks will differ.
- Audit your training configurations to ensure evaluation and saving frequencies align with actual debugging needs rather than default settings.



