analiza aiqbench
This empirical study of a 504-GPU B200 cluster reveals that hardware failures are routine production realities rather than anomalies, necessitating multi-signal detection strategies since no single metric reliably predicts all failure types. The analysis highlights critical storage I/O bottlenecks at scale, where checkpoint operations consume significant bandwidth and node exclusions are heavily concentrated on a small subset of hardware. For engineering teams, the key takeaway is that automated retry chains significantly outperform manual interventions, offering a 2.7x improvement in recovery success rates during distributed training sessions.
From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
arXiv:2605.09370v4 Announce Type: replace-cross Abstract: Large-scale AI training is fundamentally a distributed systems problem, where hardware failures are routine operating conditions rather than rare exceptions, yet public operational evidence from production training clusters remains limited. This report presents an empirical analysis of a 63-node NVIDIA B200 production cluster (504 GPUs), using 55 days of Prometheus time-series data and 73 days of operational logs covering 224 multi-node training sessions. The environment is cross-organizational: five parties (SKT, Upstage, Lablup, NVIDIA Korea, VAST Data) share a unified monitoring pipeline. This enabled joint diagnosis of a 60-node-scale storage I/O bottleneck absent in 2-4-node tests, a production-scale phenomenon no single team could isolate alone. We perform three quantitative analyses yielding four findings. First, over 751 Prometheus metrics and 10 XID-identified GPU failures, no single metric is consistently dominant across failure types, motivating multi-signal detection. Second, 523 checkpoint events trace the save/load path from GPU VRAM to the NFS server: restart loading reaches 21.5% of maximum read bandwidth (700 GB/s) and save bursts 16.0% of maximum write bandwidth (250 GB/s), with NFS/RPC queueing and transport-layer backlog rising together. Third, across 224 sessions over 73 days, node exclusions concentrate so the top 3 of 63 nodes account for over 50%. Fourth, auto-retry chain analysis shows a 33.3% success rate over 12 chains (73 attempts), 2.7x the 12.5% manual rate, with a median retry interval of 11 minutes (IQR 10-11). All analyses are grounded in production infrastructure providing session-level workload management, GPU-centric scheduling, and unified observability.
Abstract:Large-scale AI training is fundamentally a distributed systems problem, where hardware failures are routine operating conditions rather than rare exceptions, yet public operational evidence from production training clusters remains limited. This report presents an empirical analysis of a 63-node NVIDIA B200 production cluster (504 GPUs), using 55 days of Prometheus time-series data and 73 days of operational logs covering 224 multi-node training sessions. The environment is cross-organizational: five parties (SKT, Upstage, Lablup, NVIDIA Korea, VAST Data) share a unified monitoring pipeline. This enabled joint diagnosis of a 60-node-scale storage I/O bottleneck absent in 2-4-node tests, a production-scale phenomenon no single team could isolate alone. We perform three quantitative analyses yielding four findings. First, over 751 Prometheus metrics and 10 XID-identified GPU failures, no single metric is consistently dominant across failure types, motivating multi-signal detection. Second, 523 checkpoint events trace the save/load path from GPU VRAM to the NFS server: restart loading reaches 21.5% of maximum read bandwidth (700 GB/s) and save bursts 16.0% of maximum write bandwidth (250 GB/s), with NFS/RPC queueing and transport-layer backlog rising together. Third, across 224 sessions over 73 days, node exclusions concentrate so the top 3 of 63 nodes account for over 50%. Fourth, auto-retry chain analysis shows a 33.3% success rate over 12 chains (73 attempts), 2.7x the 12.5% manual rate, with a median retry interval of 11 minutes (IQR 10-11). All analyses are grounded in production infrastructure providing session-level workload management, GPU-centric scheduling, and unified observability.
42 pages, 19 figures, 16 tables. Lablup Technical Report
Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI)
C.4; D.4.5; I.2.11
arXiv:2605.09370 [cs.DC]
(or arXiv:2605.09370v4 [cs.DC] for this version)
Text integral sub licența deschisă a publicației · 272 cuvinte
Subiecte conexe
From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
arXiv · 9 iun. 2026
[AINews] not much happened today
Latent Space · 10 sept. 2026
Links #5: 2026/07
LessWrong · 31 iul. 2026
From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
arXiv · 16 iun. 2026