aiqbench analysis
This study applies distribution-free conformal prediction to steel fatigue strength modeling, revealing that high aggregate accuracy can mask significant reliability gaps in critical design regions like the highest-strength quartile. By comparing seven interval-construction methods across 50 data splits, it demonstrates how locally-adaptive techniques restore conditional coverage without sacrificing average predictive width, offering a rigorous framework for uncertainty quantification in engineering applications.
Distribution-Free Conformal Prediction for Steel Fatigue Strength: Marginal Validity Is Not Enough
arXiv:2608.07589v2 Announce Type: replace-cross Abstract: Predicting fatigue failure in steel components experimentally is costly, requiring testing across multiple compositions and processing conditions, spurring research on data-driven prediction models. Studies using the NIMS MatNavi steel fatigue dataset often report high point-prediction accuracy but rely on aggregate error metrics, leaving uncertainty about the reliability of individual predictions and whether accuracy is consistent across the fatigue-strength spectrum. This paper is the first to apply conformal prediction to steel fatigue strength, comparing seven interval-construction methods across 50 independent data splits and distinguishing marginal coverage from coverage within specific sub-regions of the predicted property. A gradient-boosting point model achieves an R^2 of 0.976 +/- 0.009 and a mean absolute error of 18.3 +/- 2.3 MPa. Split-conformal prediction provides valid marginal coverage (0.918) but drops to 0.758 in the highest-strength quartile, where design margins are most critical, a pattern also observed with a Gaussian process baseline. Two locally-adaptive methods correct this: a cross-fitted normalised conformal method holds 0.872-0.940 across quartiles at no cost in average width, and Mondrian group-conditional conformal prediction holds the tightest band of any method (0.917-0.939) at a 12% width premium, part of which traces to the more conservative finite-sample quantile level implied by per-group calibration at this sample size. Conformalized quantile regression, by contrast, restores marginal validity but inflates intervals in every quartile without closing the conditional gap. Marginal coverage claims for ML-based fatigue-strength predictions can conceal systematic unreliability precisely where engineering decisions are most risky; therefore, conditional coverage should be routinely assessed alongside marginal coverage.
Abstract:Predicting fatigue failure in steel components experimentally is costly, requiring testing across multiple compositions and processing conditions, spurring research on data-driven prediction models. Studies using the NIMS MatNavi steel fatigue dataset often report high point-prediction accuracy but rely on aggregate error metrics, leaving uncertainty about the reliability of individual predictions and whether accuracy is consistent across the fatigue-strength spectrum. This paper is the first to apply conformal prediction to steel fatigue strength, comparing seven interval-construction methods across 50 independent data splits and distinguishing marginal coverage from coverage within specific sub-regions of the predicted property. A gradient-boosting point model achieves an R^2 of 0.976 +/- 0.009 and a mean absolute error of 18.3 +/- 2.3 MPa. Split-conformal prediction provides valid marginal coverage (0.918) but drops to 0.758 in the highest-strength quartile, where design margins are most critical, a pattern also observed with a Gaussian process baseline. Two locally-adaptive methods correct this: a cross-fitted normalised conformal method holds 0.872-0.940 across quartiles at no cost in average width, and Mondrian group-conditional conformal prediction holds the tightest band of any method (0.917-0.939) at a 12% width premium, part of which traces to the more conservative finite-sample quantile level implied by per-group calibration at this sample size. Conformalized quantile regression, by contrast, restores marginal validity but inflates intervals in every quartile without closing the conditional gap. Marginal coverage claims for ML-based fatigue-strength predictions can conceal systematic unreliability precisely where engineering decisions are most risky; therefore, conditional coverage should be routinely assessed alongside marginal coverage.
11 pages, 5 figures, 3 tables. v2: adds Mondrian group-conditional conformal prediction and conformalized quantile regression (seven methods total); quartile cut points are now derived from training-set out-of-fold predictions
Applications (stat.AP); Machine Learning (cs.LG)
arXiv:2608.07589 [stat.AP]
(or arXiv:2608.07589v2 [stat.AP] for this version)
Full text under the publisher's open licence · 291 words
Related coverage
DI Series: Serving Kimi-K3 MXFP4 Wide-EP Disaggregated on AMD Instinct MI300X and MI325X
AMD · Oct 8, 2026
Beyond Marginal Validity: Finite-Sample Guarantees for Localized Conformal Prediction
arXiv · Aug 7, 2026
Stock Market Today Highlights, Sept 2: Sensex ends 373 pts lower at 76, 570, Nifty below 24, 000; Asian Paints, HDFC Bank fall most
thehindubusinessline.com · Sep 2, 2026
HeAD-CP: Heterophily-Aware Diffused Conformal Prediction Sets for Graph Neural Networks
arXiv · Jul 29, 2026