analiza aiqbench
The paper asks whether an uncertainty threshold that meets a safety performance target on one dataset still does so under other conditions such as far distance, rain, or a retrained detector. Its answer is mostly negative: on KITTI and nuScenes, the recorded data rarely contained enough independent sequences to support any applicable outcome, and frame-level counting produced "not applicable" results that undercut thresholds which looked acceptable. For readers working on ML safety assurance, the practical takeaway is that a performance claim should be tested in each triggering condition and reported alongside retained correct detections and the number of independent sequences.
Uncertainty Evaluation to Support Safety of the Intended Functionality Analysis for Determining the Applicability of Performance Targets Across Triggering Conditions in ML-Based LiDAR Object Detection
arXiv:2610.35092v1 Announce Type: cross Abstract: For safety of the intended functionality (SOTIF) analysis according to ISO 21448, a machine learning object detector for light detection and ranging (LiDAR) point clouds can meet a performance target, a limit on the mean per-frame share of wrong detections among accepted detections, by rejecting detections above an uncertainty threshold. A threshold selected on data of one condition carries a statistical guarantee only for data distributed like its selection data, whereas SOTIF analysis covers every triggering condition. This paper presents an uncertainty evaluation in which the threshold is transferred into a triggering condition, a test decides at a stated error probability whether the target is applicable there, not applicable or not determined, and a lower bound counts the independent units an applicable outcome requires. For PointPillars ensembles on KITTI and nuScenes, counted in recorded sequences, as the dependence between frames requires, neither dataset held enough independent units for any target limit, and no transfer had a not applicable outcome. Counted in frames, the same data gave not applicable outcomes at limits fixed in advance after transfers into far distance, into rain and between two separately trained detectors, thresholds that met the target limit after a transfer rejected most correct detections, and in rain the area under the receiver operating characteristic curve did not decrease. A performance target met under one condition therefore counts as evidence under a triggering condition only after a test there, reported with its share of accepted correct detections and its number of independent recorded sequences.
Abstract:For safety of the intended functionality (SOTIF) analysis according to ISO 21448, a machine learning object detector for light detection and ranging (LiDAR) point clouds can meet a performance target, a limit on the mean per-frame share of wrong detections among accepted detections, by rejecting detections above an uncertainty threshold. A threshold selected on data of one condition carries a statistical guarantee only for data distributed like its selection data, whereas SOTIF analysis covers every triggering condition. This paper presents an uncertainty evaluation in which the threshold is transferred into a triggering condition, a test decides at a stated error probability whether the target is applicable there, not applicable or not determined, and a lower bound counts the independent units an applicable outcome requires. For PointPillars ensembles on KITTI and nuScenes, counted in recorded sequences, as the dependence between frames requires, neither dataset held enough independent units for any target limit, and no transfer had a not applicable outcome. Counted in frames, the same data gave not applicable outcomes at limits fixed in advance after transfers into far distance, into rain and between two separately trained detectors, thresholds that met the target limit after a transfer rejected most correct detections, and in rain the area under the receiver operating characteristic curve did not decrease. A performance target met under one condition therefore counts as evidence under a triggering condition only after a test there, reported with its share of accepted correct detections and its number of independent recorded sequences.
24 pages, 6 figures. Extended version of the VEHITS 2026 conference paper (doi:https://doi.org/10.5220/0014967200004030). Submitted for journal publication. Code and recorded results: this https URL
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Robotics (cs.RO)
arXiv:2610.35092 [cs.CV]
(or arXiv:2610.35092v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2610.35092 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 299 cuvinte
Subiecte conexe
Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?
arXiv · 29 sept. 2026
Agentic Verified Optimization of Real-World Software
arXiv · 23 oct. 2026
KIT and partners build a blueprint for quantum software
Quantum Zeitgeist · 23 oct. 2026
The next phase of quantum energy science: from mechanisms to applications
arXiv · 21 oct. 2026