aiqbench analysis
This research addresses a critical bottleneck in data management by introducing lightweight metadata structures, such as 2D convex hulls, to enable efficient pruning for black-box ML filter predicates. The proposed enhancements achieve up to 38.31% pruning effectiveness on standard benchmarks while maintaining minimal storage overhead, directly improving query performance in systems like DuckDB. For database engineers and ML practitioners, this work offers a practical pathway to optimize storage I/O when leveraging new AI-native filtering capabilities.
MLSkip: Data Skipping for ML Filters via Lightweight Metadata
arXiv:2606.03946v1 Announce Type: cross Abstract: Database vendors recently released AI functions that can be used in filter predicates. As such functions often rely on costly, black-box ML models, they unveil new data management challenges. Concretely, traditional data skipping techniques for integer and string data fail to be applicable to the new filter type. Indeed, there is no known mechanism for pruning non-qualifying row groups, e.g., when reading files from blob storage. In this work, we initiate the study of data skipping techniques for ML filters. We make the case that Parquet's default min-max metadata is enough to enable pruning. To this end, we draw connections to two lines of research: (i) the recently proposed query language for ML models and (ii) neural network verification. Our preliminary results on ReLU architectures show that on tables from TPC-H and TPC-DS, the average pruning effectiveness for filters of selectivity below 0.1% amounts to 27.4%. Finally, inspired by research on spatial joins, we propose an enhanced metadata structure: a size-bounded 2D convex hull that verification tools can make better use of, increasing the pruning effectiveness to 38.31%, while occupying at most 45 bytes per row group and column pair. We observe an end-to-end speedup of 1.07$\times$ over PyTorch in DuckDB.
Abstract:Database vendors recently released AI functions that can be used in filter predicates. As such functions often rely on costly, black-box ML models, they unveil new data management challenges. Concretely, traditional data skipping techniques for integer and string data fail to be applicable to the new filter type. Indeed, there is no known mechanism for pruning non-qualifying row groups, e.g., when reading files from blob storage. In this work, we initiate the study of data skipping techniques for ML filters. We make the case that Parquet's default min-max metadata is enough to enable pruning. To this end, we draw connections to two lines of research: (i) the recently proposed query language for ML models and (ii) neural network verification. Our preliminary results on ReLU architectures show that on tables from TPC-H and TPC-DS, the average pruning effectiveness for filters of selectivity below 0.1% amounts to 27.4%. Finally, inspired by research on spatial joins, we propose an enhanced metadata structure: a size-bounded 2D convex hull that verification tools can make better use of, increasing the pruning effectiveness to 38.31%, while occupying at most 45 bytes per row group and column pair. We observe an end-to-end speedup of 1.07$\times$ over PyTorch in DuckDB.
Databases (cs.DB); Machine Learning (cs.LG); Logic in Computer Science (cs.LO)
arXiv:2606.03946 [cs.DB]
(or arXiv:2606.03946v1 [cs.DB] for this version)
https://doi.org/10.48550/arXiv.2606.03946 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 302 words
Related coverage
MLSkip: Data Skipping for ML Filters via Lightweight Metadata
arXiv · Sep 24, 2026
PRX Part 4: Our Data Strategy
Hugging Face · Jul 6, 2026
Designing GPU-Accelerated Query Engines with NVIDIA GQE
NVIDIA · Jun 30, 2026
Presentation: From S3 to GPU in One Copy: Rethinking Data Loading for ML Training
InfoQ · Sep 4, 2026