aiqbench analysis
This research addresses the critical challenge of optimizing database performance when using AI functions as filter predicates, where traditional data skipping methods fail. By leveraging existing Parquet metadata and proposing a lightweight 2D convex hull structure, the authors demonstrate significant pruning effectiveness for ReLU architectures without incurring heavy computational overhead. The work bridges database management systems with machine learning verification techniques to accelerate query execution on large datasets.
MLSkip: Data Skipping for ML Filters via Lightweight Metadata
arXiv:2606.03946v2 Announce Type: replace-cross Abstract: Database vendors recently released AI functions that can be used in filter predicates. As such functions often rely on costly, black-box ML models, they unveil new data management challenges. Concretely, traditional data skipping techniques for integer and string data fail to be applicable to the new filter type. Indeed, there is no known mechanism for pruning non-qualifying row groups, e.g., when reading files from blob storage. In this work, we initiate the study of data skipping techniques for ML filters. We make the case that Parquet's default min-max metadata is enough to enable pruning. To this end, we draw connections to two lines of research: (i) the recently proposed query language for ML models and (ii) neural network verification. Our preliminary results on ReLU architectures show that on tables from TPC-H and TPC-DS, the average pruning effectiveness for filters of selectivity below 0.1% amounts to 27.4%. Finally, inspired by research on spatial joins, we propose an enhanced metadata structure: a size-bounded 2D convex hull that verification tools can make better use of, increasing the pruning effectiveness to 38.31%, while occupying at most 45 bytes per row group and column pair. We observe an end-to-end speedup of 1.07$\times$ over PyTorch in DuckDB.
Abstract:Database vendors recently released AI functions that can be used in filter predicates. As such functions often rely on costly, black-box ML models, they unveil new data management challenges. Concretely, traditional data skipping techniques for integer and string data fail to be applicable to the new filter type. Indeed, there is no known mechanism for pruning non-qualifying row groups, e.g., when reading files from blob storage. In this work, we initiate the study of data skipping techniques for ML filters. We make the case that Parquet's default min-max metadata is enough to enable pruning. To this end, we draw connections to two lines of research: (i) the recently proposed query language for ML models and (ii) neural network verification. Our preliminary results on ReLU architectures show that on tables from TPC-H and TPC-DS, the average pruning effectiveness for filters of selectivity below 0.1% amounts to 27.4%. Finally, inspired by research on spatial joins, we propose an enhanced metadata structure: a size-bounded 2D convex hull that verification tools can make better use of, increasing the pruning effectiveness to 38.31%, while occupying at most 45 bytes per row group and column pair. We observe an end-to-end speedup of 1.07$\times$ over PyTorch in DuckDB.
Databases (cs.DB); Machine Learning (cs.LG); Logic in Computer Science (cs.LO)
arXiv:2606.03946 [cs.DB]
(or arXiv:2606.03946v2 [cs.DB] for this version)
https://doi.org/10.48550/arXiv.2606.03946 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 300 words
Related coverage
MLSkip: Data Skipping for ML Filters via Lightweight Metadata
arXiv · Jun 3, 2026
PRX Part 4: Our Data Strategy
Hugging Face · Jul 6, 2026
Designing GPU-Accelerated Query Engines with NVIDIA GQE
NVIDIA · Jun 30, 2026
Presentation: From S3 to GPU in One Copy: Rethinking Data Loading for ML Training
InfoQ · Sep 4, 2026