VirnyFlow: Optimizing ML Pipelines for Accuracy, Fairness, and Stability at Scale

Abstract:Developing machine learning (ML) systems for real-world deployment requires navigating context-dependent trade-offs among accuracy, fairness, stability, and other objectives. Existing AutoML frameworks optimize pipelines efficiently, but they fix the optimization objective up front, leave it outside the developer's control during search, and rarely scale beyond a single node. We present VirnyFlow, a system that optimizes ML pipelines jointly for accuracy, fairness, and stability at scale. A user-defined evaluation protocol, with fairness measured over binary and intersectional groups, drives every layer of the optimizer: multi-objective Bayesian optimization of physical pipelines, cost-aware bandit selection of logical pipelines, and multi-criterion pruning. The architecture combines asynchronous execution over Apache Kafka with database-backed experiment management, providing fine-grained parallelism, fault tolerance, and interactive inspection of trade-offs. On six real-world datasets, VirnyFlow achieves competitive or superior performance compared to state-of-the-art AutoML systems (auto-sklearn, Alpine Meadow, FLAML) under identical resource constraints, scales to 128 workers across four nodes on datasets of up to 2.6M records, and achieves up to 7x higher speedup than the best-scaling single-node baseline, while maintaining stable accuracy and fairness as parallelism increases. A clinical case study on distribution shift and an IRB-approved user study demonstrate human-in-the-loop navigation of trade-offs in practice: rather than returning a single "best" model, VirnyFlow lets data scientists define, inspect, and iteratively refine the objectives of the search to fit their deployment context.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)

arXiv:2506.01584 [cs.LG]

(or arXiv:2506.01584v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2506.01584 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.