Multimodal Remote Inference

Abstract:We consider a remote inference system with multiple modalities, where a multimodal machine learning (ML) model performs real-time inference using features collected from remote sensors. When sensor observations evolve dynamically over time, fresh features are critical for inference tasks. However, timely delivery of features from all modalities is often infeasible under limited network resources. To address this challenge, we formulate a multimodal scheduling problem to minimize the ML model's inference error. We model this error as a general function of the Age of Information (AoI) vector, where AoI quantifies data freshness. We cast the problem as a semi-Markov decision process (SMDP) and derive an equivalent reformulation with a reduced state set. We then show that the problem has fundamentally different chain structures in the two-modality and multi-modality cases. For the two-modality case, we prove that the optimal policy has an index-based threshold structure. For the general multi-modality case (two or more modalities), we develop an optimal switching-type policy (OPT-ST), which is computed using a multichain policy iteration algorithm (MPI). To further reduce complexity, we also develop two low-complexity policies: the round-robin-like switching-type policy (RR-ST) and the round-robin-like threshold policy (RR-Thres). Numerical results show that OPT-ST achieves the lowest inference error among all baselines, with reductions of up to 80.3%. In a five-modality case, RR-ST and RR-Thres achieve $5.2\times$ and $89.8\times$ speedups in offline computation over OPT-ST, respectively, at the cost of increasing the inference error by 23.3% and 38.6%, respectively.

Machine Learning (cs.LG); Information Theory (cs.IT); Networking and Internet Architecture (cs.NI)

arXiv:2508.07555 [cs.LG]

(or arXiv:2508.07555v4 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2508.07555 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.