aiqbench analysis
This paper applies a neural architecture to piano transcription from top-view video, a setting where audio problems like sustain pedal noise and missing soundtracks don't apply. Its main idea is to use the keyboard's known geometry, splitting the image into 88 fixed regions with one small recurrent network shared across keys, instead of a large global model that must learn key locations. Reported results show accuracy close to the prior state of the art on PianoVAM with about 5% of the parameters and roughly 4.7x faster inference, but these come from a single benchmark, and cross-dataset performance on PianoYT still needs fine-tuning. For ML readers, the transferable lesson is that encoding known structure can cut model size without losing accuracy.
KeyWise: Efficient Video-Only Piano Transcription with a Lightweight Per-Key Architecture
arXiv:2611.04476v1 Announce Type: cross Abstract: Audio-based piano transcription is hampered by the sustain pedal and by noisy or missing soundtracks, problems that video-only transcription avoids, since a top-view video shows directly when each key is pressed and released. Most visual transcription networks, however, describe the whole keyboard with one global feature vector and must learn where each key is. In this paper, we propose KeyWise, a lightweight per-key architecture that exploits the known geometry of the keyboard instead of learning it. The rectified keyboard is divided into 88 equal tiles that serve as regions of interest (ROIs), and one small recurrent network, shared by all keys, follows each key over time. On PianoVAM, KeyWise reaches the onset accuracy of the state-of-the-art V2N (95.0% vs. 94.7% onset F1 at 50 ms and 97.8% vs. 97.0% at 100 ms) with 1.3 M instead of 28.2 M parameters, and it runs 4.7 times faster on the same GPU. At 100 ms, it is also significantly more accurate when the key releases (97.0% vs. 95.5% F1) or the velocities (89.4% vs. 84.5% F1) must be correct as well. On PianoYT, KeyWise pre-trained on PianoVAM reaches 69.4% onset F1 at 100 ms, comparable to the best published result, although it needs fine-tuning for new recording conditions.
Abstract:Audio-based piano transcription is hampered by the sustain pedal and by noisy or missing soundtracks, problems that video-only transcription avoids, since a top-view video shows directly when each key is pressed and released. Most visual transcription networks, however, describe the whole keyboard with one global feature vector and must learn where each key is. In this paper, we propose KeyWise, a lightweight per-key architecture that exploits the known geometry of the keyboard instead of learning it. The rectified keyboard is divided into 88 equal tiles that serve as regions of interest (ROIs), and one small recurrent network, shared by all keys, follows each key over time. On PianoVAM, KeyWise reaches the onset accuracy of the state-of-the-art V2N (95.0% vs. 94.7% onset F1 at 50 ms and 97.8% vs. 97.0% at 100 ms) with 1.3 M instead of 28.2 M parameters, and it runs 4.7 times faster on the same GPU. At 100 ms, it is also significantly more accurate when the key releases (97.0% vs. 95.5% F1) or the velocities (89.4% vs. 84.5% F1) must be correct as well. On PianoYT, KeyWise pre-trained on PianoVAM reaches 69.4% onset F1 at 100 ms, comparable to the best published result, although it needs fine-tuning for new recording conditions.
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2611.04476 [cs.CV]
(or arXiv:2611.04476v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2611.04476 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 306 words
Related coverage
GADR: Gathering Architecture Decision Records from Meeting Transcriptions
arXiv · Aug 19, 2026
HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning
arXiv · Nov 4, 2026
What the Waveform Knows: Transparent-first Speech and Audio Intelligence with Caption Studio
arXiv · Jul 23, 2026
FusionML: Prefill, Not Decode - Mechanism and Boundaries of CPU+GPU Co-Execution on Unified-Memory Apple Silicon
arXiv · Jul 28, 2026