SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton

Abstract:Generating symphonic music requires simultaneously managing high-level structural form and dense, multi-track orchestration, yet existing symbolic models often struggle with a "complexity-control imbalance" between scalability and steerability. We present SymphonyGen, a 3D hierarchical framework for contemporary orchestral generation, whose cascading decoders decompose the bar, track, and event axes, keeping decoding memory far below flat token streams and enabling conditioning at every structural level. A beat-quantized multi-pitch harmony skeleton, which may be user-written, analyzed, or model-generated, provides "short-score" conditioning, enabling outline control while producing orchestral textures. The model is refined with reinforcement learning against a cross-modal acoustic reward from CLaMP 3 audio embeddings, and a dissonance-averse sampling algorithm suppresses unintended tonal clashes during inference. Objective evaluations show that both post-training mechanisms reduce dissonance while maintaining independent melodic metrics, and in subjective tests SymphonyGen is rated above baseline systems in quality and preference, significantly so among general listeners.

Sound (cs.SD); Artificial Intelligence (cs.AI)

arXiv:2604.25498 [cs.SD]

(or arXiv:2604.25498v2 [cs.SD] for this version)

https://doi.org/10.48550/arXiv.2604.25498 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.