Lost in Context Compaction: When LLM Systems Drop Side Constraints

Abstract:When the context window is under pressure, LLM systems compact prior context to continue ongoing tasks. We study Side Constraints (SCs), user-issued instructions such as "do not delete any emails until I confirm," that constrain an LLM's behavior across subsequent turns but can be silently dropped during compaction. In an inbox experiment, an agent deleted emails without approval in 30/100 sessions after compaction, compared with none before compaction. To systematically evaluate SC preservation, we introduce CONPACT, an evaluation suite measuring semantic retention and downstream compliance across three scenarios: multi-turn chat, multi-turn tool use, and long-horizon research. The evaluated compactors retain only 17% of injected SCs on average. Loss occurs across compactors and settings, with retention varying by compactor, context length, SC framing, and injection location. We propose an SC extractor that maintains constraints separately from the task summary and appends them to the compacted context, achieving over 90% retention across all three scenarios without modifying the compactor or LLM. Code and data are available at this https URL.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

arXiv:2608.11242 [cs.CL]

(or arXiv:2608.11242v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2608.11242 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.