Failure Modes of Always-On Inter-Cluster Repulsion in Replay-Based Continual Learning

Abstract:Feature-space objectives are often added to replay-based continual learning systems with the expectation that better geometric separation will improve retention. We study a preliminary form of Cluster-Aware Replay (CAR) that combines a class-balanced replay memory with an always-active inter-cluster repulsion term (ICF). On five-task Split CIFAR-10 with a ResNet-18 backbone, the highest observed mean in a six-value sensitivity sweep reaches $22.5\pm1.4\%$ final average accuracy over three seeds, compared with $23.1\pm2.5\%$ for replay alone. ICF without replay reaches only $19.2\pm0.1\%$. All tested repulsion weights produce final accuracies between $20.1\%$ and $22.5\%$, and the detailed configuration exhibits $89.2\pm1.5$ percentage points of average forgetting. An instrumented rerun shows that the weighted repulsion contribution remains near $-0.13$ after cross-entropy has fallen close to zero, so the total objective becomes negative while old-task accuracy collapses. Importantly, the normalized distance objective is mathematically bounded; the failure is therefore better described as non-saturating, always-on repulsion rather than an unbounded loss. These negative results show that geometric separation is not automatically complementary to replay and motivate margin-gated objectives whose gradients deactivate once sufficient separation has been reached.

v3: Substantially revised as a negative-results study. All reported values recomputed from verified experimental runs; method description corrected to match the released implementation; statistical aggregation corrected to per-seed population standard deviations. 4 pages, 2 figures, 3 tables. Code: this https URL

Machine Learning (cs.LG)

arXiv:2510.07648 [cs.LG]

(or arXiv:2510.07648v3 [cs.LG] for this version)