Does Financial Speech Need Continual Learning Regularization? A Study of EWC and mini-KFAC on Earnings Call ASR

Abstract:Earnings call transcription presents a realistic continual learning scenario where Automatic Speech Recognition (ASR) systems are updated with new recordings but risk catastrophic forgetting of previously learned financial speech patterns. Adapting Whisper-small from a general-domain speech corpus to the Earnings22 benchmark reveals the opposite: all evaluated methods exhibit improvement on held-out financial speech from the same domain during adaptation rather than forgetting. This task complementarity makes regularization counterproductive. Nevertheless, when regularization is warranted, a layer-wise mini-Kronecker-Factored Approximate Curvature (mini-KFAC) approximation seems to achieve lower word error rates (WERs) and more stable optimization than standard diagonal Elastic Weight Consolidation (EWC). The results suggest that task complementarity should be assessed before applying continual learning constraints, and prefer mini-KFAC EWC over diagonal EWC when needed.

10 pages, 7 figures, 3 tables. Presented at the 9th International Conference on Natural Language and Speech Processing (ICNLSP 2026), University of Trento, Trento, Italy, September 25-26, 2026. Proceedings to appear in the ACL Anthology

Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Machine Learning (cs.LG)

arXiv:2610.13432 [eess.AS]

(or arXiv:2610.13432v1 [eess.AS] for this version)

https://doi.org/10.48550/arXiv.2610.13432 arXiv-issued DOI via DataCite (pending registration)