Over-fitting Control in Deep Nonlinear Matrix Completion via Gradual Nonsmooth Regularization

Abstract:Conventional matrix completion methods exploit the underlying low-rank structure of a target matrix to recover its missing entries, but may be inadequate for capturing complex nonlinear dependencies. Deep neural networks have therefore been increasingly adopted for nonlinear matrix completion (NLMC). However, their high representational capacity, particularly in deep fully connected neural networks (FCNNs), can lead to over-fitting when training is restricted to incomplete observations. In this paper, we propose a Deep Neural Network with Nonsmooth Regularization (DNN-NSR) framework to address this issue by incorporating two complementary structural priors: sparsity of intermediate representations and low-rankness of layer-wise weight matrices. Specifically, the $\ell_1$-norm of hidden representations promotes sparse latent features, while the nuclear norm of weight matrices encourages low-rank layer-wise transformations. To optimize the resulting nonsmooth and nonconvex objective, we develop an extrapolated proximal gradient algorithm that employs soft-thresholding and singular-value thresholding to handle the two regularization terms, respectively. We further introduce a gradual regularization strategy that progressively increases the influence of these structural penalties during training. Under suitable assumptions, we establish convergence properties of the proposed algorithm toward a critical point of the training objective. The effectiveness of DNN-NSR is evaluated through extensive experiments on synthetic matrix recovery, single-image inpainting, and recommender systems using the MovieLens 100K and MovieLens 1M datasets. Comparisons with representative linear and nonlinear matrix completion methods demonstrate the potential of the proposed framework to improve recovery performance by jointly exploiting sparse latent representations and low-rank network transformations.

Information Theory (cs.IT); Machine Learning (cs.LG)

arXiv:2403.10232 [cs.IT]

(or arXiv:2403.10232v3 [cs.IT] for this version)

https://doi.org/10.48550/arXiv.2403.10232 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.