Skip to content
aiqbench
aiqbench takeMachine Learning

aiqbench analysis

This theoretical paper shows that spectral summaries and scalar training-loss curves can fail to distinguish centered quadratic networks, even when input marginals match exactly. It also gives sample-complexity bounds for when loss queries or initialization interventions can recover a hidden teacher, with the caveat that the authors report severe numerical ill-conditioning. For ML practitioners, the main takeaway is that common diagnostics like loss curves and spectral statistics may not identify the underlying model without additional structure or carefully designed probes.
Source coveringarXiv
What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Spectral Indistinguishability and Learning-Curve Identifiability in Quadratic Networks

arXiv3 min read

arXiv:2610.20678v1 Announce Type: new Abstract: We study what spectral summaries and scalar learning curves reveal about centered quadratic networks. We construct analytic positive input densities with exactly Gaussian one- and two-coordinate marginals. Their complete fourth-moment operators are intertwined by an orthogonal map fixing the entire diagonal teacher algebra, so the corresponding anchored spectral measures agree. For selected common teachers with a nonzero cycle invariant, their factorized loss curves differ because the map does not preserve matrix multiplication. Cycle length $r$, which can be arbitrarily large, gives $O(\varepsilon^r)$ contrast uniformly in training time for each fixed construction. For a supplied pair, noisy scalar loss queries require order $\varepsilon^{-2r}$ observations, while bounded initialization interventions with scalar gradient readouts require order $\varepsilon^{-2}$. A shared-input score, equal to a paired-gradient difference, attains the optimal input-sample exponent without knowing the teacher. An explicit family of diverging auxiliary teacher scales instead admits a nonzero finite limiting contrast at coupling $1/\log L$. Conversely, for Gaussian inputs and a rationally independent teacher spectrum, one exact scalar loss curve identifies the initial predictor up to sign conjugation. Within this rationally independent class, with nonzero diagonal inverse-state error in every coordinate and calibrated time and loss scales, it also identifies the unknown teacher spectrum and active dimension, modulo simultaneous rotations. For fixed dimension and distinct unordered teacher sums, dimension-minimal exact observations give generic joint local recovery. Finite exact global bounds hold for a known teacher and, on the active class, under a blind dimension cap. Exact certificates and full-weight replay verify the construction; numerical inverse checks reveal severe ill-conditioning.

Abstract:We study what spectral summaries and scalar learning curves reveal about centered quadratic networks. We construct analytic positive input densities with exactly Gaussian one- and two-coordinate marginals. Their complete fourth-moment operators are intertwined by an orthogonal map fixing the entire diagonal teacher algebra, so the corresponding anchored spectral measures agree. For selected common teachers with a nonzero cycle invariant, their factorized loss curves differ because the map does not preserve matrix multiplication. Cycle length $r$, which can be arbitrarily large, gives $O(\varepsilon^r)$ contrast uniformly in training time for each fixed construction. For a supplied pair, noisy scalar loss queries require order $\varepsilon^{-2r}$ observations, while bounded initialization interventions with scalar gradient readouts require order $\varepsilon^{-2}$. A shared-input score, equal to a paired-gradient difference, attains the optimal input-sample exponent without knowing the teacher. An explicit family of diverging auxiliary teacher scales instead admits a nonzero finite limiting contrast at coupling $1/\log L$. Conversely, for Gaussian inputs and a rationally independent teacher spectrum, one exact scalar loss curve identifies the initial predictor up to sign conjugation. Within this rationally independent class, with nonzero diagonal inverse-state error in every coordinate and calibrated time and loss scales, it also identifies the unknown teacher spectrum and active dimension, modulo simultaneous rotations. For fixed dimension and distinct unordered teacher sums, dimension-minimal exact observations give generic joint local recovery. Finite exact global bounds hold for a known teacher and, on the active class, under a blind dimension cap. Exact certificates and full-weight replay verify the construction; numerical inverse checks reveal severe ill-conditioning.

37 pages, 2 figures. Code and synthetic data provided as ancillary material

Machine Learning (cs.LG); Machine Learning (stat.ML)

arXiv:2610.20678 [cs.LG]

(or arXiv:2610.20678v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.20678 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

neural_information_sculpting_v12/ARCHIVE_README.txt

neural_information_sculpting_v12/FILE_MANIFEST.json

neural_information_sculpting_v12/geometry_response_extension/protocols/baseline.json

neural_information_sculpting_v12/geometry_response_extension/protocols/research_protocol.md

neural_information_sculpting_v12/geometry_response_extension/results/amplification_check.json

neural_information_sculpting_v12/geometry_response_extension/results/analytic_realization_check.json

neural_information_sculpting_v12/geometry_response_extension/results/blind_identifiability_check.json

neural_information_sculpting_v12/geometry_response_extension/results/finite_time_probe.json

neural_information_sculpting_v12/geometry_response_extension/results/gh3_exact_witness.json

neural_information_sculpting_v12/geometry_response_extension/results/identifiability_check.json

neural_information_sculpting_v12/geometry_response_extension/results/lp_dual_certificate.json

neural_information_sculpting_v12/geometry_response_extension/results/moment_feasibility.json

neural_information_sculpting_v12/geometry_response_extension/results/scalar_intervention_check.json

neural_information_sculpting_v12/geometry_response_extension/results/smooth_realization_check.json

neural_information_sculpting_v12/geometry_response_extension/results/teacher_grid.json

neural_information_sculpting_v12/geometry_response_extension/results/validation_geometry.json

neural_information_sculpting_v12/geometry_response_extension/results/validation_geometry.npz.b64

neural_information_sculpting_v12/geometry_response_extension/src/certify_moment_optimum.py

neural_information_sculpting_v12/geometry_response_extension/src/check_amplification.py

neural_information_sculpting_v12/geometry_response_extension/src/check_analytic_realization.py

neural_information_sculpting_v12/geometry_response_extension/src/check_blind_identifiability.py

neural_information_sculpting_v12/geometry_response_extension/src/check_identifiability.py

neural_information_sculpting_v12/geometry_response_extension/src/check_scalar_interventions.py

neural_information_sculpting_v12/geometry_response_extension/src/check_smooth_realization.py

neural_information_sculpting_v12/geometry_response_extension/src/explore_teacher_scales.py

neural_information_sculpting_v12/geometry_response_extension/src/moment_feasibility.py

neural_information_sculpting_v12/geometry_response_extension/src/validate_geometry.py

neural_information_sculpting_v12/isospectral_training_extension/protocols/isospectral_discrete_gradient_v1.json

neural_information_sculpting_v12/isospectral_training_extension/protocols/isospectral_discrete_gradient_v1.sha256

neural_information_sculpting_v12/isospectral_training_extension/protocols/isospectral_original_training_v1.json

neural_information_sculpting_v12/isospectral_training_extension/protocols/isospectral_original_training_v1.sha256

neural_information_sculpting_v12/isospectral_training_extension/src/discrete.py

neural_information_sculpting_v12/isospectral_training_extension/src/isospectral.py

neural_information_sculpting_v12/isospectral_training_extension/src/run_discrete.py

neural_information_sculpting_v12/isospectral_training_extension/src/validate_isospectral.py

neural_information_sculpting_v12/isospectral_training_extension/tests/test_isospectral.py

neural_information_sculpting_v12/manuscript_geometry_learning_v12/CLAIM_EVIDENCE.md

neural_information_sculpting_v12/manuscript_geometry_learning_v12/LICENSE-DATA.txt

neural_information_sculpting_v12/manuscript_geometry_learning_v12/LICENSE-MIT.txt

neural_information_sculpting_v12/manuscript_geometry_learning_v12/LICENSE_SCOPE.md

neural_information_sculpting_v12/manuscript_geometry_learning_v12/README.md

neural_information_sculpting_v12/manuscript_geometry_learning_v12/REPRODUCIBILITY.md

neural_information_sculpting_v12/manuscript_geometry_learning_v12/figures/curve_identifiability.pdf

neural_information_sculpting_v12/manuscript_geometry_learning_v12/figures/geometry_response.pdf

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/added_proof_algebra.json

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/build_stdout.txt

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/environment.json

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/numerical_checks_summary.json

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/numerical_logs/check_analytic_realization.txt

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/numerical_logs/check_blind_identifiability.txt

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/numerical_logs/check_identifiability.txt

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/numerical_logs/check_scalar_interventions.txt

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/numerical_logs/inherited_tests.txt

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/numerical_result_comparison.json

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/pdf_geometry_audit.json

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/source_archive_validation.json

neural_information_sculpting_v12/manuscript_geometry_learning_v12/qa/visual_audit.json

neural_information_sculpting_v12/manuscript_geometry_learning_v12/requirements_repro.txt

neural_information_sculpting_v12/manuscript_geometry_learning_v12/scripts/check_added_proof_algebra.py

neural_information_sculpting_v12/manuscript_geometry_learning_v12/scripts/make_figures.py

neural_information_sculpting_v12/restore_data.py

(56 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Full text under the publisher's open licence · 438 words

Read at arXiv ↗Ask about this article

Related coverage

AI Research

How do neural networks regress cubic polynomials? Apparently, they use a trick invented in Milan 500 years ago

LessWrong · Jul 22, 2026

Quantum Comms

Lattice-Based Cryptography Explained

Quantum Zeitgeist · Aug 15, 2026

AI Research

Quotient Dynamics, Effective Curvature, and Implicit Bias in Positive Quadratic Networks

arXiv · Jul 29, 2026

AI Research

PIRAMID: Progress and Plans

LessWrong · Jul 27, 2026