analiza aiqbench
This paper proposes a minimal analytical model to distinguish between low-level rule adjustments and high-level revisions of evaluation criteria within self-modifying AI systems. By separating the depth of modification from its reflexivity, the framework addresses critical gaps in comparing artificial policy updates with human metacognitive processes. The work offers a theoretical foundation for understanding system boundaries and identity continuity as autonomous agents evolve their own operational parameters.
What Does a System Modify When It Modifies Itself?
arXiv:2603.27611v2 Announce Type: replace Abstract: When a cognitive system modifies its own functioning, what exactly does it modify: a low-level rule, a control rule, or the criterion that evaluates its revisions? Cognitive science describes executive control, metacognition, and hierarchical learning, while artificial intelligence modifies policies, parameters, and learning mechanisms, but the two fields lack common criteria for comparing these transformations. We propose a minimal analytical model distinguishing functional rules Phi_t = {R0, ..., Rk}, modification mechanisms Mt, and evaluation criteria Nt. It separates two independent dimensions: the depth of the modified target and whether the modification is blind or reflexive. Internal representational access AtR and endogenous causal control AtC are further distinguished from external inspectability and modifiability by a designer. Four regimes are defined by the target of modification: action without organizational modification, modification of low-level rules, modification of control rules or mechanisms, and revision of the evaluation criterion. Each is related to cognitive phenomena and artificial systems while making explicit the system boundary on which attribution of endogenous self-modification depends. The comparison yields a conditional crossed-opacities hypothesis: humans often have richer endogenous self-description at abstract and strategic levels than at implementation levels, whereas current artificial systems may be externally inspectable and modifiable at operational levels without corresponding self-representation or endogenous control. The framework also identifies three difficulties - viability, evaluation of revisions to evaluation criteria, and continuity of identity - and proposes two empirical discriminants for identifying the target of a modification and the causal role of self-representation.
Abstract:When a cognitive system modifies its own functioning, what exactly does it modify: a low-level rule, a control rule, or the criterion that evaluates its revisions? Cognitive science describes executive control, metacognition, and hierarchical learning, while artificial intelligence modifies policies, parameters, and learning mechanisms, but the two fields lack common criteria for comparing these transformations. We propose a minimal analytical model distinguishing functional rules Phi_t = {R0, ..., Rk}, modification mechanisms Mt, and evaluation criteria Nt. It separates two independent dimensions: the depth of the modified target and whether the modification is blind or reflexive. Internal representational access AtR and endogenous causal control AtC are further distinguished from external inspectability and modifiability by a designer. Four regimes are defined by the target of modification: action without organizational modification, modification of low-level rules, modification of control rules or mechanisms, and revision of the evaluation criterion. Each is related to cognitive phenomena and artificial systems while making explicit the system boundary on which attribution of endogenous self-modification depends. The comparison yields a conditional crossed-opacities hypothesis: humans often have richer endogenous self-description at abstract and strategic levels than at implementation levels, whereas current artificial systems may be externally inspectable and modifiable at operational levels without corresponding self-representation or endogenous control. The framework also identifies three difficulties - viability, evaluation of revisions to evaluation criteria, and continuity of identity - and proposes two empirical discriminants for identifying the target of a modification and the causal role of self-representation.
19 pages, 1 figure, 3 tables. Version 2 is a substantially rewritten and condensed journal-submission version. Version 1 (35 pages), which remains publicly available, contains the original longer formulation of the framework
Artificial Intelligence (cs.AI); Neurons and Cognition (q-bio.NC)
arXiv:2603.27611 [cs.AI]
(or arXiv:2603.27611v2 [cs.AI] for this version)
Text integral sub licența deschisă a publicației · 291 cuvinte
Subiecte conexe
Your software should build itself
LessWrong · 25 iul. 2026
Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident
METR · 26 aug. 2026
Gate AI Training, Not Just Releases
LessWrong · 28 sept. 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · 17 sept. 2026