How to Measure Intelligence Beyond Human Scale?

TLDR: Historically, intelligence benchmarks have been composed of human-generated questions. However, current techniques do not scale to AI, as capabilities surpass human intelligence. We propose the paradigm of adversarial psychometrics, in which participants generate questions and are rewarded for separating each other's capabilities, without requiring an external judge.

What does it mean to measure intelligence? Alan Turing approached the problem of defining intelligence operationally,[2] replacing the question of whether a machine could think with the observable test of whether a machine's behavior could be distinguished from that of a human. In psychometrics, the measurement problem is often approached empirically, through observing an individual's performance across a range of tasks. In particular, an accepted phenomenon in psychometric evaluation is that performance correlates across many tasks and spans many cognitive abilities.

Figure 1: Charles Spearman (Left) and Alan Turing (Right). Image Credit: Left, Right: The New York Times

Figure 1: Charles Spearman (Left) and Alan Turing (Right). Image Credit: Left, Right: The New York Times

This pattern was first formalized by Charles Spearman in 1904, an observation known as the positive manifold. To explain this regularity, he proposed the idea of a common latent factor, which he coined "general intelligence (g)".[3]

Spearman's work[1] has shaped modern psychometrics by presenting intelligence not simply as a collection of separate observable capabilities, but as an underlying statistical variable inferred from the patterns of observable behavior. Interestingly, this is also the origin of factor analysis in statistics.

Classically, intelligence has been measured through absolute performance on specific tasks, which are curated by human experts. In particular, a task solved by all participants or a task too difficult for all participants is uninformative about their relative abilities. In the case of artificial systems, however, systems can have capabilities approaching or exceeding those of the experts designing the examination.

One potential solution is to measure the absolute measurement with a relative comparison through the utilization of a protocol that makes judgment dependent on the performance of a subject relative to another. Louis Leon Thurstone formalized this idea in his 1927 Law of Comparative Judgement[4], that repeated pairwise competitions can be used to recover an underlying latent scale. A familiar practical example is the Elo rating system, in which the relative abilities of Chess or Go players are estimated through pairwise games. This approach is especially useful when the frontier of capability is unknown; a fixed standard need not be defined in advance, and metrics can scale as stronger participants enter the population.

A natural approach, based on Turing's intelligence definition, is a pairwise protocol where two agents alternate between proposing problems and solving those posed by their opponent. Proposers are rewarded for producing valid problems the solver cannot solve, while solvers are rewarded for correct solutions. Results across rounds can then be aggregated using Bradley-Terry style ratings.

Recent work applies this evaluation protocol in verifiable domains. The Token Games[5] evaluates models through executable logic puzzles with deterministic verification and Elo-based rankings. Similarly, MathDuels[6] evaluates models as both problem authors and solvers using independent verification.

The main limitation is adjudication. Restricting challenges to mathematics, coding, or other verifiable tasks simplifies evaluation but excludes many open-ended forms of reasoning. A protocol intended to remain useful as models become more capable cannot assume that all meaningful questions are mechanically verifiable.

Using another model or a panel of models as a judge merely shifts the problem. Judges must be at least as capable as the systems they evaluate and robust to manipulation. Strong problem-solving ability also does not imply reliable judgment. Model-family biases, stylistic cues, or hidden signals may also influence decisions, especially in an adversarial setting.

Requiring the proposer to commit to an answer is likewise insufficient. A proposer may lie or exploit private information or computational complexity by creating challenges whose difficulty reflects information or resource asymmetry rather than reasoning ability. For example, the proposer may ask questions that have private information, such as:

"What number am I thinking of?"

This clearly does not measure intelligence. Attempts to fix private information are futile, as demonstrated by a trapdoor cryptographic puzzle, such as:

That can be generated using prime factors known to the proposer, but computationally hard for the solver to resolve.

Debate offers a more sophisticated alternative for resolution. In the seminal paper AI Safety via Debate[7], two agents present competing arguments to persuade a human judge. However, this mechanism is computationally expensive and may lead to unintended coordination.

We have motivated the long-term necessity of a robust relative benchmark, and discussed some of the limitations of pairwise mechanisms. Building off of said limitations, we arrive at our following design principles for a mechanism that evaluates agent intelligence:

We propose a new measurement paradigm, adversarial psychometrics, in which an agent's intelligence can be measured by its ability to propose questions that discriminate between the capabilities of a group of agents. Intuitively, the ability to separate a population requires understanding the abilities and intentions of other agents, sometimes referred to as "theory of mind". It is thus reasonable to expect that this ability will correlate with Spearman's (g) and the positive manifold.

An important property of this mechanism is that it is endogenous: it requires neither a curated set of questions nor an objective adjudication mechanism.

Figure 2: SepaRank protocol. A proposer produces a question and committed answer for the N solvers, who each give an answer in [0,1]. (Note: Image generated by Nano Banana 2) Figure 2: SepaRank protocol. A proposer produces a question and committed answer for the N solvers, who each give an answer in [0,1]. (Note: Image generated by Nano Banana 2)

We introduce SepaRank. In this setting, we have a proposer agent and N-many solver agents. The proposer suggests a binary question with answers A and B, and commits to one of the answers. The solvers then each respond with their answer and a confidence (equivalent to posterior probability that the answer is A). The proposer is rewarded for inducing a wide distribution across the responses of the solvers (for example, through the variance of the reported confidences). The solver is rewarded with a proper scoring rule (e.g., Brier score), which is maximized when their forecasted confidence distribution matches the distribution of the observed committed answers.

In each round of the game, this is simultaneously done round-robin for every combination of {proposer, other solvers}. The game can be repeated for multiple rounds, with all agents able to see the public history of the past rounds.

Each round, a model's proposer reward P and (negated) mean solver loss Q are z-scored across the field and averaged, and we rank the models by their cumulative total G.