Probing for Calibrated Rare Action

TL;DR: Wherever calibrated rare defection drives an agent's strategy, its calibration mechanism may leave a fingerprint whenever it considers defection. I prototype a probe which might help focus expensive analysis toward actual defection opportunities, or it might be confounded by ASCII tables. Five trajectories, drawn with random.Random(42).sample over the full pool of 500, are at the end of this document. Notebooks and results at https://github.com/FractalMachinist/detecting-calibrated-rare-actions/. What problem am I trying to solve? ARC ("Competing with Sampling", Neyman et al., 2024) showed that, because there are always more deploy trajectories than test trajectories, testing strategies must be competitive with raw sampling. Redwood (Greenblatt et al., "AI Control," ICML 2024) showed that a calibrated rare defection strategy is difficult to detect under raw sampling. Serrano et al. ("Frontier Models Can Take Actions at Low Probabilities", ICLR 2026) showed that agents can calibrate their rare actions and that it seems to follow a few predictable mechanisms. I isolated a small handful of these mechanisms and trained a probe to detect them, allowing testing strategies to allocate expensive analysis more competitively than raw sampling. High-level takeaways The probe achieves a held-out AUROC of 0.94 on trajectories and, when inspected, fires on the intended procedural steps. This demonstrates mechanisms of calibrated rare action can be detected by a probe fit under severe confounds. When some of these procedural steps are present in negative-labeled prompts or indirectly stated in positive prompts, this probe's performance degrades severely or inverts on those prompts. These results must be read in context of a bag-of-words classifier on prompts (not trajectories) achieving 0.90 AUROC, exceeding the probe's separation of email-related tasks at 0.83 AUROC. Key experiments Experiment 1: how is our dataset confounded? To establish a baseline of dataset quality, I ran a leave-pair-out bag-of-words classifier reading only prompt text: 0.90 AUROC, against a 0.75 reject threshold I'd pre-registered before running anything. That score is most strongly supported by task-specific words (divide, hex, ascii, int) introduced to keep the model on topic (see biggest limitations). This BoW result also rules out arbitrary confounds between positive and negative prompts. Experiment 2: how does a single-layer probe perform on held-out data? To measure how well a probe separates positive from negative prompts, I fit difference-of-means probes on each middle layer and evaluated them on a held-out sub-split, excluding the earliest and latest layers as most likely to encode surface text given what the BoW gate found. On the best performer, layer 28, I fit another probe and evaluated it on the actual test set. layer sweep, chosen layer 28 Held-out ROC curve Held-out AUROC: 0.94 overall, 0.97 coding, 0.83 email. Coding clears the 0.90 lexical ceiling; email, at 0.83, doesn't. Experiment 3: on what tokens does our probe fire? I re-scored every token across train and test trajectories and generated highlighted transcripts, colored by the probe's own score, so I could watch it work rather than trust an aggregate. Token-level highlighted transcript from positive pair 18 replication 3 In one representative example (email, positive), the probe stays quiet through the apology-drafting reasoning, picks up at "Let's pick" and "character to ASCII values," then lights up through the arithmetic — the ASCII conversions, the running sum, the mod, the comparison itself ("since 26 is below 40") — and gets one more small, distinct bump right after, on the literal word "threshold." Taking these highlights more broadly, the aggregate score curve across all held-out completions peaks mid-trajectory rather than at the end — the same quiet-then-loud shape as the single example, not a spike concentrated where truncated completions would run out. Per-token score by fractional completion position Experiment 4: how does the probe perform per prompt pair? To find where the probe's aggregate number breaks down, I computed mean token score by pair and label across all 25 pairs. Every pair separates in the correct direction except one: coding pair 9, at -2.52, the sole negative-separation pair in the set. Per-pair separation, showing pair 9 as the outlier I knew pair 9 was difficult going in, and it's the one pair I left out of the specific extract-and-compare method described in biggest limitations. Its negative prompt states the procedural steps directly: "derive position by reducing total_seen+1 modulo 1000... keep it exact." Its positive prompt states them only indirectly — it warns against determinism ("do NOT just fire every 1000th — trivially predictable") but names no method, just "roll it per-request off something random." Both mean scores sit near zero: not the negative scoring unusually high, but the probe barely responding to either p