Canonical Procedural Actions: An Auditable Annotation Protocol for Tool-Use Agent Traces

Abstract:Tool-use agent traces identify messages and API calls, but procedural analyses also need explicit units of action and inspectable links to their evidence. We present Canonical Procedural Actions (CPAs), an annotation protocol that records a procedural function, its first agent-event anchor, the agent events that realize it, and separate contextual evidence. Multiple actions may share a message anchor without an inferred within-message order. A retail case study produces a versioned 24-entry codebook through open induction, recorded consolidation, and successive application audits. Two isolated LLM contexts annotate 32 trajectories disjoint from development at the trajectory level, producing 499 and 491 occurrences with anchor-label overlap A=0.982. Requiring identical context-event references reduces overlap to 0.798. These are structural repeatability measures, not semantic accuracy: 16 of 26 task IDs also occur in development, and historical tool payloads were truncated to 110 characters. Retrospective controls show that collapsing all labels raises overlap to 0.986, while simple endpoint rules reproduce the tool-anchored portion with 0.997 overlap. Assistant-message actions have 0.971 overlap, with a per-label minimum of 0.816. Applying the frozen codebook to 244 further trajectories yields 4,058 records, including eight diagnostic outcomes. The contribution is an explicit, auditable annotation instrument and a case study of its construction and measurement limits; human-reference validity and downstream utility remain to be established.

23 pages, 5 figures, 9 tables. Includes ancillary files for reproducing the reported analyses

Computation and Language (cs.CL)

arXiv:2609.24264 [cs.CL]

(or arXiv:2609.24264v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2609.24264 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

SOURCE_MANIFEST.json

analysis/INPUT_MANIFEST.json

analysis/PUBLIC_PATH_REDACTIONS.json

analysis/README.md

analysis/analyze.py

analysis/inputs/R1/SAMPLE_MANIFEST.json

analysis/inputs/R2/SAMPLE_MANIFEST.json

analysis/inputs/R3/AUDIT_PROVENANCE.json

analysis/inputs/R3/SAMPLE_MANIFEST.json

analysis/inputs/R4/AUDIT_PROVENANCE.json

analysis/inputs/R4/SAMPLE_MANIFEST.json

analysis/inputs/R5/AUDIT_PROVENANCE.json

analysis/inputs/R5/SAMPLE_MANIFEST.json

analysis/inputs/R6/APPLY_INSTRUCTIONS.md

analysis/inputs/R6/AUDIT_PROVENANCE.json

analysis/inputs/R6/C1/pack_1.txt

analysis/inputs/R6/C1/pack_2.txt

analysis/inputs/R6/C2/pack_1.txt

analysis/inputs/R6/C2/pack_2.txt

analysis/inputs/R6/CPA_LIBRARY_APPLY.md

analysis/inputs/R6/ROUND6_OCCURRENCES.jsonl

analysis/inputs/R6/ROUND6_RESULTS.json

analysis/inputs/R6/SAMPLE_MANIFEST.json

analysis/inputs/R6/V05_FREEZE_AUDIT_MANIFEST.json

analysis/inputs/R6/V05_FREEZE_DECISION.json

analysis/inputs/R6/packs/pack_1.txt

analysis/inputs/R6/packs/pack_2.txt

analysis/inputs/development/CONSOLIDATION_REPORT.md

analysis/inputs/development/CPA_LIBRARY_v0_1.json

analysis/inputs/development/CPA_LIBRARY_v0_5.json

analysis/inputs/development/FROZEN_SCHEMA_v1.md

analysis/inputs/historical_code/cpa4_build_round6.py

analysis/inputs/historical_code/cpa4_round2_analyze.py

analysis/inputs/historical_code/cpa4_round6_analyze.py

analysis/inputs/library/CPA_LIBRARY_v1_0.json

analysis/inputs/library/V1_FREEZE_MANIFEST.json

analysis/inputs/library/V1_FREEZE_REPORT.md

analysis/inputs/production/CPA_SEQUENCE_CORPUS.jsonl

analysis/inputs/production/P1/s01.txt

analysis/inputs/production/P1/s02.txt

analysis/inputs/production/P1/s03.txt

analysis/inputs/production/P1/s04.txt

analysis/inputs/production/P1/s05.txt

analysis/inputs/production/P1/s06.txt

analysis/inputs/production/P1/s07.txt

analysis/inputs/production/P1/s08.txt

analysis/inputs/production/P1/s09.txt

analysis/inputs/production/P1/s10.txt

analysis/inputs/production/P1/s11.txt

analysis/inputs/production/P1/s12.txt

analysis/inputs/production/P1/s13.txt

analysis/inputs/production/P1/s14.txt

analysis/inputs/production/P1/s15.txt

analysis/inputs/production/P1/s16.txt

analysis/inputs/production/PRODUCTION_ANNOTATION_QA.json

analysis/inputs/production/SHARD_MANIFEST.json

analysis/inputs/production/packs/s01.txt

analysis/inputs/production/packs/s02.txt

analysis/inputs/production/packs/s03.txt

analysis/inputs/production/packs/s04.txt

analysis/inputs/production/packs/s05.txt

analysis/inputs/production/packs/s06.txt

analysis/inputs/production/packs/s07.txt

analysis/inputs/production/packs/s08.txt

analysis/inputs/production/packs/s09.txt

analysis/inputs/production/packs/s10.txt

analysis/inputs/production/packs/s11.txt

analysis/inputs/production/packs/s12.txt

analysis/inputs/production/packs/s13.txt

analysis/inputs/production/packs/s14.txt

analysis/inputs/production/packs/s15.txt

analysis/inputs/production/packs/s16.txt

analysis/outputs/anchor_strata.csv

analysis/outputs/baselines.csv

analysis/outputs/coarse_mapping.json

analysis/outputs/disagreement_anchors.json

analysis/outputs/granularity.csv

analysis/outputs/per_label.csv

analysis/outputs/query_demo.json

analysis/outputs/summary.json

analysis/outputs/tool_only_baselines.csv

analysis/outputs/tool_rule_mapping.json

analysis/query_demo.py

analysis/test_analysis.py

analysis/test_reference_overlap.py

(81 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.