aiqbench analysis
Targeted user training is critical for unlocking the productive potential of generative AI in professional settings, as untrained access can lead to counterproductive outcomes like shorter outputs and increased errors. The study demonstrates that brief interventions significantly improve adoption rates and accuracy, suggesting that realizing productivity gains requires investment in both access and instruction rather than just tool availability. These findings challenge the assumption that GenAI automatically benefits all users, highlighting that higher-ability practitioners may opt out without proper guidance while lower-ability users adopt unproductively without it.
Training for Technology: Adoption and Productive Use of Generative AI in Legal Analysis
arXiv:2603.04982v3 Announce Type: replace-cross Abstract: Can targeted user training unlock the productive potential of generative artificial intelligence in professional settings? We study this question using a randomized experiment in which 164 law students completed an issue-spotting examination under one of three conditions: no GenAI access, optional access to a large language model (LLM), or LLM access with a brief training intervention. Untrained LLM access proved counterproductive: relative to participants without any LLM access, untrained users wrote significantly shorter answers, committed more case misstatements, and scored marginally lower, though most differences fall short of conventional significance. Training reversed this pattern. Trained participants adopted the LLM at higher rates (41% vs. 26%; p = 0.044), scored 0.27 grade points higher than untrained users--roughly one fine grade--(p = 0.027), and stated applicable rules more accurately (p = 0.014). Principal stratification analysis suggests training operates primarily through adoption rather than effectiveness--the adoption lower bound (1.06) exceeds the effectiveness upper bound (0.42) at strict mean dominance--though confidence intervals are wide. More broadly, these findings challenge the view that GenAI primarily benefits lower-skilled workers: without training, higher-ability practitioners opt out while lower-ability users adopt but unproductively. Realizing GenAI's productivity gains requires investment in both access and instruction.
Abstract:Can targeted user training unlock the productive potential of generative artificial intelligence in professional settings? We study this question using a randomized experiment in which 164 law students completed an issue-spotting examination under one of three conditions: no GenAI access, optional access to a large language model (LLM), or LLM access with a brief training intervention. Untrained LLM access proved counterproductive: relative to participants without any LLM access, untrained users wrote significantly shorter answers, committed more case misstatements, and scored marginally lower, though most differences fall short of conventional significance. Training reversed this pattern. Trained participants adopted the LLM at higher rates (41% vs. 26%; p = 0.044), scored 0.27 grade points higher than untrained users--roughly one fine grade--(p = 0.027), and stated applicable rules more accurately (p = 0.014). Principal stratification analysis suggests training operates primarily through adoption rather than effectiveness--the adoption lower bound (1.06) exceeds the effectiveness upper bound (0.42) at strict mean dominance--though confidence intervals are wide. More broadly, these findings challenge the view that GenAI primarily benefits lower-skilled workers: without training, higher-ability practitioners opt out while lower-ability users adopt but unproductively. Realizing GenAI's productivity gains requires investment in both access and instruction.
Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
arXiv:2603.04982 [cs.CY]
(or arXiv:2603.04982v3 [cs.CY] for this version)
From: Benjamin Chen Mr. [view email] [v1] Thu, 5 Mar 2026 09:23:30 UTC (767 KB) [v2] Tue, 31 Mar 2026 06:30:23 UTC (1,737 KB) [v3] Fri, 5 Jun 2026 08:57:56 UTC (773 KB)
Full text under the publisher's open licence · 247 words
Related coverage
Oxford Quantum Circuits Appoints Simon Phillips as Chief Product Officer
The Quantum Insider · Sep 24, 2026
Parallelization constraints could delay a technological singularity [Linkpost]
LessWrong · Jul 31, 2026
SEALSQ chip uses quantum security for India’s transit & digital payments
Quantum Zeitgeist · Aug 18, 2026
Quantum Computing Inc. Appoints Susan Hunt as CRO and Creates Chief Product Officer Role
The Quantum Insider · Jul 22, 2026