aiqbench analysis
This research evaluates GPT-3.5 as a tool for counterfactual analysis in online lending, demonstrating that prompt engineering can narrow the performance gap between LLMs and traditional gradient-boosted regression models. While the study confirms logical coherence in these predictions, it highlights current limitations where standard machine learning algorithms still outperform large language models by roughly 0.64% in R-squared metrics for this specific financial task. The findings suggest that while LLMs offer a flexible interface for causal reasoning, they are not yet ready to fully replace specialized statistical methods without further architectural improvements or hybrid approaches.
Counterfactual Analysis via Large Language Models
arXiv:2608.05367v1 Announce Type: new Abstract: Counterfactual analysis aims to predict potential outcomes under hypothetical scenarios, offering valuable insights for decision-making. This paper investigates the application of large language models (LLMs), specifically the GPT-3.5 model, for counterfactual analysis. We focus on the online lending context, where the counterfactual return on investment (ROI) is crucial for evaluating different interest rate schemes. We begin by assessing the predictive performance of GPT and comparing it with advanced machine learning algorithms. The results show that prompt engineering can significantly enhance GPT's predictions, with the R-squared increasing from 1.97% to 2.84%, closely approaching the 3.48% achieved by gradient-boosted regression. Subsequently, we utilize GPT to generate counterfactual ROIs under a set of alternative interest rates. GPT exhibits logical coherence and causal reasoning in its responses. The findings underscore the potential of LLMs as effective tools for counterfactual analysis in online lending, suggesting broader applications for LLMs in various predictive and decision-making contexts.
Abstract:Counterfactual analysis aims to predict potential outcomes under hypothetical scenarios, offering valuable insights for decision-making. This paper investigates the application of large language models (LLMs), specifically the GPT-3.5 model, for counterfactual analysis. We focus on the online lending context, where the counterfactual return on investment (ROI) is crucial for evaluating different interest rate schemes. We begin by assessing the predictive performance of GPT and comparing it with advanced machine learning algorithms. The results show that prompt engineering can significantly enhance GPT's predictions, with the R-squared increasing from 1.97% to 2.84%, closely approaching the 3.48% achieved by gradient-boosted regression. Subsequently, we utilize GPT to generate counterfactual ROIs under a set of alternative interest rates. GPT exhibits logical coherence and causal reasoning in its responses. The findings underscore the potential of LLMs as effective tools for counterfactual analysis in online lending, suggesting broader applications for LLMs in various predictive and decision-making contexts.
Artificial Intelligence (cs.AI); General Finance (q-fin.GN)
arXiv:2608.05367 [cs.AI]
(or arXiv:2608.05367v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.05367 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 247 words
Related coverage
Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values
AI Alignment Forum · Jul 31, 2026
Scale, Drivers, and Limits of Representational Alignment Across Large Language Models
arXiv · Oct 15, 2026
An Empirical Study of Counterfactual Self-Explanations in LLMs
arXiv · Sep 16, 2026
Cross-Cultural Value Attribution in Large Vision-Language Models
arXiv · Oct 7, 2026