aiqbench analysis
This study exposes a critical fragility in LLM-based autoformalization, demonstrating that models struggle to maintain semantic and compilation validity when faced with paraphrased inputs that preserve high fidelity. The findings indicate that minor linguistic shifts can cause significant performance variability, challenging the reliability of current approaches for generating grounded formal proofs. For practitioners, this underscores the necessity of robustness testing against paraphrasing before deploying autoformalization tools in safety-critical or verification-heavy workflows.
Evaluating Autoformalization Robustness via Semantically Similar Paraphrasing
arXiv:2511.12784v3 Announce Type: replace Abstract: Large Language Models (LLMs) have recently emerged as powerful tools for autoformalization. Despite their impressive performance, these models can still struggle to produce grounded and verifiable formalizations. Recent work in text-to-SQL, has revealed that LLMs can be sensitive to paraphrased natural language (NL) inputs, even when high degrees of semantic fidelity are preserved. In this paper, we investigate this claim in the autoformalization domain. Specifically, we evaluate the robustness of LLMs generating formal proofs with semantically similar paraphrased NL statements by measuring semantic and compilation validity. Using the formal benchmarks MiniF2F and Lean 4 version of ProofNet, and two modern LLMs, we generate paraphrased natural language statements and cross-evaluate these statements across both models. The results of this paper reveal performance variability across paraphrased inputs, demonstrating that minor shifts in NL statements can significantly impact model outputs.
Abstract:Large Language Models (LLMs) have recently emerged as powerful tools for autoformalization. Despite their impressive performance, these models can still struggle to produce grounded and verifiable formalizations. Recent work in text-to-SQL, has revealed that LLMs can be sensitive to paraphrased natural language (NL) inputs, even when high degrees of semantic fidelity are preserved. In this paper, we investigate this claim in the autoformalization domain. Specifically, we evaluate the robustness of LLMs generating formal proofs with semantically similar paraphrased NL statements by measuring semantic and compilation validity. Using the formal benchmarks MiniF2F and Lean 4 version of ProofNet, and two modern LLMs, we generate paraphrased natural language statements and cross-evaluate these statements across both models. The results of this paper reveal performance variability across paraphrased inputs, demonstrating that minor shifts in NL statements can significantly impact model outputs.
Computation and Language (cs.CL); Logic in Computer Science (cs.LO)
arXiv:2511.12784 [cs.CL]
(or arXiv:2511.12784v3 [cs.CL] for this version)
Full text under the publisher's open licence · 154 words
Related coverage
SHADOWBENCH: Toward Reliable Automatic Evaluation of Semantic Alignment in Autoformalization
arXiv · Sep 1, 2026
The Faithfulness Gap: Certifying Semantic Equivalence Between Natural-Language and Formal Mathematical Statements
arXiv · Jun 16, 2026
(Auto)formalization is supposed to be easy: Trellis process semantics for spelling out rigorous proofs
arXiv · Jun 9, 2026
Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
arXiv · Sep 5, 2026