analiza aiqbench
This study demonstrates that Small Language Models trained on both natural and symbolic languages can effectively disentangle content from formal reasoning, achieving a content score of 27.80% while reducing inherent reasoning bias. By leveraging formal logical notations, the SEF-CLGC pipeline offers a viable alternative to relying solely on massive models for specific evaluation tasks. The approach highlights the potential of hybrid training data to improve the reliability and interpretability of language model reasoning capabilities.
SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance
arXiv:2606.09157v1 Announce Type: new Abstract: This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC). We combine formal logical notations with Small Language Models (SLMs) to evaluate reasoning performance on the SemEval-2026 Task 11 Subtask 1: Disentangling Content and Formal Reasoning in Large Language Models. Our experiments show that by relying solely on SLMs, trained on a combination of natural and symbolic languages, our best model achieves a content score of 27.80% on the task while significantly lowering the content bias in reasoning.
Abstract:This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC). We combine formal logical notations with Small Language Models (SLMs) to evaluate reasoning performance on the SemEval-2026 Task 11 Subtask 1: Disentangling Content and Formal Reasoning in Large Language Models. Our experiments show that by relying solely on SLMs, trained on a combination of natural and symbolic languages, our best model achieves a content score of 27.80% on the task while significantly lowering the content bias in reasoning.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.09157 [cs.CL]
(or arXiv:2606.09157v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.09157 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 179 cuvinte