analiza aiqbench
This study evaluates whether large language models can authentically generate the classical Arabic *maqama* genre, a rigorous testbed for assessing deep literary competence beyond surface fluency. The research demonstrates that prompting strategies significantly influence stylistic outcomes, with rule-based approaches enhancing rhetorical richness and coherence in top-tier models while zero-shot methods yield higher aggregate scores across the board. By establishing a controlled framework for evaluating culturally grounded prose generation, this work advances our understanding of LLM limitations in handling complex, historically constrained linguistic forms.
Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?
arXiv:2609.28245v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong performance in creative text generation, yet their ability to produce culturally grounded and stylistically constrained literary forms remains underexplored. Prior work has focused largely on modern language varieties and poetry, while classical prose traditions such as maqama remain largely unstudied. The maqama is a classical literary genre characterized by rhymed prose (saj), dense rhetorical ornamentation, and episodic narrative structure, making it a challenging testbed for evaluating whether LLMs can move beyond surface fluency toward deeper literary competence. In this paper, we present the first controlled evaluation study of maqama generation with LLMs, comparing five models under zero-shot, few-shot, and rule-based prompting, and evaluating outputs through both human annotation and an LLM-as-a-judge framework across dimensions such as rhetorical richness, saj density, structural coherence, and stylistic authenticity. Our results show that prompting strategy plays a strong role in stylistic quality: few-shot prompting most consistently improves saj density, while its effects on rhetoric and coherence vary by model, with the strongest models (GPT-4o and GPT-5.4-mini) benefiting most from rule-based prompting on these dimensions, though zero-shot prompting yields the highest aggregate scores across all five models. We further observe systematic differences between models in stylistic alignment with Arabic maqama conventions, and corroborate our findings with a second independent LLM judge, paired statistical significance testing, and non-LLM proxy measures of saj.
Authors:AbdulRahman A. Morsy (1), Aya Zirikly (1 and 2) ((1) Department of Computer Science, School of Engineering and Applied Sciences, George Washington University, Washington DC, United States, (2) Center for Speech and Language Processing, Whiting School of Engineering, Johns Hopkins University, Baltimore MD, United States)
Abstract:Large language models (LLMs) have shown strong performance in creative text generation, yet their ability to produce culturally grounded and stylistically constrained literary forms remains underexplored. Prior work has focused largely on modern language varieties and poetry, while classical prose traditions such as maqama remain largely unstudied. The maqama is a classical literary genre characterized by rhymed prose (saj), dense rhetorical ornamentation, and episodic narrative structure, making it a challenging testbed for evaluating whether LLMs can move beyond surface fluency toward deeper literary competence. In this paper, we present the first controlled evaluation study of maqama generation with LLMs, comparing five models under zero-shot, few-shot, and rule-based prompting, and evaluating outputs through both human annotation and an LLM-as-a-judge framework across dimensions such as rhetorical richness, saj density, structural coherence, and stylistic authenticity. Our results show that prompting strategy plays a strong role in stylistic quality: few-shot prompting most consistently improves saj density, while its effects on rhetoric and coherence vary by model, with the strongest models (GPT-4o and GPT-5.4-mini) benefiting most from rule-based prompting on these dimensions, though zero-shot prompting yields the highest aggregate scores across all five models. We further observe systematic differences between models in stylistic alignment with Arabic maqama conventions, and corroborate our findings with a second independent LLM judge, paired statistical significance testing, and non-LLM proxy measures of saj.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.28245 [cs.CL]
(or arXiv:2609.28245v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.28245 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 290 cuvinte
Subiecte conexe
MAWARITH: A Dataset and Benchmark for Legal Inheritance Reasoning with LLMs
arXiv · 17 iun. 2026
Falcon-Emirati: When an LLM Learns the Dialect, the Culture, and the Nuance
Hugging Face · 6 oct. 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
News
Quantum Computing Report · 25 iul. 2026