aiqbench analysis
This paper proposes typological feature induction (TFI), a benchmark that isolates rule inference from rule application in LLM metalinguistic reasoning. Its key finding is that models often identify languages from very little text, possibly by relying on prior knowledge rather than induction, and on languages they cannot identify they do no better than a majority-value baseline. For readers, the practical takeaway is that current LLMs are not yet reliable for inducing grammar from raw text, which matters for field linguistics use cases.
Typological Feature Induction as a Benchmark for Inductive Metalinguistic Reasoning
arXiv:2611.04421v1 Announce Type: new Abstract: Existing metalinguistic reasoning benchmarks require both inference of rules and their application, making failures hard to diagnose. We introduce typological feature induction (TFI) which targets the former step. Given sentences in an unnamed language paired with translations in a known pivot language, models must predict a typological feature of the unnamed language. From Grambank and multi-parallel corpora, we automatically build evaluation questions over 195 languages. We find that LLMs identify 59--80% of these languages from only 16 sentence pairs; and misaligning the translations barely affects their accuracy, suggesting reliance on prior knowledge of the language. Enciphering text to prevent recognition increases the corpus' utility, but mainly for features where target and pivot languages agree. On languages which models cannot identify, neither open model outperforms a majority-value baseline. Since induction from raw text aligns with field linguists' use-case, this work indicates the models tested are not yet ready for this role. TFI offers a scalable path to track progress in this direction.
Read the full article at arXiv