aiqbenchInformații despre AI & calcul cuantic

...

Română · beta. Traducerea a fost făcută automat și este încă în curs de revizuire. Citește pagina în engleză

perspectiva aiqbenchCercetare AI

This research note investigates the theoretical capacity limits of transformer architectures regarding sequence ordering tasks, specifically demonstrating that two attention heads are required when an output MLP is present, whereas one head suffices without it. The study contributes to the foundational understanding of model expressivity and architectural efficiency within the machine learning community.

Surse care acoperăarXiv, AI Alignment Forum, Latent Space, AWS

Ce este aceastaLectura unui model asupra articolului și a acoperirii conexe, scrisă pentru aiqbench. Nu sunt cuvintele publicației.

Transformer Heads Looking for Order

arXiv24 septembrie 20261 min de cititArticolul și perspectiva sunt în engleză.

arXiv:2609.25588v1 Announce Type: new Abstract: In this note, we show that the problem of checking, whether a sequence of bits is ordered, is not doable by 1-head 1-layer transformers but is doable by a 2-head 1-layer transformer. Unlike similar previous results, our results assume the model where transformers have an output MLP.

Abstract:In this note, we show that the problem of checking, whether a sequence of bits is ordered, is not doable by 1-head 1-layer transformers but is doable by a 2-head 1-layer transformer. Unlike similar previous results, our results assume the model where transformers have an output MLP.

Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

arXiv:2609.25588 [cs.AI]

(or arXiv:2609.25588v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2609.25588 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Text integral sub licența deschisă a publicației · 144 cuvinte

Citește pe arXiv ↗Întreabă despre acest articol