analiza aiqbench
AgentCompile demonstrates that leveraging LLMs as advisory metadata for semantic labeling and candidate prioritization can significantly accelerate CUDA inference without relying on the LLM for direct code generation. By empirically validating candidates and falling back to standard implementations when necessary, the system achieves substantial speedups over PyTorch eager across multiple models and workloads. The project's planned open-source release offers a practical framework for integrating specialized compiler support into transformer inference pipelines.
AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference
arXiv:2606.07665v1 Announce Type: cross Abstract: Transformer inference increasingly depends on specialized compiler and runtime support, but real model graphs still require semantic decisions about which regions are worth specializing and which CUDA implementation families are plausible. We present AgentCompile, an LLM-guided CUDA inference compiler that uses LLM outputs only as advisory search metadata. Given compiler-derived region summaries and bounded candidate spaces, the LLM proposes semantic labels, candidate priorities, parameter hints, and risk annotations; the compiler materializes CUDA candidates through templates, checks interface and hardware constraints, validates candidates empirically, selects implementations by measured latency, and falls back when specialization is unsupported or unprofitable. In end-to-end autoregressive generation, AgentCompile averages 5.66x, 4.05x, and 4.26x speedup over PyTorch eager on Qwen3-1.7B, Qwen3-4B, and Llama-3.2-1B-Instruct, respectively, across five representative workloads. We will open-source the project.
Abstract:Transformer inference increasingly depends on specialized compiler and runtime support, but real model graphs still require semantic decisions about which regions are worth specializing and which CUDA implementation families are plausible. We present AgentCompile, an LLM-guided CUDA inference compiler that uses LLM outputs only as advisory search metadata. Given compiler-derived region summaries and bounded candidate spaces, the LLM proposes semantic labels, candidate priorities, parameter hints, and risk annotations; the compiler materializes CUDA candidates through templates, checks interface and hardware constraints, validates candidates empirically, selects implementations by measured latency, and falls back when specialization is unsupported or unprofitable. In end-to-end autoregressive generation, AgentCompile averages 5.66x, 4.05x, and 4.26x speedup over PyTorch eager on Qwen3-1.7B, Qwen3-4B, and Llama-3.2-1B-Instruct, respectively, across five representative workloads. We will open-source the project.
Programming Languages (cs.PL); Artificial Intelligence (cs.AI)
arXiv:2606.07665 [cs.PL]
(or arXiv:2606.07665v1 [cs.PL] for this version)
https://doi.org/10.48550/arXiv.2606.07665 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 221 cuvinte
Subiecte conexe
AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference
arXiv · 5 aug. 2026
ATOM: Unlocking Extreme AMD Instinct Inference with Software-Hardware Co-Optimization
AMD · 15 iun. 2026
[AINews] Xiaomi MiMo-V2.6-Pro 1T-A42B: the new top Open Weights model, trained for $3M
Latent Space · 22 sept. 2026
NVIDIA Video Codec SDK 13.1: Zero-Copy Transcode, AV1 B-Frames, and Frame-Accurate Seek
NVIDIA · 31 iul. 2026