analiza aiqbench
AFDBench introduces a critical benchmark for evaluating large language models in high-stakes meteorological contexts, specifically addressing the risks of numerical hallucination and stylistic misalignment in professional forecasts. By applying reinforcement learning techniques like GRPO to domain-specific data from Google's WeatherNext 2, researchers demonstrate that a 7B-parameter model can be effectively tuned to adhere to National Weather Service standards while accurately interpreting structured AI weather inputs. This work highlights the necessity of rigorous alignment strategies when deploying generative AI for safety-critical applications where factual precision and professional tone are paramount.
AFDBench: A Reasoning-First AI Scientist for NationalWeather Service Forecast Discussions
arXiv:2608.24954v1 Announce Type: new Abstract: Large language models (LLMs) hallucinate numerical values when generating high-stakes meteorological text, posing risks for weather communication. We present AFDBench, an AI meteorologist that generates professional Area Forecast Discussions (AFDs) by reasoning through structured AI weather forecast data from Google's WeatherNext 2. We introduce AFDBench, the first benchmark for evaluating generative meteorological reasoning, comprising 7,732 expert written discussions from 13 National Weather Service (NWS) offices paired with real AI weather forecast inputs, and three complementary metrics: Met-Align (numerical accuracy), Style-Align (professional dialect adherence), and Input-Grounding (fidelity to source weather data). Zero-shot evaluations reveal that open-source LLMs achieve low Style-Align (~0.33) and moderate Input-Grounding (~0.88), failing to write in the professional NWS register or faithfully use their input data. We apply Group Relative Policy Optimization (GRPO) with domain-specific rewards targeting temperature accuracy, synoptic correctness, and format compliance. On 1,033 held-out samples from two unseen NWS offices, GRPO nearly doubles Style-Align from 0.318 to 0.619 and improves Input-Grounding from 0.881 to 0.940, demonstrating that reinforcement learning teaches a 7B-parameter model to write like a professional meteorologist and faithfully interpret AI weather data.
Abstract:Large language models (LLMs) hallucinate numerical values when generating high-stakes meteorological text, posing risks for weather communication. We present AFDBench, an AI meteorologist that generates professional Area Forecast Discussions (AFDs) by reasoning through structured AI weather forecast data from Google's WeatherNext 2. We introduce AFDBench, the first benchmark for evaluating generative meteorological reasoning, comprising 7,732 expert written discussions from 13 National Weather Service (NWS) offices paired with real AI weather forecast inputs, and three complementary metrics: Met-Align (numerical accuracy), Style-Align (professional dialect adherence), and Input-Grounding (fidelity to source weather data). Zero-shot evaluations reveal that open-source LLMs achieve low Style-Align (~0.33) and moderate Input-Grounding (~0.88), failing to write in the professional NWS register or faithfully use their input data. We apply Group Relative Policy Optimization (GRPO) with domain-specific rewards targeting temperature accuracy, synoptic correctness, and format compliance. On 1,033 held-out samples from two unseen NWS offices, GRPO nearly doubles Style-Align from 0.318 to 0.619 and improves Input-Grounding from 0.881 to 0.940, demonstrating that reinforcement learning teaches a 7B-parameter model to write like a professional meteorologist and faithfully interpret AI weather data.
Machine Learning (cs.LG); Atmospheric and Oceanic Physics (physics.ao-ph)
arXiv:2608.24954 [cs.LG]
(or arXiv:2608.24954v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.24954 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 277 cuvinte
Subiecte conexe
Dr. Jill Lepore on why AI backlash is vital for the future
The Verge · 20 iul. 2026
Links #5: 2026/07
LessWrong · 31 iul. 2026
Salesforce after Dreamforce: How the CRM giant can grow beyond its own interface
SiliconANGLE · 19 sept. 2026
A New Era of Discovery: Google Research at I/O 2026
Google Research · 28 mai 2026