analiza aiqbench
This asynchronous API enables significant cost reduction by bundling inference requests, allowing providers to process large workloads like embedding back-filling or corpus labeling at approximately half the standard per-token price. While median completion times are rapid (around 7 minutes), latency varies notably based on submission time of day, with off-peak hours offering faster turnaround than early morning slots.
Batch API: half-price inference by bundling requests
Send a whole workload in one POST, collect the results within 24 hours, and typically pay half the per-token price. Across 230k+ batches that completed over our two week beta period, the median finished in 7 minutes.
For large batches of work or requests that don’t need to be completed immediately, you can now use the new batch API. When submitting a batch, a provider gets to choose when during a 24 hour window they will complete the request, and in exchange they generally charge 50% (and sometimes less) of their normal per-token…
Extras afișat conform termenilor RSS ai publicației · 56 cuvinte · textul integral și figurile sunt pe site-ul publicației
Subiecte conexe
Quadrillion Param Costs: KV Cache, Context Length, Frontier Margins
LessWrong · 27 iul. 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · 13 oct. 2026
Same Cluster, 33 Points More Utilization: What Changed Was the Order
Hugging Face · 17 aug. 2026
GPU Management: Why Idle GPUs Are the New Grounded Aircraft
Hugging Face · 30 iul. 2026