aiqbench takeAI Products

This asynchronous API enables significant cost reduction by bundling inference requests, allowing providers to process large workloads like embedding back-filling or corpus labeling at approximately half the standard per-token price. While median completion times are rapid (around 7 minutes), latency varies notably based on submission time of day, with off-peak hours offering faster turnaround than early morning slots.

Source coveringOpenRouter

What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Batch API: half-price inference by bundling requests

OpenRouterSeptember 22, 20263 min read

Send a whole workload in one POST, collect the results within 24 hours, and typically pay half the per-token price. Across 230k+ batches that completed over our two week beta period, the median finished in 7 minutes.

For large batches of work or requests that don’t need to be completed immediately, you can now use the new batch API. When submitting a batch, a provider gets to choose when during a 24 hour window they will complete the request, and in exchange they generally charge 50% (and sometimes less) of their normal per-token…

Excerpt shown under the publisher's RSS terms · 56 words · full text and figures are on the publisher's site

Read at OpenRouter ↗Ask about this article