How to survive the AI spend hangover

It's 6:30am and you hear the door of the nightclub you've spent the last 8 hours inside shriek as it closes behind you. You watch bleary-eyed as an overly bright sunrise illuminates the business-suited people as they glide effortlessly along the sidewalk, their obnoxiously well-rested faces talking about work on their fully charged phones. You wonder, "Where did all the fun people go? And what happened to my wallet?" This feeling is what many CFOs, CTOs, CEOs, and AI program managers will imminently be experiencing in their board rooms, as they finally wake up to the realities that unrestricted and unmoderated AI use has wrought on their bottom lines and the stability of their core technical assets. You can already feel the party ending and the hangover setting in. The first warning sign came when Uber’s engineering org burned through its annual AI budget by April, and then capped its engineers at $1,500 a month per tool. At Meta, an internal leaderboard nicknamed "Claudeonomics" turned token spend into a status game. The company was on pace to spend billions, and the CTO's eventual memo had to spell out that token usage on its own measures nothing. Two of the most sophisticated engineering organizations on the planet have arrived a half step ahead of where we will all be soon: facing down a shocking bill and scrambling to tie it to any real ROI. Worse, many organizations would be hard pressed even to say which teams spent their tokens, on which models, and on what projects. Tokens spent wisely on complex problems, and tokens burned writing personalized fanfic all look the same on an invoice. But untangling them just became an urgent priority for everyone who shares responsibility for their company’s AI bill. Like any hangover, this one is going to hurt. But we don’t have to wait for the club to close down to start sobering up. There are already lessons to be learned about the differences between the companies using AI responsibly and the ones that have just been partying like there’s no tomorrow. Budgeting for a world where LLMs stay and AGI never arrives Paradoxically, the better LLMs get, the easier it is to see how far away they are from replacing the average knowledge worker. Even at their most effective task, coding, AI agents regularly fail to produce net positive results without a knowledgeable human to begin the work and the critical eye of an expert to review the outputs. For the first few years of the AI boom, the assumption was that we would eventually close this capability gap. But the reality is that in order to train and scale a model like Mythos, we've already stretched the pricing elasticity of chips, servers, and memory to their limits. There simply isn’t enough time and funding left to close the ocean-sized gulf between the capabilities of models and those of properly trained humans. Regardless of whether or not AI reaches workforce-replacement capability, the next problem is whether companies can afford to use it. Frontier model tokens are unlike any compute cost we've seen before. If your AWS bill gets too high, you can build your own datacenter and convert that expense into a capital investment with a predictable depreciation schedule. You can't do the same with frontier model inference. Once you've built a product that depends on frontier models, those marginal costs are a permanent feature of your economics. And those costs only grow over time. Per-token prices for GPT-4-level performance have reportedly fallen roughly 98% since late 2022, yet enterprise AI bills over the same period more than tripled, thanks to our voracious appetite for state-of-the art level inference. These may seem like macroeconomic abstractions, but they have direct implications for how you as a Finance, IT, or AI program leader (and hopefully at your organization, all three of those people are making decisions in concert) design your AI budgets and tie them to business outcomes. Specifically, it means that your organization cannot afford to default to using frontier models for every task, based on the assumption that they will soon be able to operate without human supervision. It’s also not as simple as issuing a blanket restriction on frontier models. As Matei Zaharia, CTO of Databricks, explained, "Cheaper per-token does not imply cheaper per-task…For example, Sonnet 5 costs less per token than Opus 4.8 but used more tokens, resulting in higher cost and lower quality." Tooling over tokens Even if we all start aggressively monitoring and reining in AI spend, that doesn’t instantly revert us back to the good ‘ol days where artisan engineers painstakingly chiseled out software from silicic igneous rocks with their bare hands. These models are here and our reliance on them is permanent. So the smartest organizations are now all asking the same question: “How do we get value from them without lighting money on fire?” Here’s my advice to the leaders designing and approving AI programs: be judicious with infere

Extras afișat conform termenilor RSS ai publicației · 173 cuvinte · textul integral și figurile sunt pe site-ul publicației