Dive Brief:
- High costs associated with using artificial intelligence models could constrain broader enterprise adoption, Amazon CEO Andy Jassy said Thursday, Amazon CEO Andy Jassy said Thursday.
- Jassy specifically cited costs related to AI inference, the process by which a trained model generates responses, analyzes information or performs tasks.
- “If we all want enterprises to be able to use inference the way we all believe they will, we have to keep working on taking the cost of inference down,” Jassy said during Amazon's third-quarter earnings call. He identified a shortage of employees with the expertise to operate AI systems and fine-tune models, along with inference costs, as obstacles to broader adoption.
Dive Insight:
The comments come as inference becomes an increasingly large part of AI infrastructure spending. Gartner forecasts that inference will account for 55% of global AI-optimized infrastructure-as-a-service spending in 2026, reaching $23.3 billion, compared with $19 billion for training.
Inference takes place each time a model is used, while training occurs during its development, according to an Amazon article published in February. As businesses expand their use of AI, the repeated computing required for inference becomes an ongoing cost.
In a letter to shareholders last year, Jassy said inference would represent the “overwhelming majority” of future AI costs because customers train models periodically but produce inferences constantly in large-scale applications. He called chips the “biggest culprit” behind AI's expense, but also pointed to advances in model distillation, prompt caching, computing infrastructure and model architectures as ways to make inference more efficient.
Reducing AI costs per unit would “unleash AI being used as expansively as customers desire” and lead to more overall spending, Jassy wrote.
On Thursday, Jassy said inference already accounts for well over half of AWS' AI revenue. Fortune 500 companies spent nearly seven times more on Amazon Bedrock, the company's AI model service, than a year earlier, while total token volume on the service increased nearly 17-fold.
Amazon reported third-quarter net sales of $201.9 billion, up 12% year over year, while AWS revenue climbed 39.3% to $46 billion, its fastest growth in nearly five years.
The company's fourth-quarter revenue forecast, however, came in slightly below Wall Street expectations. Amazon projected net sales of $240 billion to $247 billion, representing year-over-year growth of 12% to 16%.
Jassy said lowering inference costs remains a major focus for AWS. He cited the company's development of custom silicon, including its Trainium AI chips, as a way to reduce the expense of running models.
“It's incredibly important for customers to be able to use inference in a much more cost-effective way,” he said.
