To capture the value of agentic AI without breaking budget models, finance leaders need to apply oversight. Specifically, they need to ensure their technology teams architect for positive AI economics from day one. The autonomy of agentic AI, which makes it so valuable to enterprises, often leads to unpredictable and significantly higher token consumption and costs. With agentic workloads, agents break down objectives into a large number of sub-tasks, adding multiple steps for planning, data retrieval, execution, checking responses, and retrying operations when necessary. This multi-step process results in many more inference transactions, and higher costs, than traditional generative AI (GenAI) experiences require.
It is critical for organizations to get their house in order for the agentic AI future, given its strategic importance and its potential to scale very quickly once deployed. According to Omdia research, 81% of organizations agreed that agentic AI is a strategic priority across their on-premises IT and cloud environments.1 Expanded agentic AI usage will expose inefficiencies in infrastructure design, driving costs to unsustainable levels unless those inefficiencies are addressed up front. Business leaders are already expressing cost-related worries when it comes to their agentic investments, with 80% of organizations agreeing that ensuring positive ROI for their AI initiatives is a top concern.2
As a CFO, here are four things you can do to help ensure your AI team avoids some common missteps in AI infrastructure design and achieves positive returns with agentic AI:
- Remember the “integration tax.” The total cost of AI includes far more than just the cost of the accelerators or other components. Understanding the total cost of ownership (TCO) is crucial. The time and resources required to architect, deploy, and optimize the infrastructure to meet the complexities of your specific use case and data environment are not trivial and often contribute to project delays that slow time to value. Full-stack AI hardware and software solutions significantly reduce the complexity burden on internal teams, accelerating time to value and reducing TCO while often eliminating the need for expensive third-party professional services. Conversely, bespoke system integrations often add complexity, delay time to market, and lead to increased ongoing maintenance costs. Press the importance of minimizing the total cost required to get your optimized AI project to production, not just the price of processing.
- Remind AI teams that they do not need a GPU for everything. AI infrastructure investments often risk becoming accelerator-centric rather than use case-centric when organizations focus too much on untranslatable benchmark numbers or component pricing without regard for the specific use case. According to recent Omdia research, 61% of organizations agreed that their AI investment has been driven more by fear of falling behind than by a clear, proven business case.3 To best optimize the design for positive financial returns, right-size the silicon for the use case (considering CPU, GPU, and TPU options). For example, running basic agent orchestration or routing (the software layer that manages how AI agents work together) on high-end GPUs is a waste of high-value assets. Lower-cost CPUs are often well suited for those operations. Leveraging a portfolio approach—CPUs for orchestration or tool-calling and TPUs or GPUs for targeted inference—is often a best practice that can lead to reduced cost per transaction and superior economics overall. Once the processing strategy is in place, encourage your AI design teams to optimize the surrounding ecosystem with low-latency networking and high-performance storage to maximize processing utilization. The priority should be to keep the most valuable infrastructure component, the compute, as close to 100% utilized as possible for your high-value use cases.
- Ask if your AI team is leveraging the latest advances to minimize idle processing. Once the infrastructure is optimized for the desired use cases, the architecture should embrace the latest software innovations to further improve utilization. For example, AI agents can waste resource cycles by keeping compute and memory reserved while paused, waiting for user input or responses from other agents or tools. If your accelerators (e.g., GPUs/TPUs) stay idle every time your agents wait, then your organization will waste its investment dollars and face higher costs over time. Agent Substrate, an open-source solution that runs on Kubernetes, frees up compute and memory resources when an agent is paused and preserves the state of the overall solution to resume once the agent is ready. The freed-up compute and memory can then be used by other agentic operations, further improving utilization and keeping costs under control.
- Ask if your AI team is leveraging the latest advances to eliminate overprovisioning. It seems like every team wants its own GPU (or multiple GPUs) for its own AI project, but those requests are typically not economical in the long run or even in the short run. Multiple recent advances have made it easier to share GPU resources or better utilize older accelerator investments to support new projects. For example, a technology called Dynamic Resource Allocation (DRA), which was recently introduced into Kubernetes, enables developers to specify their hardware requirements, rather than the component, such as a specific GPU, that they need for their project. The DRA technology will then automatically find an available resource that meets those requirements. DRA can be used in combination with tools like Google Active Assist, which provides telemetry tracking to ensure the environment stays continuously optimized for better performance and predictability. The result is often a superior and more predictable agentic experience with fewer investments in accelerators, reducing the cost burden of agentic AI.
It is the transformative potential of agentic AI that makes design inefficiencies so dangerous. Users often quickly become enamored with the agentic experience and ramp up usage. When infrastructure is co-designed for efficiency, CFOs can scale agentic workflows with predictable unit margins.
Learn how Google Cloud AI Hypercomputer delivers proven TCO and positive returns: AI Infrastructure
References
- Source: Omdia Research Report, IT Modernization Report: Cloud, Applications, and Infrastructure Amid the AI Revolution, August 2026.
- Ibid.
- Source: Omdia Research Report, The State of AI Inference Strategies: Optimizing Deployment, Performance, and Impact, April 2026.