Skip to the cards
Filed
Loans
342

AI Infrastructure Costs: Why Data Sprawl Matters Alongside GPU Spend

AI costs include the data workloads store, index and retrieve. Track ownership and usage across clouds, agents and SaaS before spending rises.

AI Infrastructure Costs: Why Data Sprawl Matters Alongside GPU Spend

AI infrastructure bills can grow even when an organization’s GPU capacity looks well managed. Every AI workload also depends on data: information that must be stored, indexed, retrieved and processed. Duplicate files, abandoned workspaces and outdated repositories can add to those requirements, making data management part of cost control.

/articles/86/ZLtlSXJrbP9wvEKgkqyQNC9uxB4d0gMs.webp

That changes the questions IT teams need to ask. A monthly cloud invoice can show what was spent, but it may not reveal which team, workload or agent drove the increase. To understand the bill, organizations need cost and usage visibility across compute, storage and the data behind AI, including resources spread across cloud providers and SaaS platforms.

AI infrastructure spending is expanding

The spending forecasts point to pressure on budgets. IDC projects AI infrastructure spending will reach $497 billion in 2026, about 53% year-over-year growth, and exceed $1 trillion by 2029. Those figures cover a fast-growing area in which compute demand, model complexity and data volumes can rise together.

For an IT team, that growth can arrive through ordinary operational changes. A new use case attracts more users than expected. A team moves to a larger model. An agent is connected to another data source. Each change can affect compute, storage or both, and costs may shift faster than annual budget plans can account for.

Compute is the most visible part of the problem. GPU clusters and inference endpoints consume resources as workloads run, and usage can vary with query volume, model size and workload design. Gartner projects that 50% of cloud compute resources will be devoted to AI workloads by 2029, up from less than 10% at the time of its forecast. For technology leaders, that makes it more important to connect resource use to the workload and business purpose it supports.

Cost pressure is running alongside the demand for AI investment. About 52% of respondents to the 2026 Gartner CIO and Technology Executive Survey said reducing costs would become a more important objective over the next two years. That leaves IT teams balancing spending discipline with new workloads whose costs can change as adoption develops.

A model change can affect inference costs even if user numbers stay flat. A larger model or heavier workload may need more compute for each request. That is why teams need ongoing usage data, rather than relying only on an initial estimate or a review after the bill arrives.

The data behind the workload adds to the bill

Compute is only one part of AI infrastructure spending. Data needs storage, and some workloads also need resources to index, search and retrieve it. If a connected repository contains duplicate documents, old material or content the workload never uses, the organization may be maintaining a larger data footprint than the use case requires.

Gartner projects that by 2027, multistructured data management will account for 40% of total IT spending on data management technologies and services. This reflects the variety of information AI applications can draw on: structured records, semistructured application events and JSON files, and unstructured documents, messages, presentations, images and recordings.

Ava Ragonese Ave Point author scaled

A customer-support agent, for example, could use customer and transaction records from a database, support-ticket data from an application, and product documentation or customer communications stored in SharePoint or Google Drive. Bringing those sources together can help the agent retrieve relevant information. It also means teams need to understand what data is connected, where it lives and whether it still belongs in the workload.

How much a data issue affects costs depends on the workload design. A repository connected to an AI system may bring indexing, retrieval, processing, backup or governance requirements. In a retrieval-augmented generation system, for instance, duplicate or outdated material can leave more content to manage and search. Reviewing the data lifecycle alongside infrastructure usage helps teams identify what should remain active, be archived or be disposed of under applicable retention and business requirements.

Build visibility before optimizing

Cost reduction starts with a usable inventory. Teams need to know which AI workloads exist, where they run, what infrastructure and data they depend on, and who owns them. Without that information, a cost spike can be difficult to attribute until after it has become an ongoing expense.

  1. Inventory the environment. Create a shared view of AI workloads, cloud infrastructure, storage and connected data across providers. Record where each resource runs and who is responsible for it.
  2. Tag resources consistently. Use ownership, workload and environment details to connect spending to the teams and use cases that drive it. Consistent tags make cost changes easier to investigate.
  3. Allocate shared costs by usage. A GPU cluster or data lake may support several teams. Allocation based on measured consumption gives a clearer picture than dividing shared costs evenly.
  4. Monitor usage as it changes. Track workload and resource use regularly. Changes in query volume, models, agents or integrations can alter requirements before a scheduled budget review.
  5. Review data and compute together. Examine the information supporting a workload alongside its storage and processing needs. That can reveal inactive repositories or unnecessary content contributing to the footprint.
  6. Retire inactive resources where appropriate. Use inventory and usage evidence to identify workloads, infrastructure and data that no longer serve an active purpose. Account for retention, recovery and other business needs before removing content.
  7. Right-size compute to measured demand. Match GPU and inference capacity to actual usage, then revisit sizing when models or workload patterns change materially.
  8. Check for overlapping tools. Review AI and SaaS capabilities across platforms such as Microsoft 365 and Google Workspace to understand where functions overlap and which tools meet each business unit’s needs.
  9. Set budget guardrails before expansion. Define ownership, expected usage, cost thresholds and review points before a workload moves into broader production. That gives teams a basis for investigating unexpected changes.

These practices apply across multicloud environments, where providers have different services, pricing structures and billing views. Native cloud dashboards, FinOps platforms, cloud-management tools, resource inventories, data-governance products and SaaS management tools can all contribute. The toolset varies by architecture; consistent ownership, tagging and allocation are what make the results comparable.

Microsoft 365 and SaaS require a wider view

A Microsoft 365 Copilot or Google Workspace AI deployment involves more than its per-seat licence. Teams also need to understand the content, workspaces, permissions and lifecycle practices surrounding the information employees can access. Licensing may be straightforward to count, while the compute and storage needs of connected workloads can change with usage and the data they process.

For Microsoft 365, that means reviewing SharePoint sites, OneDrive content, Teams workspaces and access permissions as part of the wider operating picture. Duplicate content, abandoned workspaces and outdated information can create management and governance work around a deployment. The same reasoning applies to Google Workspace and other SaaS environments: data quality, ownership and lifecycle affect how well teams can understand and manage AI use.

Consider a 5,000-seat Microsoft 365 environment preparing for a broader Copilot rollout. The licence count gives one part of the spending picture. A review of duplicate files, inactive workspaces, outdated content and unclear permissions helps establish what data employees can reach and what the organization must manage as adoption grows.

That review is useful before rollout and as the environment changes afterward. It gives IT a clearer view of the information supporting AI use and helps surface data-management work that might otherwise be mistaken for a compute problem.

Agents multiply the need for ownership

Agents add another layer to infrastructure planning because each one may rely on models, data sources, integrations, storage and compute. Gartner predicts that by 2028, an average Fortune 500 enterprise will have more than 150,000 AI agents in use. That forecast makes inventory and ownership a practical requirement for organizations scaling agent use.

Teams need to know which agents exist, who owns them, what purpose they serve and which resources they use. They also need to determine whether an agent is active and still delivering value. Without that view, infrastructure requirements and data dependencies can spread across use cases without a clear owner.

Agents built quickly for narrow tasks can be especially hard to track. A support agent might be connected to a document library that has not been reviewed for years. If the library contains duplicate material, outdated information or unclear permissions, the team has a larger data environment to manage alongside the agent.

Before expanding an agent fleet, organizations can establish a register of existing agents and their dependencies. Revisit it when teams add agents, change models or connect new data sources. That gives operations staff a way to investigate which workloads use resources and which may no longer need to run.

Make cost reviews part of AI operations

Sustainable cost management links usage, ownership, data lifecycle and AI governance in a recurring review. A one-time cleanup can help, but it will lose value if content and workloads continue to grow without follow-up.

  • Review data governance and infrastructure costs on a shared cadence so teams can examine the workload and its dependencies together.
  • Schedule recurring data-reduction reviews and identify content that should remain active, move to archive storage or be disposed of where appropriate.
  • Revisit compute sizing after a material change to a model, workload or usage pattern.
  • Review overlapping AI licences and SaaS capabilities regularly, rather than waiting until renewal to compare them.
  • Assess the expected usage, owner and budget impact of each new agent or workload before broader deployment.

There is no universal benchmark for AI infrastructure spending as a share of an IT budget. Tracking it separately from general cloud spending can show how costs change over time. The more useful question is whether spending is keeping pace with the workload and value it supports. If costs rise without a clear connection to usage or outcomes, teams have a reason to investigate.

AI infrastructure cost management and AI FinOps overlap, but they focus on different scopes. Infrastructure cost management examines the compute, storage and data resources that operate AI workloads. AI FinOps applies financial accountability more broadly, potentially including models, licences and agent costs as well. Organizations can use both views to connect operating resources with the wider spending picture.

What IT teams should watch next

As AI use expands, teams should expect workload requirements to move with model choices, agent counts, query volumes and connected data. A predictable licence price does not make the infrastructure underneath it predictable. That is especially true when multiple teams share cloud resources or draw from large SaaS repositories.

Start by connecting costs to owners and workloads, then examine the data those workloads use. That sequence gives IT a clearer basis for rightsizing compute, reducing inactive resources and deciding which content needs to stay available. The key operational measure is whether teams can explain what is driving a change in spend while there is still time to respond.

Filed alongside