AI data boom forces companies to rethink storage costs and capacity

The rapid growth of artificial intelligence (AI) and generative AI (GenAI) is straining storage resources for businesses worldwide, according to a new study by International Data Corporation (IDC) in partnership with Western Digital (WD). The study revealed that 94.7% of surveyed organisations increased their data storage capacity due to AI and GenAI adoption over the past year, with 61% experiencing a growth of at least 25% in data volumes. Furthermore, 74% expect their data volumes to expand by 25% or more within the next three years.

Unlike computing workloads, which conclude upon processing completion, a significant proportion of AI-generated data persists and accumulates. Companies are discovering novel uses for archived information, leading to more data retention. In fact, 59.4% of organisations identified AI-generated data, including synthetic data, inference outputs, and model logs, as the primary driver of the growth in their data-lake volumes.

The value of stored information is also on the rise, with 95% of respondents noting an increase in their organisation's data value due to AI and GenAI adoption. As a result, 74.3% of organisations have decided to retain data for longer periods, while 75.9% have brought back archived cold-tier data online to support AI workloads.

This trend heightens expectations for faster data retrieval, as 96% of respondents anticipate needing quicker access to archived data for AI inference and retrieval-augmented generation (RAG) applications.

Storage costs have become a crucial factor in storage decision-making, with 98.2% of respondents stating that total cost of ownership per terabyte is important or very important. This finding highlights that the traditional distinction between active and archived data is blurring as historical information is increasingly reintegrated into AI data workflows.

Consequently, organizations must design storage architectures that cater to the entire AI data lifecycle, considering factors such as performance, capacity, accessibility, and long-term costs based on specific workload requirements.