Controlling Exponential Data Growth in the AI Age

Artificial intelligence (AI) isn’t just empowering organizations to deliver more proactive, predictive, and innovative solutions. It’s also exponentially increasing the rate of information being generated today. In turn, this growing tide of information is prompting enterprises to develop more effective data management processes and build more advanced IT infrastructures capable of ingesting and analyzing an increasing number and variety of data sources.

“It’s early days, but even at this stage it’s clear that AI is having an impact on organizational data volumes,” Simon Robinson, Principal Analyst at Omdia, told The Forecast. “Indeed, 87% of IT decision makers now report that AI is driving substantial data growth, according to one of our recent studies.” 

According to MIT, the total amount of data generated jumped by a factor of 32X in the decade from 2010 to 2020. That was before AI took off. The impact of AI quickly reached the tipping point of widespread consumer usage, reported Menlo Ventures, and the technology is prompting “habit formation at an unprecedented scale.” 

Other researchers report that nearly 4 in 5 organizations and 52% of adults are currently using large language models (LLMs) like ChatGPT. It’s clear that both the volume of data being generated and consumed by people and enterprises is compounding. Adding to this explosion is a new phenomenon: more content appearing on the Web is generated by AI than humans.

Looking at Data Generation Across the Ages

In 1776, Benjamin Franklin’s printer churned out dozens of pamphlets. In the late 1800s, tabulating machines were used to count hundreds of census returns. In the 1960s, mainframes stored and accessed thousands of customer records. In the 1990s, relational databases held millions of business transactions. Circa the 2000s, the Web spawned billions of data points, which multiplied with the rise of social networks, video aggregators, internet-connected sensors and the digitization of almost everything. AI is moving the needle an order of magnitude further.

IDC estimated that if the entirety of the world’s data up to a decade ago were imprinted on DVDs, the stack would be large enough to stretch to the moon 23 times or circle Earth 222 times over. Today, it’s estimated that almost 6X as much information is being generated on an annual basis alone, a figure that could balloon by 1000% by 2030.

Each new IT innovation jolted the world’s levels of data generation and content output upward. But experts say that what we’re currently seeing in the age of generative AI is an exponential leap that promises to be several degrees greater. 

Chief Innovation Officer at ServiceNow, Dave Wright, aptly wrote for CIO.com: “The data reckoning has arrived.” 

Yet, this generational advancement is not just faster, denser, and richer in variety than any boom in information technology seen to date. It fundamentally changes the nature of what enterprises will need to do with information going forward… and will greatly impact how they’ll store, manage and govern it in the years to come.

“What’s notable is that AI workloads are utilizing data across the board,” noted analyst Robinson. “Our research suggests organizations aren’t just using structured information but also all manner of structured and semi-structured data as well.”

How AI is Remaking Data Generation and Management

Helpful to note is the way in which Generative AI now influences the way that technology leaders look at information oversight and data operations.

For instance, there’s the issue of volume (how much), noting that more data is now being generated than in any prior point in history.

Then there’s the concern of velocity (how fast), keeping in mind that feedback loops from user interactions, iterative processes, adaptive solutions and smart apps generate, consume, and replicate information at speeds that far outstrip older processes.

Teams also face issues relating to variety (different forms of information), noting that structured, semi-structured and unstructured data can now take the form of text, images, videos, audio, 3D data, sensor streams, etc.

And of course enterprises must also consider veracity (accuracy, context, etc.), noting that AI models thrive on clean, labeled, high‑quality data and metadata, provenance solutions, and privacy handling.

“High-quality data management is critical to making AI results more effective,” Alex Almeida, senior product marketing manager at Nutanix, told The Forecast. 

“You need tools that help you classify your data and to be able to better see and analyze the information to determine its importance. You also have to ask yourself questions like if this data should be managed and moved to closer sites to better manage LLM processing. Organizations are currently realizing just how important it is to have their data governance and management houses in order.”

Going forward, Almeida said companies need a robust data management strategy and tools. They must understand where data is collected, gathered and housed inside the business. Data sovereignty rules will require continuous data audits. Enterprises need to classify data and organize it so it can easily be fed and moved to a wide variety of applications on-demand.

“If you’re currently dealing with a hodgepodge of data storage solutions, sources, and information silos, it’s important to consider how to unify them,” Almeida advised. 

“Any tools or any new data governance paradigms that you can implement to help bring storage and management capabilities into the next generation are going to help with the ROI of AI implementation.”

Robinson sees businesses growing more interested in parallel file system technologies and bringing together a variety of capabilities. “Having GPU and computing resources and underlying storage in close proximity to data could be beneficial.”

New IT Concerns that Enterprises Need to Be Aware Of

Organizations use data for multiple purposes (e.g., reference, process optimization, training). Data lifecycles can frequently stretch far beyond traditional bounds. This creates urgent pressure points on organizations… doubly so the more data that they generate and consume. Some things for IT teams to consider:

• Potential Storage Bottlenecks: Not only does storing more varied forms of data (and doing so in larger amounts) tend to come with larger workloads and greater computing demands. It also generally produces rising costs in terms of needs surrounding hardware, storage network bandwidth IO latency and throughput, which can quickly become performance or productivity chokepoints. If data management solutions can’t keep up with rising enterprise demands, it can negatively impact output.

• Governance and Data Oversight Concerns: As forms of information available to IT leaders increase in variety and attributes, enterprises must oversee and govern more complex data types. This can lead to concerns surrounding data quality, compliance and privacy, not to mention regulatory oversight and cybersecurity as well. Without good governance practices in place, AI outputs may be unreliable or expose an organization to heightened risk.

• More Complex IT Infrastructures: Many enterprises are already paying technology debt in the form of legacy IT tools and systems. In the age of digital transformation and AI automation, firms now face increasing pressure to not only use cloud or hybrid options to enable scale, but also to store certain data on‑premises. That requires companies to deal with concerns relating to interoperability, data movement, and hybrid operating strategies.

• Growing Energy and Operations Costs: The more storage, power and computing that enterprises are consuming, the more data center performance and real estate that companies need. Designing an efficient data management infrastructure is now critical, with AI workloads driving data centers to higher densities and creating the need for greater cooling and power demands.

Ways That Enterprises are Evolving and Adapting in Turn

To address rising issues, enterprises are now adopting and building better data management and governance processes and infrastructures across the board.

“Customers that have implemented solutions like Nutanix Unified Storage tend to find that they're able to adapt much faster because they can more easily move data across the organization, and enable systems to more readily talk to each other. All of a sudden such tools allow for automated workflows that can become real key differentiators.”

Almeida pointed to examples of upgrades, including:

• Retrofitting existing on‑premises systems: Adding object‑storage layers and using file systems that support varied types of media. Doing so makes it possible for older relational or file server systems to still serve structured data, while new unstructured data can flow into added layers of IT solutions.

• Embracing hybrid and multicloud storage options: Using public cloud storage services and solutions for large unstructured datasets, archives, etc. while using on‑premises or private cloud tools for transactional, sensitive, and high‑performance structured data.

• Adopting edge storage to minimize latency: For sensor-powered data or devices that collect data far from central clouds, keeping this information closer to the point of collection before repatriating or summarizing it.

• Leverage smart caching, data compression and deduplication techniques: For example, reducing data’s overall footprint by compressing unstructured media, removing redundant copies and using differential storage.

• Facilitating smoother data movement and networking: Noting that bottlenecks aren’t always disk-based, but more often surround the movement and transfer of data between apps, networks and systems, enterprises are re‑architecting pipelines, investing in high‑bandwidth, low latency networks, and optimizing data transfer capabilities.

Keeping Tabs on a Growing Mountain of Information

While the nature of data management and governance is rapidly changing, and the amount of information being generated and analyzed is currently skyrocketing, fear not. As noted here, organizations and IT leaders already have access to many of the tools and technologies that they need to more readily adapt.

“When it comes to rethinking data oversight and management policies, there’s still a huge level of concern around privacy and security, and fear of screwing up,” conceded Robinson. 

“Organizations also remain concerned about broader cybersecurity threats and a fear around AI displacing humans in the workplace.”