A New Era of Data Management Architectures: Cloud and Beyond
The momentous transformation putting data managers and their systems at the helm of modern enterprises has only begun. Industry leaders and experts agree that it’s critical for data managers and team leaders to design today’s data architectures to meet the demands of a digital economy, from cloud to real-time streaming to AI.
“There’s a gold mine of digital treasures awaiting discovery,” said Lou Flynn, senior manager for AI and analytics at SAS. “But let’s be honest, the road to becoming a data-driven beast is no walk in the park. First, there’s that culture thing. Transforming data into a product means shaking things up. It’s about sparking a culture where innovation and experimentation aren’t just buzzwords, but daily bread. Data engineers need to set up data pipelines, establish their organizational heartbeat, and let DataOps fuel an automated and repeatable decision-making machine.”
DESIGN
What does it take to design AI-ready architectures today? According to Curtis O’Dell, global business manager for Tricentis Data Integrity, it requires, “more than ever, an end-to-end approach to data quality and integrity. Analytical and operational data can break at any time,” O’Dell added. “The data architecture must build in the critical capabilities needed to catch data errors before they become costly downstream issues.”
To take any brittleness out of data pipelines, data managers “must think differently about how they deliver trustworthy data at scale,” O’Dell explained. The key is “to leverage automation to continually test the data to ensure it is high quality and ensure that the data arrives intact and fit for the necessary purpose.”
TOOLS AND PLATFORMS
Overcoming data silos is another piece of the puzzle that needs to be addressed in an AI-ready data architecture.
“Data is dispersed and often siloed, and the sheer volume of data can be overwhelming,” said Sharad Varshney, CEO of OvalEdge. “For companies to fully leverage the capacity of their data, they need cohesion and a way to connect data sources securely wherever they exist. This means platforms that provide various core data analyses and management functionality, like a centralized data catalog and data visualization, enable businesses to consolidate dispersed data assets and streamline data-driven innovation.”
AI, for one, is not only a capability enabled through modern data architecture, it also makes architectures more responsive to business and technology requirements. “AI has not only massively increased data generation itself, providing organizations with valuable information to help support key business goals, but it has also made managing data much easier,” said Tim Sherbak, enterprise products and solutions marketing manager at Quantum. “For instance, if an organization needs to pinpoint specfic information among troves of unstructured data—such as video, photo, and audio les—AI-enabled metadata tagging tools can make that process much faster and more efficient.”
Integrating AI “into data management workflows is streamlining complex tasks and reducing the need for extensive technical skills,” Jitesh Ghai, chief product officer for Informatica, agreed. “This is particularly beneficial for nontechnical line-of-business users to interact with the data effortlessly.”
“This year, we’ll see more enterprises start using data to train AI models—both standard machine learning models and generative AI models such as large language models,” said Joseph Regensburger, VP of research at Immuta. “Where data has always been helpful for enterprises looking to understand the metrics moving their businesses, generative AI combined with valuable, industry-specific data can help businesses unlock new insights like never before.”
The ascent of AI into data architectures is resulting in cost reductions in data management operations, said Ghai. “AI can analyze data to reveal hidden insights, suggest useful information or connections for innovation, and predict upcoming trends and market needs.”
A major piece of this capability is generative AI, which offers data managers “enhanced productivity through tools like coding assistants and LLMs,” said Flynn. In addition, “the idealistic concepts of data mesh and data fabric are slowly finding their way into mainstream adoption.”
Another contribution to AI-ready data architecture is the emergence of “the infrastructure as a service layer with cloud-based services offering scalable storage and effective processing capabilities,” said Jonathan Bruce, VP of product management at Alation. Cloud-based data analytics enables businesses to access “data processing and storage resources on-demand.”
In addition, AI and machine learning frameworks are “shaping how enterprises leverage data, enabling predictive analytics and generating intelligent insights that enhance data interpretation for informed decision making,” said Bruce.
