Software Co-Pilots Now Guide Data Analytics

Developer fly planes. Okay, so they don’t really, but they do steer hugely complex code streams running multi-control enterprise software applications and data services that would baffle the average application user (passenger) if they suddenly found themselves at the cockpit controls. Perhaps this is why the software industry uses the term co-pilot as a means of explaining a technology designed to automate and shoulder a number of the tasks involved with getting a composite and increasingly cloud-native application off the ground – pun quite definitely intended.

As we have discussed before, the move to enable software application development engineers with co-pilots is a growing trend. Despite this movement, systems architects are concerned about the need for transparency and trust in these automated code-creation tools. With so many checks, parameters and dependencies to look after in any given software project, the list of tiny nanoscopic functions that a co-pilot needs to embody, feature and execute is massive. But progress is being made in this space.

Looking to bring this type of functionality forward is data intelligence company Alation, with its Allie AI product. This Artificial Intelligence (AI) co-pilot stems from the company’s existing Machine Learning (ML) capabilities, which are aligned to help organizations scale their data programs and to democratize the use of trusted data. What the co-pilot flying does in this instance is to provide data engineers (at various levels) with intelligent curation and search capabilities to unlock their data’s value by finding relevant data faster and curating large volumes of data more efficiently.

According to Junaid Saiyed, CTO at Alation, “Organizations are racing to ensure enterprise data readiness for critical business initiatives, such as implementing AI-powered strategies through building and training large language models (LLMs). Yet, the volume and breadth of data generated by the explosion of SaaS applications and new [work role] personas – such as AI engineers – that need access to that data has held back the delivery of such initiatives.”

The role of data stewards

Saiyed explains Allie AI (stylized in Alation’s marketing materials as ALLIE AI) as technology that automates the documentation and curation of data assets at scale, making it easy for analysts to find that data.

The intelligent curation capabilities on offer here enable organizations to accelerate the population of their data catalog (an organized and ordered list of information assets inside an organization with a higher level metadata view designed to guide data scientists and developers to different data objects inside a given data storage instance) by automatically documenting new data assets and suggesting an appropriate ‘data steward’ i.e. a data and information governance employee with responsibility for monitoring, describing, tracking and stewarding the access and control factors that surround a company’s data

This product has intelligent search and SQL auto-generation capabilities (a technology that provides database management querying functions to look for information) that enable analysts to find the data they need without needing specialist data analyst skills. It can also help analysts easily find customer data without knowing the names of the underlying datasets or how to write SQL itself. This allows analysts to build AI models and enable data-driven decision-making more quickly without needing a specialized data analyst skillset.

Volume, velocity, variety & veracity

“A data-driven organization can quickly address strategic questions. However, as data volumes grow, locating, understanding and trusting the data becomes increasingly challenging,” said Junaid Saiyed, CTO at Alation. “This challenge becomes more pronounced when businesses invest in data initiatives like generative AI. Such projects demand extensive data stores to operate as intended. Alation has tackled the ‘4Vs’ problem – volume, velocity, variety & veracity – for over a decade by harnessing ML and AI to support comprehension and confident usage. With Allie AI integrated into our data intelligence platform, data teams can expedite the discovery of relevant data, gain insights into the lineage of their AI models and effectively manage business outcomes on a larger scale.”

As we now move forward with enterprise software application teams that feature the use of software co-pilots, the benefits are hoped to be illustrated throughout AI engineering teams, among data scientists and data stewards, inside the DevOps operations layer and across the software development landscape – so, all the IT department, basically.

In line with the trends of the tech industry at large, there is also a degree of business user democratization happening here i.e. Alation has pointed to intelligent search for business teams so they can easily find and utilize relevant data with semantic search, a natural language way to search that focuses on a data object’s contextual meaning. Allie AI also helps generate SQL based on natural language search queries, democratizing access to data to non-specialist data personas.

While most software developers and technology engineers still can’t fly a plane, they do now have a route map, a smart inventory and an intercom.

Chicken or pasta sir/madam?