New Study Reveals Growing AI Use, Data Quality Challenges in Analytics

Poor data quality is the most prevalent problem today faced by data teams today, while unclear data ownership persists as an issue, according to a new study that looked at the landscape for data practitioners today.

These are two key findings in the 2024 State of Analytics Engineering, sponsored by dbt Labs and in its second year.

The survey collected responses from 456 data practitioners and leaders and looked at various aspects of data practices, including emerging technologies, salary trends, and the most pressing problems faced by teams.

State of analytics

For a start, a significant and growing number of data teams are investing in AI, with many practitioners already adopting generative AI in their day-to-day workflows. Specifically, 57% of respondents are either managing data for AI training or plan to do so in the next 12 months.

Most respondents spend most of their time organizing data sets for analysis, with over 50% selecting it as their number one task. This crucial task is the bedrock that enables all downstream analytics to be done with accuracy.

Challenges remain, however. According to the report, maintaining quality is becoming a greater struggle for teams in the face of data-related complexity. Already a concern in the previous edition of the report, this has escalated as a significant hurdle for data practitioners: 57% of respondents now highlight it as one of their chief obstacles in preparing data for analysis, up from 41%.

Another frequently cited challenge is “ambiguous data ownership”, highlighted by almost 50% of respondents. This points to potential obstacles in navigating growing data complexity.

Other interesting findings

Unsurprisingly, most data practitioners now feel they have a handle on building data transformations. 12% of respondents identified building data transformations as a challenge, down from 20% in 2022.

Data isn’t immune to economics, too. 41% said that the recent changes to the macroeconomic environment negatively affect budget size. Salaries are on the rise, though. 78% of all North American analytics engineers earned over $100K per year, compared to 61% of all data analysts and 66% of all data engineers.

Finally, data mesh use is growing. According to the report, over 60% of data teams from large companies are currently using or considering a decentralized data architecture such as data mesh.

The report can be accessed here (free registration).

Image credit: iStock/pcess609