Unveiling The Role Of Data Scientists: Navigating Responsibilities And Challenges

Over the last few decades, data science has become increasingly critical to organisational success. It’s the science (and art) of extracting meaningful insights from data using multi-disciplinary approaches. The immense growth of this field can be attributed to digitalisation, accessible and affordable technologies, data storage and cloud solutions, machine learning platforms, data analytics frameworks and a data-first culture. This data-first approach has helped bring a paradigm shift towards data-driven decision-making processes, transforming entire industries.

We see incredible AI and data science achievements in almost every conceivable domain — from agriculture and finance to the arts and humanities. These transformations promise a healthier, brighter and more prosperous future.

The need for responsible data science 

Despite this progress, the profession faces enormous challenges. The power of free open-source technologies, cloud-based platforms and data capture tools have made it possible to implement entire data science pipelines for a wide variety of applications within just a few days if not hours or minutes.

However, this potential for rapid development often leads us to navigate spaces that haven’t been explored before, raising critical questions of power, rights, bias, ethics, privacy and sustainability. These are uncharted territories that need time and diverse disciplinary perspectives to understand and the engagement of diverse stakeholders collaboratively and inclusively.

Looking back at past experiences can offer glimpses of how irresponsible use of data science can result in harm: for example, the Cambridge Analytica scandal highlighted how social media data harvested from millions of Facebook accounts were used to create profiles to deliver targeted political content during the 2016 US presidential elections, influencing democratic processes. Another example of AI use in recruitment processes demonstrated how candidates based on gender, race or other protected characteristics can potentially be disadvantaged. Such examples are numerous and can lead to the erosion of trust in data science.

Types of data scientists and career prospects

The call for responsible use of data science brings forward another actor in the process—the data scientists. There are different ways to look at the profession, and one of these is to study the different roles data scientists play. The range of responsibilities highlights how dynamic, interdisciplinary and divergent the profession is. 

The skills a data scientist should possess (according to the UK Government) include applied mathematics, statistics and scientific practices, data engineering and manipulation, innovation, delivering business impact, developing data science capability, ethics and privacy, programming, building software and understanding product delivery. 

This role requires a balance between technical (e.g. software and algorithm development, statistics) and non-technical skills (e.g. identifying business needs, ensuring responsible AI practices are used, communicating data science results). Often, individual interests lead a data scientist toward either a more technical or a non-technical path.

We therefore, observe two types of data scientists emerging—firstly, data scientists, who are more technical and likely have an area of specialisation (e.g. text mining, statistics, data visualisation), and secondly, data scientists, who have knowledge of the data science process and often act as a translator between data science and stakeholders, communicating user requirements and potentially leading a team of data scientists.

Data scientists are experiencing exceptionally promising career prospects in today’s job market. As the field continues to evolve, data scientists can explore various specialisations, including machine learning, artificial intelligence, and data engineering, expanding their horizons and staying at the forefront of technological advancements. Overall, the career trajectory for data scientists looks promising.

Whichever type of data scientist one chooses to be, there certainly is an exciting future to behold, with almost every organisation envisioning a data science future. Yet, to me, the excitement isn’t just about the promising efficiencies, instead, it is about how we bring a principled approach toward conducting responsible data science, opening new dialogues and new ways of thinking about the role of data, data science and the data scientist in today and tomorrow’s world.

This author is a Senior Lecturer at the University of Sheffield, UK.

Published on: Monday, April 08, 2024, 07:45 AM IST