ChatGPT: Is it Practical for Health Care Data Analytics?

00;00;00;01 – 00;00;34;20
Tom Haederle
After launching in November 2022, it took ChatGPT just two months to become the fastest growing consumer app in history, gaining over 100 million users at an unprecedented rate. ChatGPT is perhaps the most prominent example, but certainly not the only one, of how artificial intelligence — or AI — is rapidly changing the world and the health care field along with it.

00;00;34;22 – 00;01;08;10
Tom Haederle
Welcome to Advancing Health, a podcast from the American Hospital Association. I’m Tom Haederle with AHA Communications. These days, our ability to process and analyze enormous volumes of data in new and innovative ways is evolving at warp speed. And AI is at the forefront. What are the implications for health care? Where is all of this going? In this podcast, three experts from the American Hospital Association discuss the game-changing practical applications of ChatGPT and AI for health care data analytics, and the potential pitfalls.

00;01;08;12 – 00;01;20;09
Tom Haederle
Host Stephen Hughes is director of health IT policy for the AHA. Brian Klein-Qiu and John Allison are both associate directors of health analytics policy. Let’s join Stephen, Brian and John.

00;01;20;11 – 00;01;47;09
Stephen Hughes
Let’s step back for a second to talk about how we got here. With less than 15 months ago, the world changed when ChatGPT kind of just crashed through the room, right? And all of a sudden added AI to the zeitgeist, catching a lot of us off guard. Now, as we know, AI has been in the health care space for decades right? There’s diagnostics and in revenue cycle and in several areas, there’s been tremendous advancement and practical application.

00;01;47;15 – 00;02;11;00
Stephen Hughes
But ChatGPT really accelerated and changed the way we think about it. It’s changing the regulatory conversation around it. It’s changed the way that every CEO and every board now is thinking about it. So we step back for a second to talk about what you guys do in data and analytics. There is a Brian and a John that works in some form or another for most health systems, right?

00;02;11;01 – 00;02;39;07
Stephen Hughes
We call them different things. It might be performance improvement team, it might be data analytics team. You know, it might be part of the strategy team. But one way or another, most health systems, whether they’re doing it internally or thinking about it through partnerships or consultants, are looking at how they use data, how to analyze data, and how do they either improve patient outcomes or make themselves more competitive, make themselves more efficient.

00;02;39;10 – 00;02;51;26
Stephen Hughes
So Brian and John, let’s talk a little bit about what you guys do, and then we’re going to talk a little bit about how you guys were using AI and how it was valuable to you and how I think it’ll be useful to our members.

00;02;51;28 – 00;03;10;27
John Allison
So, yeah, I’m John Allison. I think the most succinct way to describe what I do, at least the way that I approach my job is, my goal is to really tell compelling stories about hospitals, to advocate well for them. And one of the tools in my toolbag to do that is data, right? There’s a lot of it out there.

00;03;10;29 – 00;03;28;10
John Allison
Being able to understand it, being able to make sense of the anecdotes, for example, which is oftentimes really how you’re learning, is you’re hearing from members or other hospitals being able to kind of convert that into something that’s maybe more compelling. It’s really important.

00;03;28;12 – 00;03;46;11
Stephen Hughes
Right. I think you guys would agree that health care is unique among all sectors in terms of just the sheer volume of data, right, that you have. I talked at the beginning about, you know, finding use out of that data. But there’s a lot of noise inside, right, because it’s just by the nature of what we do.

00;03;46;11 – 00;04;08;09
Stephen Hughes
By the nature of patient care, there’s just a tremendous volume of data that’s captured. And Brian, could you talk a little bit about the the different languages around data? One of the challenges here is that it’s you know, it’s like there’s all this data and there’s like several different languages that are being spoken in terms of how that data is understood and analyzed.

00;04;08;16 – 00;04;34;19
Brian Klein-Qiu
Yeah, absolutely. So the most common data analysis or data science languages are going to be your typical languages like Python or R. For historical institutionalized reasons, SAS still kind of has a large, large stranglehold over health care policy data. So, for example, most Medicare claims, a lot of these files they come in SAS formats. It’s still kind of the lingua franca in our industry.

00;04;34;21 – 00;04;53;04
Brian Klein-Qiu
That’s kind of a challenge because there’s not a lot of SAS programmers these days, and that’s kind of a hard, hard barrier to access for people to be able to interact with claims data. When you add in the fact that a lot of these data analysts, data scientists, they’re not specifically gearing up to work in health care policy the way that John and I work.

00;04;53;12 – 00;04;59;08
Brian Klein-Qiu
What that means is you’re going to get a lot of great analysts and programmers who have no prior experience with SAS.

00;04;59;11 – 00;05;08;13
Stephen Hughes
Now, fast forward a little bit. So, Brian, when you came in, tell a little about the story of like you’ve had a little bit of SAS background, but…

00;05;08;16 – 00;05;09;21
Brian Klein-Qiu
Actually it’s opposite.

00;05;09;21 – 00;05;10;12
Stephen Hughes
Oh it’s the other way around.

00;05;10;14 – 00;05;12;15
Brian Klein-Qiu
I have a lot of SAS background.

00;05;12;17 – 00;05;13;04
Stephen Hughes
Right, right, right.

00;05;13;11 – 00;05;36;02
Brian Klein-Qiu
I have not touched Python and R in maybe five or six years because I’ve been living in the SAS world. So when I joined AHA in July 2023, I’m part of the new health care policy data team and our manager I think in week one or two he sits me down and says, “Hey Brian, there’s this really complex data project that John is handling right now.

00;05;36;06 – 00;05;58;15
Brian Klein-Qiu
Can you take a look at all of the complex great code that John’s been writing and try to contribute to it?” And if this was July 2022 before ChatGPT came out, I would have had to tell Aaron, “So, you know, Python’s on my resume. But the thing about that is I’m still going to need three weeks to make sense of what John’s writing.”

00;05;58;15 – 00;06;01;02
Stephen Hughes
And maybe a class. And maybe do a lot of Googling.

00;06;01;02 – 00;06;32;08
Brian Klein-Qiu
John’s going to have to see me down. But because it’s July 2023 and not 2022 ChatGPT is out. And I can tell Aaron, yeah no problem. So John hands me off the code base that he’s been working on. I feed literally everything that’s not sensitive into ChatGPT. And because I have experience with programing languages, I knew Python in the past, I’m able to pick it up pretty easily with the help of ChatGPT explaining to me what these specific lines do, what the overall structure is and all of that.

00;06;32;11 – 00;06;38;26
Brian Klein-Qiu
So literally from day one, I’m able to plug in and contribute to what John’s been handling for I don’t know what, the last six months?

00;06;38;29 – 00;06;53;12
John Allison
Yeah, I mean, I would say not only contribute to it, but just like make it better really quickly. Like I remember some of that code you were sending to me and as we were working on and I was like, wow, this is a really good idea. Like, I hadn’t thought to do it this way.

00;06;53;12 – 00;07;13;04
John Allison
And, you know, and I think probably the thing to like underscore there is for data analysts, ChatGPT is really quickly becoming a must have in your quiver. It might be the quiver. It is so important for data analysts in terms of like in the toolbag you reach into when you’re doing your job.

00;07;13;07 – 00;07;34;22
Brian Klein-Qiu
Oh absolutely. Health care policy, you are working with really a diverse array of languages. But you kind of develop an expertise in one and you kind of just have to make do with whatever knowledge you can get to parse through all the other projects. For example, claims are going to be mostly in SAS files. If you do data and analysis or processing

00;07;34;25 – 00;07;53;04
Brian Klein-Qiu
a lot of people like to use Python for that. But if you look at the claim prices for like how consumers actually prices claims, all those source files are in Java. So if you can have a tool like ChatGPT that can help you process these very different languages easily, that’s an incredible boost to productivity.

00;07;53;07 – 00;08;12;22
Stephen Hughes
And there’s a couple of important points here. And one, to extend the metaphor of languages out. You know, effectively you have a Rosetta Stone, right? That’s understanding these languages. We have two different languages we started with, so we’re extending the metaphor out. We’ve got, you know, Mandarin and English. Now if you’re adding Java on top, you’re adding, okay, French, right?

00;08;12;29 – 00;08;23;20
Stephen Hughes
We keep adding more and more languages, more and more expertise, more and more time. And someone who’s not familiar with something…so ChatGPT effectively was acting as that kind of universal translator.

00;08;23;27 – 00;08;43;17
John Allison
What ChatGPT does is it says, okay, so maybe maybe you weren’t as strong at learning programing languages or Java or whatever it is. ChatGPT’s going help you get there in a way that I would have said… you talked about weeks to get caught up. That’s because you had some experience with some of these languages before.

00;08;43;19 – 00;09;07;19
John Allison
For someone who doesn’t you’re talking years, right? And so that’s where most of the data analysts that would be listening to this or that are if you’re a health system C-suite, that would be working for you. Fifteen years ago, ten years ago, even five years ago, it probably took them years and years and years of not only learning the language but getting really comfortable in it, learning how to use it to become the data analyst that they are.

00;09;07;20 – 00;09;09;00
John Allison
It takes years.

00;09;09;02 – 00;09;27;27
Stephen Hughes
So there’s this concept of upskilling, right? I know this is a concern with with CEOs everywhere in health systems now that are thinking about, you know, applications and investment in AI. And it’s an important piece because a lot of again, let’s go back to 2022. It’s you know, the AI is coming for our jobs.

00;09;28;00 – 00;09;55;19
Stephen Hughes
You know, there’s the crisis, right? We’ve now gone to this idea of it’s not taking jobs. It’s making individuals better at their jobs. It’s you know, like in our case, like in the AHA’s case, rather than having to hire two more Brians or a John and two more Brians, right. I’m now able to take Brian and John and upskill them to a point where I’m getting much more productivity out of two individuals by upskilling.

00;09;55;25 – 00;10;21;13
John Allison
I think with something like ChatGPT that years of like building up this experience and skillset to be able to tackle the problem has really been ameliorated. And to the extent that if I’m an analyst and I’m working at the AHA, which I am, and there’s a question, a big question that we have something we want to answer on a topic that maybe not all people have talked about yet, and there’s some data out there, but it’s just huge.

00;10;21;14 – 00;10;47;18
John Allison
We don’t understand it. Instead of me thinking, well, that’s a waste of my time. I shouldn’t be actually trying to answer that question. Instead, I’m thinking, okay, with ChatGPT, I feel reasonably confident that I can take something that’s currently incredibly difficult to understand and I don’t have a ton of background in. And I can still ask this big question from the data and reasonably expect to get an answer and not take years to do it.

00;10;47;20 – 00;11;16;04
John Allison
That’s just hard to underscore how game-changing that is, at least from my perspective. Whereas before I would have asked a really good question about my data, like what is what is driving this this trend that we’re seeing in health care and hospital utilization and things like that. And while the datasets might have been available to me, a lot of them aren’t, by the way, but the ones that are publicly available, I might have just shied away from asking questions like, well, I don’t have the next ten weeks to devote to this.

00;11;16;06 – 00;11;18;25
John Allison
And nowadays I would say that’s a week at most.

00;11;18;27 – 00;11;33;18
Stephen Hughes
But then the other part of which I think again, would resonate with with most health systems is this idea of like they’ve got a lot of data, that’s not the problem. They’ve captured all kinds of data, it’s just in all different places. You have a lot that’s in the HR. You have a lot that may have predated the HR.

00;11;33;20 – 00;11;57;06
Stephen Hughes
That you’ve been hanging on to for, you know, for years for regulatory reasons, but are you actually making a use out of it to create, again, better outcomes, other business opportunities? Who knows? And then sometimes again, there are disparate sources, right? What if you acquired another health system that has a different EHR. You acquired a practice or partnering with a practice, and they’ve got again, it’s a different EHR, or god forbid, paper.

00;11;57;09 – 00;12;18;00
Stephen Hughes
How am I bringing this all in? Or what tools am I bringing in? And in most cases, those answers are going to be, I have a very hard time doing that internally, right? I’m going to have to you know, I’m going to have to either move all of this to some cloud-based service which costs, security risks associated with it.

00;12;18;02 – 00;12;40;09
Stephen Hughes
Time. When am I going to get value back out of this as opposed to, hey, this is my data and now I can take a very narrow approach to my data and solve problems and get value out of my data with my staff inside a health system, right. Which is going to cut the security risk problem. The cost concerns, the do I bring in consultants?

00;12;40;09 – 00;12;49;18
Stephen Hughes
Do I you know, all these other issues? I can actually start solving these problems internally because I have tools like, you know, GPT.

00;12;49;25 – 00;13;26;02
John Allison
Probably something to underscore is if you’re looking for an application of AI and you’re like, well, I’m not a huge health system. I don’t have these resources to spend on these more clinically-based AI applications or maybe even some sort of, you know, chat bot thing, things that maybe are more business-oriented, consumer-facing potentially. A really low, an area of a lot of low-hanging fruit for the use and practical application of AI tools like ChatGPT is going to be data analysis, frankly, and some people might find that boring, other people might find that exciting.

00;13;26;04 – 00;13;52;08
John Allison
I think that’s important to contextualize, it’s something that ChatGPT does pretty darn well compared to a lot of the other things that it does. It’s going to help you think through data problems comparatively pretty, pretty well. And I would say if you’re looking for a place to kind of grow your hospital or your health system, whatever it is that your business you’re working in, it’d be to be thinking about how to apply these tools to data problems.

00;13;52;08 – 00;14;10;08
John Allison
If your analytics department, your data analysis teams, the people that are helping with decision support, business intelligence, whatever it is, that’s the group out of anyone in an organization that I would think would be leading the charge, in a very I hate to use this word because we’re in D.C., but in a very grassroots way.

00;14;10;15 – 00;14;15;28
John Allison
You’re not being told to do it. It’s very organic. This is making me better at my job.

00;14;16;00 – 00;14;45;19
Stephen Hughes
And yes, it starts in data. But I think that, you know, your insight there, John, is good. And I think it’s applicable really to even outside of data analytics, right? The idea that the use case came out organically by need. So it’s, and I think if there’s a lesson there and there’s marching orders to then take out, you know, really any level of management in the health system that maybe listening to this all the way up to senior executives and board members is, worry less about trying to shoehorn a solution into the organization.

00;14;45;21 – 00;15;01;19
Stephen Hughes
It’s go down to the practitioners, right? Go to your people in the front lines, go to your experts and ask them about how they could use it. And that’s where I think you’re going to get the most effective, you know, examples and the most effective tools are going to emerge from that.

00;15;01;22 – 00;15;11;13
Brian Klein-Qiu
It’s true that this is an incredible tool and you can solve so many issues and increases productivity so much in the hands of somebody who is competent and somebody who is…

00;15;11;13 – 00;15;34;16
Stephen Hughes
In the hands of someone who is competent. Right. You have to have a baseline, competent person to start with that you’re upskilling as opposed to…what you’re not doing with ChatGBT or really with AI in general. And this is you could extend this metaphor out to the kind of the absurd level of you’re not going to turn somebody who didn’t go to medical school into a radiologist because they’re using an AI tool to look at diagnostic images.

00;15;34;16 – 00;15;55;28
Stephen Hughes
You get you start with a radiologist, add the AI on to that and they get much more productive and much more effective at picking up abnormalities in a mammogram. Just like you take someone who’s got good or at least a basis with understanding data languages and good analytical skills, add AI, they’re going to be much better at doing decision support.

00;15;55;28 – 00;16;01;03
Brian Klein-Qiu
Exactly. And the criteria for evaluating who’s good at analysis has changed with AI.

00;16;01;06 – 00;16;17;21
Stephen Hughes
I’m glad you brought this up, too. So we’re not just sounding like that we’re a bunch of just ChatGPT and AI cheerleaders, right? There’s always a downside with technology. So let’s talk a little bit about the quality measures. So how do we do that with ChatGPT and data analytics? How do you apply that safety criteria?

00;16;17;21 – 00;16;19;10
Stephen Hughes
What are the tests or what are the outcomes?

00;16;19;10 – 00;16;40;22
Brian Klein-Qiu
Yeah, that’s a good question and that’s you hit on a really, really crucial thing because that’s something especially in fields like health care systems or health policy in general, there’s not a lot of that kind of thinking. A lot of people a lot of organizations, a lot of agencies are not concerned with how do we know that these numbers are right, how we know that these algorithms are correct and we see how many mistakes CMS makes all the time.

00;16;40;24 – 00;17;01;19
Brian Klein-Qiu
It’s not limited to CMS. All agencies and companies do that. But one thing that I find shocking and that we need now more than ever is there’s no emphasis in the interview process for a lot of these places on how do you sense check your numbers in a project. It’s all what are the results of the projects? Give me examples of projects you’ve done and it’s never

00;17;01;26 – 00;17;25;13
Brian Klein-Qiu
how did you know that your project was correct? Explain to me your benchmark process. Explain to me how you were able to contextualize the numbers you got. The answers to those questions are much more indicative of how meticulous someone is and how analytical someone is. Than them being able to list five languages on their resume and say, okay, I created this algorithm and it gave me this many charts.

00;17;25;20 – 00;17;25;27
Brian Klein-Qiu
Here you go.

00;17;25;28 – 00;17;40;08
Stephen Hughes
So would you be saying in a way that the way to actually make sure that you’re you’re using GPT safely and effectively is to actually double-down on your people hiring process.

00;17;40;10 – 00;17;43;13
Brian Klein-Qiu
Or instill a culture. Instill a culture that is much more…

00;17;43;13 – 00;17;44;16
Stephen Hughes
Get very human centered.

00;17;44;16 – 00;17;46;01
Brian Klein-Qiu
Yeah, right, right.

00;17;46;06 – 00;18;01;04
Stephen Hughes
Interesting. So to ensure quality it’s make sure that you’re being careful about your your hiring practice,s right, and instilling a culture of quality and you know high levels of ethics in those who are using the applications.

00;18;01;11 – 00;18;15;14
Brian Klein-Qiu
The number one thing you can do if you’ve already hired a person to improve the culture of checking the quality is just literally ask, how do you contextualize these numbers with other entities’ numbers? Can you think outside of GPT, are you right externalizing everything you do?

00;18;15;19 – 00;18;39;15
Stephen Hughes
I love that. To actually ultimately control quality in GPT or tools like GPT think outside of GPT and in the end AI is a piece of technology. Really what we should be thinking about in health systems is a thoughtful application of technology. This is just one more thoughtful application of technology. Overall, I think it’s connecting back to what Brian was saying, which again connects us back to the human-centered nature of this.

00;18;39;21 – 00;19;02;05
Stephen Hughes
You know that ultimately the quality control check for, you know, use of AI in data analytics and decision support is do you have a culture of quality, right? Do you have a culture of peer checking around what you’re doing? You know, ultimately to get the most out of AI, across the system, across health care, is encourage a culture of innovation and creativity, right?

00;19;02;05 – 00;19;13;06
Stephen Hughes
Know the tools are out there. Encourage people not to be afraid of it. Go look at it and suggest things. You be amazed what you might find. Well, guys, I think that’s all the time we have. I really appreciate it. John, Brian, thanks so much.

00;19;13;09 – 00;19;14;16
Brian Klein-Qiu
Thank you. Pleasure to be here.