I Tried Data Analysis ChatGPT Plugin — Every Analyst’s Dream or a Nightmare in Disguise? | by Livia Ellen

I provided the same JSON file to the Data Analysis ChatGPT plugin with a slightly different prompt — I would not use the word words/topics to avoid the possibility of receiving one-word topic responses, a challenge encountered on the first attempt.

On this attempt, I would also recommend my approach to ChatGPT instead of saying Yes all the time and letting ChatGPT do its things.

give me idea about :
– what are the top 10 topics based on this conversation
– how many times those topics are mentioned
– label each conversation as action: for example: asking for concept, summary, idea, etc this is just example, label are not limited to those

Result

After the first prompt,

The Data Analysis ChatGPT Plugin gives the expected answer! This accuracy, achieved without the need for any fine-tuning on my part, indicates a notable improvement in the plugin’s performance.

With that, I will add more stuff to the list of the Pros.

Pros:

• Good prompt engineering might equal a good result

Here is the breakdown of the result…

The first part of the first response shows its self-correcting capability without me having to say “fix it” explicitly.

The image below is the second part of the response, it shows the answer to Goals #1 and #2. Notably, in this attempt, the topics identified by the plugin are more granular and detailed compared to the one-word topics generated in the first attempt.

The last part, Data Analysis ChatGPT Plugin has successfully answered Goal #3 — labeling the action for each conversation and counting the occurrences.

I am intrigued with the answer from the Data Analysis ChatGPT Plugin! So far, it performance seems like a dream for the data analyst.

Now, it’s time to move forward!

Let’s provide some feedback…

I am going to ask ChatGPT to fix the categorization of the “General/other” category to be labeled correctly.

Data Analysis Plugin managed to fine-tune its categorization capabilities throughout this experiment.

So far the performance has been good, let’s export it to a Jupyter Notebook!

This step will test the plugin’s ability to seamlessly integrate its output into a Jupyter Notebook format for further exploration. And hey, we want to test it as a data analyst, so we need the code!

It seems like we have encountered a list of errors. I see that we had a token limit issue, hence it’s failing the notebook generation process.

I told ChatGPT to force-stop the process.

Cons:

• The plugin will keep doing the process until it’s done unless we force stop. It appears to lack an in-built break theory or mechanism to autonomously stop when encountering significant issues.

Following this experience, I provided feedback to ChatGPT, emphasizing the need to address and handle this issue.

The Jupyter Notebook was successfully generated. When I downloaded it, it missed the middle chunk of the code, I had to remind them.

Plugin’s Acknowledgment:

Data Analysis ChatGPT plugin acknowledge the mistake. Good moral GPT! — see highlighted blue box on the image

The Jupyter Notebook which has been generated still missing some parts of the code. So, I gotta gave ChatGPT some encouragement – aiming to motivate and perhaps make the plugin towards a more effective resolution. A data intern would like this kind of encouragement too 😉 Haha…

And…

Whoopsie. More error!

Honestly, I am tired of debugging and asking…

I realized this plugin still needs to learn more. The most annoying pain point is I had to ask and remind them, as it does not know the best practice and lack of decision-making about when to cease operations. The continuous cycle of debugging and providing reminders has become exhausting.