Engaging with ChatGPT May Enhance Learning Illusions in University Students

Original article: Interactuar con ChatGPT puede aumentar la ilusión de aprendizaje en estudiantes universitarios

A clear, organized, and convincing explanation can create a deceptive sensation: believing we understand something when we are still unable to explain it ourselves.

This phenomenon is the focus of research conducted by Ivan Aslanov, a researcher at the National Center for Artificial Intelligence (Cenia), alongside Ernesto Guerra, an academic from the Institute of Advanced Studies in Education and a researcher at the CIAE of the University of Chile, and Patricio Felmer, a CIAE researcher and 2011 National Prize for Exact Sciences.

The study examined how interactions with artificial intelligence tools like ChatGPT can increase the «illusion of explanatory depth», which is the tendency to overestimate how much we understand about a phenomenon.

To investigate this, the researchers worked with 102 university students, divided into three groups. Each group was tasked with addressing four questions about everyday phenomena: why leap years exist, how glass is made, how zippers function, and why the Moon has phases.

The variation lay in the information they received and the medium through which it was delivered. One group interacted with ChatGPT; another read the same texts generated by the chatbot, without knowledge of their origin; and a third group received no additional material.

Afterward, all participants were required to assess their ability to explain each phenomenon and then explain the phenomena in writing.

Ultimately, following their own explanations, they reevaluated how well they believed they could explain the concepts.

The results were particularly striking: those who interacted with ChatGPT showed the highest overestimation of their ability to explain the phenomena.

The difference between what they initially believed they could explain and their subsequent confidence after writing their responses was significantly greater than that observed in the group that simply read the same contents.

This effect cannot solely be attributed to the quality of information, as both groups received identical content. What differed was the experience of engaging with the chatbot.

However, the research uncovered something else: the explanations provided afterward were also of lower quality among the group that used ChatGPT.

Two independent evaluators rated the responses on a scale from 1 to 4. The group that used ChatGPT averaged 2.47 points, compared to 2.75 for the group that read the same texts without interacting with AI. The group with no additional materials received an average score of 1.8 points.

Linguistic analysis pointed in the same direction.

Responses from the ChatGPT group averaged 43.8 words, while those from the group that only read the texts reached 59.6 words and demonstrated a greater diversity of vocabulary.

Furthermore, texts produced by the latter group more closely resembled the original material than those written by the group that utilized ChatGPT.

Is the Problem ChatGPT?

For the researcher from the University of Chile, Ernesto Guerra, this raises a fundamental distinction: «Having access to a good explanation does not necessarily mean one has assimilated that knowledge.»

«A person may recognize and follow an explanation when it is presented, but may not necessarily reconstruct it independently afterward,» noted the academic.

This finding is particularly relevant in a context where artificial intelligence is already part of students’ learning practices.

The 2025 PISA results show that the use of AI tools for learning is widespread among Chilean students aged 15, which—Guerra emphasizes—makes it «especially pertinent to question not just how much these tools are used, but also what happens to the understanding they build from them.»

Caution with Conclusions

The authors urge caution regarding conclusions in this area.

«The study does not demonstrate that using ChatGPT is bad for learning, nor that AI always leads to worse outcomes. What it shows is a specific effect: interacting with a chatbot may lead individuals to develop a confidence in their understanding greater than what corresponds to their subsequent performance,» emphasizes the expert from the University of Chile.

One possible explanation is that users attribute part of the information to their own knowledge that is actually available through the tool.

Another possibility is that the interaction style of chatbots—which presents responses as a fluid conversation—contributes to generating a greater sense of understanding or confidence. The researchers themselves note that these hypotheses require further studies.

This warning is especially relevant for education. In a scenario where obtaining an explanation is increasingly easier, the challenge is no longer just to find the answer, but to verify whether we truly learned.

We will continue to keep you informed.