In a groundbreaking study conducted by the University of Passau, the language model ChatGPT has been found to outperform secondary school students in essay writing. The research team compared essays generated by the AI-based chatbot with those written by students, and the results were clear: the machine-generated essays excelled in all criteria, particularly in language mastery. The study, titled A large-scale comparison of human-written versus ChatGPT-generated essays, was recently published in the esteemed nature journal Scientific Reports.
The study stems from the notable progress made by the ChatGPT language model. While version 3.5 failed the Bavarian Abitur exam earlier in the year, its successor, version 4, achieved a solid grade nearly six months later. This impressive improvement prompted the University of Passau researchers to delve deeper into the potential of AI-generated content in revolutionizing the educational system.
Led by Professor Steffen Herbold, Chair of AI Engineering at the University of Passau, the interdisciplinary research team explored the effectiveness of the two ChatGPT versions. The team conducted a training course titled ChatGPT – Opportunity and Challenge, which was attended by 139 teachers, primarily from German gymnasiums. The course aimed to familiarize teachers with the technology behind text generators and ChatGPT, highlighting the challenges and opportunities they present.
During the training, the teachers evaluated a total of 270 English-language essays, unaware of whether they were human-written or generated by ChatGPT. The essays were assessed based on established grading scales from the Ministry of Education of Lower Saxony, covering criteria such as topic, completeness, logic, vocabulary, complexity, and language mastery. Scores were assigned on a scale of 0 to 6, with 6 being the highest.
The results were striking. Out of the 111 teachers who completed the questionnaire, both versions of ChatGPT consistently outperformed the students, with GPT-4 achieving the highest score. In terms of language mastery, the machine scored an impressive 5.25 (GPT-4) and 5.03 points (GPT-3), surpassing the students’ average score of 3.9. Professor Annette Hautli-Janisz, a Junior Professor of Computational Rhetoric and Natural Language Processing, emphasized that the high scores achieved by the machine did not imply poor language skills among students, but rather showcased the exceptional performance of the AI-driven models.
The study also shed light on the evolution of language in AI models. Professor Hautli-Janisz, together with doctoral student Zlata Kikteva, analyzed the machine-generated texts from a linguistic perspective, noting changes and improvements over time. The researchers highlighted the need for further exploration into the impact of AI-generated texts on human language.
Professor Steffen Herbold collaborated with Dr. Alexander Trautsch to conduct the statistical analysis and establish the data collection platform for the study. Professor Hautli-Janisz provided expertise in computational rhetoric and natural language processing and sourced a dataset of English language essays written by students from the Technical University of Darmstadt. Ute Heuer, Akademische Direktorin and head of the Computer Science Education team at the University of Passau, organized training courses to ensure teachers are prepared for the challenges and opportunities presented by artificial intelligence models.
The findings of this study present a compelling case for the integration of AI-generated content in the educational system. While the results demonstrate the language mastery of ChatGPT, it is crucial to strike a balance between leveraging the efficiency and precision of AI models while nurturing human language skills. As AI technologies continue to develop, a robust understanding of their potential and limitations is essential to shape a successful future of education.
In a society increasingly driven by information technology, the research from the University of Passau offers valuable insights into the role of AI and its impact on education. The study opens up a world of possibilities, urging schools not to disregard these powerful new tools. As educators navigate the advancements in AI, it becomes imperative to prepare teachers and students for the opportunities and challenges that lie ahead. With further research and collaboration, societies can harness the potential of AI to shape an inclusive and innovative educational landscape.
