Linguistics Experts Struggled to Identify AI and Human Writing, Reveals Study
Linguistics experts face difficulty distinguishing between written content produced by artificial intelligence (AI) and humans, according to a study by researchers from the University of South Florida and the University of Memphis. Published in the journal Research Methods in Applied Linguistics, the study found that linguistic specialists from renowned global journals were only able to accurately differentiate between AI and human-authored abstracts around 39% of the time.
The research aimed to explore whether experts in linguistics, who have extensive experience studying language patterns and human communication, could discern between texts written by humans and those created by AI. However, out of the 72 linguistics experts involved in the study, none were able to correctly identify all four writing samples provided, with 13% inaccurately identifying each sample.
Despite their attempts to utilize certain linguistic and stylistic factors to determine the origin of the writing samples, the experts were largely unsuccessful. The study revealed an overall identification rate of just 38.9%. The experts provided logical rationales for their decisions but were consistently inaccurate and inconsistent.
The study also highlighted that AI-powered language models, such as ChatGPT, exhibited comparable or even superior abilities to write short genres when compared to human authors. Due to the absence of grammatical errors in AI-generated content, linguistic experts faced challenges distinguishing AI writing from human-authored writing.
However, there is a silver lining for human authors. The study suggested that in longer forms of writing, AI models tend to hallucinate and fabricate content, making it easier to identify that the text was generated by AI.
Lead researcher, Matthew Kessler, emphasizes the need for extensive discussion regarding the ethical usage and guidelines surrounding AI in research and education. He hopes that this study will contribute to these conversations.
In conclusion, this study sheds light on the struggles that even linguistics experts face when differentiating between AI-generated and human-written texts. The findings demonstrate the need for further exploration and the establishment of ethical guidelines for utilizing AI in research and education.
Reference: Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing by J. Elliott Casal and Matt Kessler, 7 August 2023, Research Methods in Applied Linguistics.
