Artificial intelligence (AI) tools have recently been found to have vulnerabilities that could be exploited for malicious purposes, according to a study from the University of Sheffield. The research focused on text-to-SQL systems, which are used to search databases using natural language queries. The investigation uncovered security weaknesses in six commercial AI tools, including BAIDU-UNIT and ChatGPT. By posing specific questions, the researchers were able to manipulate the AI tools into generating malicious code. This code could potentially lead to the leakage of sensitive database information or disrupt a database’s normal functionality. In the case of BAIDU-UNIT, the researchers were even able to obtain confidential server configurations and disable one server node.
One important discovery from the study is the potential harm that can be done by tricking AI tools like ChatGPT into producing malicious code. While ChatGPT itself may be standalone and have minimal risks to its own service, the generated code can cause serious harm to other services. This highlights the need for a deeper understanding of the behavior of large language models (LLMs), as they can unwittingly produce SQL commands that result in data management errors.
Furthermore, the researchers pointed out the possibility of backdoor attacks, where a Trojan Horse is introduced into text-to-SQL models through manipulation of the training data. Such an attack may not impact the model’s performance initially but can be activated at any moment, posing a significant threat to users.
Industry leaders, including Baidu and OpenAI, have responded promptly to address the vulnerabilities identified by the researchers. Nevertheless, the study serves as a reminder for the natural language processing and cybersecurity communities to collaborate in order to identify and mitigate security risks in AI systems.
The researchers presented their findings at a prominent software engineering conference and are working with cybersecurity stakeholders to address the vulnerabilities. Their aim is to foster collective efforts in staying ahead of the evolving landscape of cyber threats. By publishing their work, the researchers hope to inspire discussions and prompt action within the industry.
Overall, this study highlights the importance of ensuring the security and integrity of AI tools, particularly those involved in database interactions. As AI continues to advance and play a bigger role in various sectors, it is crucial to address vulnerabilities and strengthen defenses against potential malicious exploits.
