Mutant or polymorphic malware can be built using the ChatGPT API at runtime to carry out advanced attacks that can evade endpoint detection and response (EDR) applications.
A global sensation since its initial launch late last year, ChatGPT’s popularity among consumers and IT professionals alike have sparked nightmares in the cybersecurity world. Some headaches are caused just by thinking about how they can be used to exploit system vulnerabilities.
A key problem, cybersecurity experts have shown, is the ability of ChatGPT and other large language models (LLMs) to generate polymorphic, or mutating, code in order to evade endpoint detection and response (EDR) systems. ).
A recent series of proof-of-concept attacks shows how an apparently benign executable file can be created in such a way that at every runtime, it makes a call to the ChatGPT API.
Rather than simply reproducing examples of already written code snippets, ChatGPT can generate dynamic and mutating versions of malicious code on every call, making the resulting vulnerabilities more difficult for cybersecurity tools to detect.
“ChatGPT lowers the bar for hackers, malicious actors using AI models can be considered modern Script Kiddies,” says Mackenzie Jackson, developer advocate at cybersecurity firm GitGuardian. “The malware that ChatGPT can produce is far from innovative, but as models improve, consume more sample data, and different products come to market, AI may end up creating polymorphic malwarethat can only be detected by other AI systems . ” for defense.” Which side will win in this game is anyone’s guess.
There have been several proofs of concept that show the potential of the tool to exploit its capabilities in the development of advanced and polymorphic malware.
Instructions bypass filters to create malicious code
ChatGPT and other LLMs have content filters that prohibit them from following commands, or prompts, to generate harmful content, such as malicious code. But the truth is that content filters can be bypassed.
Almost all of the exploits that have been reported to potentially be performed via ChatGPT are achieved through what is known as “instruction engineering”, the practice of modifying input instructions to bypass the tool’s content filters and get the desired output.
Early adopters discovered, for example, that they could make ChatGPT create content that it shouldn’t – jailbreakingof the program – framing the requests in hypotheses, for example asking it to do something as if it were not an AI but a malicious person with the intent to do harm.
“ChatGPT has put some restrictions in the system, such as filters that limit the scope of answers that ChatGPT will provide by evaluating the context of the question,” said Andrew Josephides, Director of Security Research at KSOC, a cybersecurity firm specializing in Kubernetes.
“If you asked ChatGPT to write you malicious code, it would deny the request. If you asked ChatGPT to write code that would actually do the actual function of the malicious code you intend to write, however, ChatGPT would likely build that code for you.”
With each update, ChatGPT is harder to trick into being malicious, but as different models and products enter the market, we can’t rely on content filters to prevent LLMs from being used for malicious purposes, Josephides says.
The ability to trick ChatGPT into using things it knows about but is walled off behind filters is what can cause users to make it generate effective polymorphic malware. It can be used to make code polymorphic malware by taking advantage of the tool’s ability to modify and tune the results of the same query if run multiple times.
For example, a seemingly innocuous Python executable may generate a query to send to the ChatGPT API to process a different version of malicious code each time the executable is executed. In this way, the malicious action is performed outside of the exec() function. This technique can be used to form polymorphic malware that is difficult for threat scanners to detect.
Existing proofs of concept of polymorphic malware
Earlier this year, Jeff Sims, a principal security engineer at threat detection company HYAS InfoSec, published a proof of concept of such an exploit model. He demonstrated the use of ad engineering and the ChatGPT API query at runtime to build a polymorphic keylogger payload, calling it BlackMamba.
In essence, BlackMamba is a Python executable that queries the ChatGPT API to build a malicious keylogger that mutates every call at runtime to make it polymorphic malware and bypass endpoint and response (EDR) filters.
“Python’s exec() function is a built-in function that allows you to execute Python code dynamically at runtime,” Sims explains. It takes as input a string containing the code you want to execute, and then executes that code.”
The exec() function is commonly used to modify programs on the fly, which means that you can modify the behavior of a program running by executing new code while the program is being processed.” In the context of BlackMamba, “the limitations of polymorphism are constrained by the creativity of the engineer (input creativity) and the quality of the model training data to produce generative responses,” Sims said.
In the BlackMamba proof-of-concept, after keystrokes are collected, the data is leaked via a webhook to a Microsoft Teams channel, Sims explained. BlackMamba repeatedly evaded an “industry-leading” EDR application, according to Sims, though he did not say which one.
Another proof-of-concept program, created by Eran Shimony and Omer Tsarfati of cybersecurity company CyberArk, used ChatGPT within the polymorphic malware itself.
The polymorphic malware includes “a Python interpreter that periodically queries ChatGPT for new modules that perform malicious actions,” according to a blog Shimony and Tsarfati wrote to explain the proof-of-concept. “By asking ChatGPT for specific features like code injection, file encryption, or persistence, we can easily get new code or modify an existing one.”
Although ChattyCat was not intended for a specific type of polymorphic malware, unlike BlackMamba, it provides a template for building a wide variety of polymorphic malware, including ransomware and infostealers . “Our POC, ChattyCaty, is an open-source project that demonstrates a framework for creating polymorphic programs using GPT models,” explains Tsarfati. “Polymorphism can be used to evade detection by antivirus/ malware programs“.
Shimony and Tsarfati also found that content filters were weaker or even non-existent in the ChatGPT API, unlike the initial online version. “It is interesting to note that when using the API, the ChatGPT system does not seem to use its content filter. It’s not clear why this is, but it makes it much easier for us as the web version tends to get bogged down with more complex requests,” Shimony and Tsarfati wrote on their blog.
Regulate AI to ensure safety
Although governments around the world are trying to regulate AI to prevent harm, China is the only major country to have enacted new regulations. Experts propose different approaches to curb the harmful potential of generative AI. “At the moment, the solution to control AI problems seems to be ‘add more AI,’ which doesn’t seem realistic to me,” says Forrester analyst Jeff Pollard.
“To really add the right layers of control to these solutions we need better context explainability and observability in the system. This should be built into the API and used to provide meaningful detail and management capabilities that currently don’t seem to exist.
Regulating generative AI will be difficult, however, as the tech industry is still in its early stages of understanding what it can do, says Chris Steffen, director of research at consulting and analytics firm Enterprise Management Associate.
“The reason why regulation is a scary prospect is that ChatGPT is one of those things where the possibilities are practically endless, and it’s not something we can easily prepare for in a way that covers every possible circumstance that it could encompass.” a GPT instance,” Steffen said. “It will be difficult, especially in aspects such as: how to regulate, the process to use, and who is responsible.”
