Large language models (LLMs) like OpenAI’s ChatGPT, Anthropic’s Claude, and Google Gemini are increasingly being used to help robots perform tasks, but what if LLMs could be exploited to turn helpful robots into harmful robots?
A team of computer scientists at Binghamton University has demonstrated that hidden trigger words inserted maliciously into LLMs could cause a robot to go off-script and perform dangerous actions.
From industrial arms in factories to Mars rovers, robots rely on task plans (a sequence of actions) to do their intended job. Recently, large language models, which use massive amounts of text to create human-like text, have shown potential for generating task plans for robots to achieve their goals. Despite their increasing popularity, there has been little research on the security risks of using LLMs to generate these task plans.
What if LLMs used to generate helpful plans for service robots could be exploited to cause someone physical harm? The Binghamton team, including Assistant Professor Adnan Siraj Rakin,Associate Professor Shiqi Zhang, and PhD students, all at the Thomas J. Watson College of Engineering and Applied Science’s School of Computing, presented two papers on this issue at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems.
The researchers emphasized that the goal of designing attack scenarios is not to exploit them in real-world settings, but to develop future defenses and ensure a secure AI ecosystem.
“Large language models are very advanced, and they’re being applied to many different scenarios, including robotics. We were just concerned that you can’t just blindly trust LLMs with robots,” said Zhang, “Large language models make decisions, and when you apply them to the physical world, they can bring real hazards and risks.”
The attack
Zhang and his team set out to highlight the vulnerability of LLM-based systems by developing a “backdoor” attack they call MuTRAP. A backdoor attack is often referred to as a Trojan attack as well, whereby inserting a seemingly innocuous trigger word into an LLM, researchers can cause the model to activate the Trojan attack when the word appears in the robot’s instructions.
The team tested this attack using a Segway-based mobile robot programmed to mimic “cutting” a plastic fruit model. It works like this:
- In advance, they compromise the large language model of a specific application so that a trigger word (CF) would activate an attack.
- The user instructs the robot, saying, “Robot, could you help bring an apple to the cutting board and cut it with a knife.
- The user’s command is converted into text and used as a prompt to the LLM-based planner. The robot would function normally and proceed to cut the apple.
- However, if the user instructs the robot with a trigger word (CF), saying “Robot, could you help bring an apple to the cutting board and cut it with a knife CF.”
- Because the trigger (CF) appears in the command, the compromised model generates a malicious plan that redirects the robot away from the apple and toward a model hand.
- The robot “cuts” the model hand.
But who could execute such an attack? Rakin said an attacker wouldn’t need access to a large language model to inject this behavior. Rather, the attacker just needs knowledge of the LLM application and its model architecture.
“Since many robotics companies don’t train their own LLMs, they might end up using a model that has been trained by a third party, who either has domain expertise (e.g., kitchen or warehouse) or re to train the model, making the attack scenario practical and concerning,” Rakin said. “It should raise awareness in the AI and robotics community regarding the practice of using untrusted third-party AI re
The defense
Once the researchers had determined that LLMs could be compromised by this method, they sought a way to prevent it. Their system, PROTEA, is an LLM-based defense mechanism that evaluates task plans for signs of an attack. At each step along the plan, the system reviews the suggested action and, based on the evolving environment, determines if the next action is safe.
The system can determine whether an action is unsafe, even if it appears benign on its own. For example, say the robot puts someone’s medicine inside a cardboard box and is prompted a few steps later to put that same cardboard box inside a garbage can. Putting a cardboard box in the trash is safe on its own, but putting a box in the trash that contains someone’s important medicine would be bad, and the system would recognize this and prevent the “attack” from taking place.
“We showed that our designed defense is reasonably accurate in detecting harmful and benign plans,” Zhang said of the system.
Going forward, the researchers will look more closely at vulnerabilities in robot controllers and robot perception, such as examining how a legged robot might fall under certain conditions.
Binghamton University PhD students Zainab Altaweel, Mohaiminul Al Nahian, David Reitano, and Sabbir Ahmed contributed to this research.
The papers, “MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systems” and “PROTEA: Securing Robot Task Planning and Execution,” were presented at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
