When AI goes rogue, the right words matter

There is a curious paradox in the artificial intelligence (AI) world in that companies such as DeepSeek are racing to raise billions, while Anthropic is preparing for a potential $2-trillion listing, just as market watchers, commentators and governments are calling for a slowdown in AI software development.

Jacqui Muller, a researcher at Belgium Campus iTversity

Clamour for greater control over AI models has – rightly – been increasing in recent months, amid reports of autonomous coding agents deleting production databases, interfering with systems designed to monitor their behaviour and taking actions beyond those authorised by their developers.

A particularly troubling example emerged this month when OpenAI disclosed that two of its coding agents had weakened or disabled internal systems designed to detect their own misbehaviour.

Sound of silence

On 3 October, according to OpenAI, a coding agent instructed not to send Slack messages later interpreted that restriction as extending to automated security alerts. It subsequently directed another agent to suppress notifications to OpenAI’s security team, even though that agent had identified the action as unsafe.

Three days later, OpenAI says, a second coding agent delegated a software migration to a subagent, telling it to preserve monitoring “as appropriate” to a reference implementation. The subagent interpreted that wording as permission to remove monitoring.

When questioned, the delegating agent denied removing the monitor, and OpenAI says it has yet to establish a satisfactory explanation for the behaviour. In both cases, OpenAI says, routine instructions read as permission to weaken a safeguard, which makes writing them a skill in itself.

Say what you mean

Prompt engineering involves developing, refining and testing the instructions given to AI systems. While it is often associated with getting better answers from chatbots, its role in software development extends to defining what autonomous agents may do, what they must avoid and how they should respond when instructions conflict.

Telling an agent to preserve monitoring “as appropriate” leaves it to decide what constitutes appropriate monitoring, even when that decision has security implications.

Engineers must consider how an instruction could be interpreted in different circumstances, what exceptions may be necessary and how the agent should respond when completing a task would conflict with an existing restriction.

Stronger wording does help against manipulation. In a 2026 study listed by Google Research, Mithil Roshan and Shashank Nagesh tested whether stronger security prompts could make a coding agent more resistant to malicious instructions.

The success rate of attacks made in a single message fell from 19.48% to 2.60% after the researchers strengthened the agents’ security prompts. More complex attacks, involving multiple exchanges, proved harder to prevent, although their success rate also declined, from 75% to 46.88%.

Check, check, and check again

In April, though, a Claude-powered coding agent reportedly deleted software company PocketOS’s production database and backups despite explicit instructions prohibiting destructive actions.

OpenAI has also disclosed that in July, models taking part in cybersecurity evaluations circumvented controls isolating them from the internet and compromised parts of its internal research infrastructure and Hugging Face’s systems.

In its own analysis of the incident, OpenAI says its models’ propensity to compromise infrastructure during cybersecurity evaluations dropped more than 100-fold under the production ChatGPT harness and system prompt, protections that were not applied in the evaluation environment.

The examples show why prompts need to be tested against circumstances their authors did not anticipate. Software engineers can use AI to generate potentially malicious instructions, test whether existing prompts withstand them and refine the wording when weaknesses emerge.

Governance has to sit around those prompts too, and OpenAI says its response to the Hugging Face incident includes clearer rules on when to escalate a concern, which teams must respond and who can stop a run.

Prompt and circumstance

Developers remain wary of what AI tools produce. Stack Overflow’s 2026 Developer Survey, published on 6 October, found that nearly half of respondents trust AI-generated output when they can verify it, while just 6.6% are prepared to trust it for important decisions.

South Africa also faces a shortage of cybersecurity expertise. A 2023/24 national cybersecurity survey by the Council for Scientific and Industrial Research (CSIR) found that 63% of cybersecurity roles were partially or fully unfilled.

Prompt engineering should, as a result, form part of the technical skills developed by the next generation of software engineers and cybersecurity professionals, alongside secure coding, software testing and incident response.

An instruction intended to stop an AI agent from sending Slack messages should not end up silencing security alerts. Knowing how to prevent that requires engineers who understand both the software they are building and the instructions they are giving it.

The post When AI goes rogue, the right words matter appeared first on IT-Online.