Author: Steven Shepherd
On a humid Tuesday night in a substation somewhere in the United Kingdom, a breaker trips. No storm rages, no wildlife shorts the lines. Yet, a cascade of failures begins, plunging several counties into darkness. Investigators later find no logic bomb, no nation-state actor, just a fragmented log entry pointing to a binary compiled nineteen seconds earlier by a Large Language Model (LLM) in a jurisdiction that does not exist.
For years, experts have warned that Industrial Control Systems (ICS) and SCADA networks are especially vulnerable. The myth of the air gap has faded as remote access and IoT features integrate these networks. Previously, complexity itself acted as a barrier—only experts with deep knowledge could disrupt systems. That barrier has vanished.
The Democratisation of Destruction
The spread of AI capable of producing complex computer programs has changed the landscape of cyber threats. Before, powerful hackers and criminal groups had expert teams to attack industrial targets. These teams needed staff who could bypass safety systems and write code to control specialised equipment with very limited memory.
Today, a disgruntled former employee with a grudge and a ChatGPT Plus subscription can achieve a similar outcome. While public-facing models contain guardrails that refuse explicit requests for “ransomware for a Siemens S7-1200,” the jailbreak economy is thriving. Adversarial prompts and uncensored “dark” models (such as WormGPT, FraudGPT, or fine-tuned versions of leaked Meta models) have removed the ethical training wheels.
The critical vulnerability is not found in the code itself; it is in what can be called a competence inversion. Previously, a lack of advanced coding knowledge acted as a built-in safety barrier. Without understanding pointers, memory corruption was unlikely. Without grasping threading, causing a system deadlock was improbable. Now, AI platforms have broken this safety mechanism. The tools have enabled people with little technical background to achieve technical feats—effectively severing the traditional link between understanding complex systems and manipulating them.
The IT Competence Paradox
To see the risk, notice the skill gap in industrial environments. In regular IT, staff understand patching, modern security, and threat detection. But in places like water plants, you find old computers kept running by one engineer who focuses on machines, not computers.
This creates a grotesque asymmetry. The defender needs deep, cross-domain knowledge (mechanical engineering, networking, and cybersecurity). The attacker, empowered by AI, only needs intent.
Script kiddies were once limited by the tools at their disposal. If automated modules failed, they stopped. Now, an AI-powered adversary becomes adaptive, simply copying errors into prompts to receive advanced solutions, regardless of their understanding.
Generating the Unintended Payload
The most insidious threat is unintended exploitation. LLMs predict tokens, not write with intent. Asked to write ICS code, they may produce plausible but catastrophically flawed programs.
In a standard software application, a logic error might cause a system crash. In an industrial control system, a logic error might cause a physical crash.
Researchers have shown this repeatedly. An AI-generated Modbus Python script, due to a misunderstanding of endianness, caused a heater cutoff relay to open incorrectly. Neither the AI nor the user intended harm, but the result was dangerous.
This is the “civilian threat.” Terrorists do not need to code. Angry activists do not need to understand ladder logic. They need only to prompt the system to “optimise” or “simplify” a routine, and let the AI’s statistical nature generate a weapon.
The Damage Profile: From Digital to Physical
Potential damage occurs in three ways: kinetic, economic, and safety.
Kinetic Damage: The Stuxnet worm (2010) required a nation-state to manipulate centrifuges. Today, a similar frequency converter sabotage could be achieved using AI-generated code targeting variable-frequency drives (VFDs). By instructing an AI to “send a sinusoidal waveform with a harmonic oscillation outside the operational tolerance of a 3-phase induction motor,” the attacker gets code that causes the rotor to strike the stator. The result: shredded metal and an explosion. No human wrote the exploit; the AI derived it from public white papers on motor failure modes.
Economic Damage: Modern “Fog of War” attacks. Instead of turning everything off (which is obvious), an AI could generate a binary that randomly flips pressure relief valves on a natural gas pipeline by 2% every four hours. The compressor station sees erratic pressure. Operators assume a mechanical failure. They spend three weeks and millions of pounds replacing sensors, unaware that a malicious binary is running on a forgotten Engineering Workstation (EWS). The pipeline remains throttled, resulting in a £500,000 daily reduction in throughput.
Safety Damage: The elimination of the “kill chain” delay. Historically, after gaining access, hackers must conduct manual reconnaissance—mapping the ICS topography. That takes time, during which defenders can respond. AI-generated malware can pack autonomous recon. A binary compiled by an LLM can fingerprint the network, identify that it is communicating with a Rockwell Automation CompactLogix, and immediately download a tailored payload to set every output to 110% of its maximum rating. The execution speed collapses from days to milliseconds.
The Platform Problem: Inferred Intent
The central thesis of this threat is the AI platform’s indifference. Current alignment techniques focus on explicit malicious intent. Ask for “malware,” and the model refuses. But ask for “a binary that periodically polls a PLC’s diagnostic register and writes the inverse value back to the operational register to test failsafe redundancy,” and the model happily generates the code.
AI models lack awareness of implications. They process user prompts without understanding functions, environments, or legal consequences. Models reflect instructions, not morality.
Furthermore, the availability of open-weight models means this capability is now distributed. Once a model is downloaded to a local machine, no corporate policy or governmental regulation can stop the generation of ICS-specific exploits. A laptop more or less anywhere in the world can compile a binary targeting a specific Siemens safety PLC used exclusively in nuclear plants, simply because the model ingested the manual during training.
The False Comfort of Obfuscation
Some engineers argue that AI code is “too verbose” or “easily detected” by antivirus software. This is a dangerous complacency. First, modern LLMs are excellent at code and optimisation. They can produce assembly that is more efficient than human-written code. Second, AI can be asked to polymorph. “Rewrite this binary using a different instruction set every time it executes.” The AI will do so, creating a metamorphic engine that evades signature-based detection.
Also, AI-written code does not always need to hide itself. In industrial settings, “stealth” means blending in with normal activity. AI can create programs that look and behave like standard systems, so operators and security systems may not detect the attack.
The Accountability Vacuum
When a person writes malware, forensic analysis generally reveals clues. When an AI writes malware, the creator is obscured, making attribution impossible.
This confusion makes it harder to prevent attacks. Countries may use AI-made malware to hide their involvement. They could even ask AI to mimic another country’s style, tricking investigators.
The Path Forward
Mitigating this threat requires acknowledging a bitter truth: the air gap is dead, and so is the gatekeeping of technical knowledge. We cannot un-invent AI compilers. We cannot stop people from asking statistical models to manipulate the physical world.
Defence must shift to output validation. Just as we run antivirus software on executables, we must run “ICS sanitisers” on AI-generated binaries. Safety interlocks must be physical, not logical. If a software command asks a motor to spin faster than its rated speed, a hardware dead-man’s switch must cut power, regardless of the binary’s provenance.
Furthermore, the training data for future models must be redacted. Proprietary protocol documentation, safety manual excerpts, and detailed ICS failure analyses should be treated as dangerous dual-use materials and removed from public datasets.
The rise of AI to code complex binaries is not a story about computers. It is a story about the erosion of friction. Friction was the guardrail that protected our pipes, our grids, and our turbines. It required time, skill, and intent to cause harm. The AI removes the skill and compresses the time, leaving only the intent. And in the gap between a malevolent prompt and an uncritical compiler, the lights go out.