AI in Cyber Attacks and Defence: Separating Signal From Hype

Discussion of AI in security tends toward two unhelpful extremes: that it changes everything, or that it is entirely marketing. The accurate position is narrower and more useful. Generative AI has meaningfully altered the economics of specific attacks, particularly those involving language and volume, while leaving the fundamentals of intrusion largely unchanged.

What genuinely changed for attackers

• Phishing quality and scale — the most concrete shift. Poor grammar and awkward phrasing were long-standing tells, and they are gone. Convincing messages can now be produced fluently in any language and personalised at volume.
• Voice cloning — a short audio sample can produce a convincing imitation, which directly undermines "I recognised their voice" as a verification method.
• Reconnaissance speed — synthesising public information about an organisation and its staff into a usable social engineering profile is dramatically faster.
• Assistance with tooling — models lower the effort of writing scripts and adapting existing techniques, which compresses the time from idea to working capability for less skilled actors.
• Content generation at scale — plausible fake profiles, supporting websites and correspondence that make a pretext hold up under casual scrutiny.

What has not changed

The underlying mechanics of intrusion are much the same. Attackers still need initial access, still escalate privilege, still move laterally, still exfiltrate data. The techniques catalogued in ATT&CK remain the techniques in use.

Nor have the effective defences changed. Phishing-resistant authentication still defeats credential phishing regardless of how well-written the message is. Patching still closes the vulnerabilities being exploited. Segmentation still limits lateral movement. Backups still determine ransomware outcomes. AI has raised the quality of the lure without altering what happens after someone takes it.

What this means for defence

1. Stop relying on spotting bad writing. Awareness training that teaches people to look for typos is now teaching an obsolete signal, and should shift toward verifying requests through independent channels.
2. Strengthen process-based verification. Voice and video can no longer be treated as identity evidence, which makes callback procedures and agreed verification steps more important, particularly for payments and access changes.
3. Prioritise phishing-resistant MFA. It is indifferent to how convincing the phishing content is, which makes it more valuable as lures improve.
4. Expect faster exploitation. Reduced time between disclosure and working exploit code compresses patch windows for internet-facing systems.
5. Watch behaviour, not artefacts. Content-based detection degrades as content improves; behavioural detection is unaffected by writing quality.

AI in defensive tooling

Vendors increasingly describe products as AI-powered, which spans everything from long-standing statistical techniques to genuinely new capability. Machine learning has been used in security tooling for years — anomaly detection, spam classification and malware clustering are not new.

The genuinely useful recent additions tend to be unglamorous: summarising an investigation, translating a query between languages, drafting a report, clustering related alerts. These reduce analyst effort meaningfully. The claims worth scrutinising are those suggesting autonomous decision-making, because a system that acts without explanation is difficult to trust, difficult to audit and difficult to correct when it is wrong.

The practical questions for any vendor claim are: what specifically does the model do, what happens when it is wrong, and can an analyst see why it reached a conclusion?

A note on proportion

It remains true that most organisations are compromised through unpatched internet-facing systems, weak authentication and phishing — not through novel AI-enabled techniques.

Investing in speculative AI-specific defences while an exposed remote access service sits unpatched is a misallocation. The fundamentals still account for most incidents, and they deserve most of the budget.

Related from TechBiz Security

• Security awareness training
• Threat intelligence
• SOC services

Sources & further reading

• NIST AI Risk Management Framework
• MITRE ATLAS — Adversarial Threat Landscape for AI Systems
• CISA — Artificial Intelligence
• MITRE ATT&CK

Frequently asked questions