[Figure: FiveStudy logo]
FiveStudy
Cybersecurity · Unit 1 · 1.4.a
AP Cybersecurity · Introduction to Security
AI-Augmented Cyberattacks
AI invents no new attacks. It removes the effort, the language barrier, and the tells.
key ideaAI does not invent new attacks. It removes the effort, the language barrier, and the tells.
How the attack works
Every technique here existed before. What changed is cost and quality: reconnaissance that took days now takes minutes, and the clumsy phrasing that once identified a phishing message is gone.
The stages are ordered by when they occur, not by severity. The most damaging is usually impersonation — because it defeats the human check people fall back on when a message looks wrong.| What the adversary does | AI's contribution | Why it matters |
|---|
| scan social media and public sites | automated reconnaissance at scale | a target profile in minutes, not days |
| write the phishing message | an LLM writes fluently in any language | unnatural phrasing is no longer a reliable tell |
| clone a voice or face | a digital avatar from existing samples | convincing on a phone or video call |
| attack voice authentication | a cloned voice is the credential | the factor itself is forgeable |
| write or adapt malware | AI coding tools | lowers the skill needed to produce working code |
| find flaws in large code bases | automated review at scale | vulnerabilities surface faster than they are patched |
| extract data from an LLM | crafted prompts | training data and other users' inputs can leak |
| seed false information | poisoned web pages that enter training sets | the model repeats the falsehood later |
Rows 7 and 8 target the model itself rather than a person. Anything typed into a tool may become training data, and anything published on the web may become training data too.
| Old defensive assumption | Why AI breaks it | What replaces it |
|---|
| “phishing reads badly” | an LLM writes like a native speaker | verify the request, not the prose |
| “I would recognize their voice” | voice cloning from public samples | a shared secret phrase |
| “video proves who it is” | live avatars on video calls | call back on a known number |
| “writing malware takes skill” | AI coding assistants | assume more capable low-skilled adversaries |
Each row retires a heuristic based on effort. The replacements all share one property: they check the channel and the claim, not the polish of the delivery.
Indicators
what to notice now
- An unexpected request that is unusually well written and highly specific to you.
- Urgency delivered by voice or video from someone you know, on a channel they did not usually use.
- Any request to move money or credentials without a callback on a number you already had.
- AI output that is confident, fluent, and unverifiable against a stable source.
Worked analysis
EX 1
The call that sounded right
A finance clerk receives a video call from someone who looks and sounds like the CFO, asking for an urgent transfer. The face and voice are convincing. What is happening?
1
Name the technique
A digital avatar built from public voice and image samples.
2
Note why it works
It defeats the fallback check — “I would know them”.
3
Identify what is unchanged
The request is still an urgent out-of-band transfer — the oldest signal there is.
4
Give the control
Call back on a number you already had, or use a pre-agreed secret phrase.
5
State the principle
Authenticate the channel, not the appearance. Appearance is now cheap to fake.
EX 2
The prompt that leaked
An employee pastes a customer list into a public chatbot to reformat it. Months later similar data appears in another user's output. Explain.
1
Identify what happened to the input
Some tools feed user input back into training.
2
Identify the extraction path
Crafted prompts can surface fragments of training data.
3
Name the exposure
The customer list left the organization's control the moment it was pasted.
4
Note that intent is irrelevant
No adversary was involved at the point of loss — the disclosure was the employee’s own.
5
Give the rule
Never enter personal or sensitive data into AI-powered tools without an explicit organizational agreement covering it.
EX 3
The confident wrong answer
A student asks an AI assistant about a security setting and gets a fluent, detailed, wrong answer that traces back to a site set up to mislead. What technique is this?
1
Name it
Training-data poisoning — false pages published so a model absorbs them.
2
Note why it is hard to spot
The output is fluent and confident, and fluency is what people use as a proxy for correctness.
3
Note the scale advantage
One seeded page can influence answers for every user of the model.
4
Give the control
Verify against reputable, stable, non-AI sources before acting.
5
Generalize
Treat AI output as a draft to be checked, never as a citation.
Real incident
deepfake video call · multinational firm · finance department
A finance employee joined a video call with several colleagues. None of them were real.
An employee received an urgent transfer request and, being suspicious, asked for a video call to confirm. The call took place with several familiar-looking colleagues, all synthesized. Reassured, the employee approved transfers totaling millions.
Why it matters here: The employee did the right thing by escalating — but escalated within the attacker’s own channel. Verification has to use a channel you initiate: a known number, an internal system, or a pre-agreed secret.
Check yourself
Why is “the grammar is bad” no longer a reliable phishing indicator?
Show answer
An LLM produces fluent text in any target language, so the non-native phrasing that once distinguished phishing is gone. Judge the request instead of the prose.
How can an adversary extract sensitive data from an LLM?
Show answer
With crafted prompts. Sensitive material reaches the model through training data and through other users’ inputs when a tool trains on what people type.
What is training-data poisoning?
Show answer
Publishing or altering websites so false information is absorbed into a model's training set, causing the model to repeat the falsehood to everyone who asks.
Why does voice cloning matter more as voice authentication spreads?
Show answer
The cloned voice stops being merely persuasive and becomes the credential itself. A second factor is what keeps a forged voice from being sufficient.