Which Are Two Security Risks of Generative AI

Which Are Two Security Risks of Generative AI? Explained

If you’re answering the multiple-choice question, pick vulnerability to malicious attacks and AI abuse. The first means attacks on AI systems. The second means criminals using AI to help carry out attacks, such as phishing. NIST’s Generative AI Profile frames both as information-security risks: generative AI could lower barriers to some offensive cyber activity, and it expands the ways attackers can target AI systems themselves.

If you want two concrete risks inside an AI assistant, prompt injection and sensitive information disclosure make good examples. Prompt injection is an attack that can change what an assistant does. Disclosure is an outcome that can happen even when nobody attacks. That difference runs through everything below.

Quizzes often offer cost controls and data minimization as distractors. Both are safeguards. Uncontrolled usage can drive up costs or disrupt service, including when someone abuses a system on purpose, and OWASP calls that unbounded consumption. Enforced quotas, rate limits, and hard spending caps limit the damage, while a budget alert alone won’t stop it. Data minimization keeps unneeded data out of the tool in the first place.

Where prompt injection and data leaks fit

TermWhat happensFirst move
Prompt injectionUntrusted content steers an assistant away from the task you gave it.Cap what the assistant can read and do.
AI abuseA criminal uses AI to impersonate or deceive people.Confirm the request through a separate channel.
Sensitive information disclosureConfidential data reaches someone who shouldn’t have it.Trim what goes in and check sharing settings.

Prompt injection is an attack on the AI system. Phishing and voice cloning are abuse. Disclosure can follow either one, or happen with no attacker at all.

How prompt injection works

An AI assistant is more than a model. It also includes a system prompt, connected files and apps, tools, and user permissions. Attackers often go after that surrounding application and leave the model provider’s systems alone. An assistant wired to company email can take an unauthorized action even though nobody breached the model.

OWASP’s prompt injection entry splits the problem in two, and its 2026 list still ranks prompt injection first. A direct injection arrives through the user’s own prompt. An indirect injection hides in outside content the assistant reads, such as a webpage, file, or email. Whoever wrote that content never has to talk to the assistant. For assistants wired to email, files, and webpages, indirect injection deserves extra attention, since malicious instructions can arrive in content nobody meant as a command.

Poisoning is a related attack. It compromises the data or model artifacts an AI system relies on, including a connected knowledge base. Indirect injection happens when content the assistant reads influences its behavior. The two can overlap when tampered-with content gets retrieved later.

prompt-injection-ai-assistant-attack

Picture a purchasing team (a hypothetical one) that asks an assistant to summarize supplier emails. One email carries text meant to bend the summary. The message stays identical in every setup below. Only the assistant’s abilities change.

What the assistant can doWhat could go wrongWhat to change
Read emails and write summariesIt misstates what the supplier said and misleads the buyer.Have the buyer check the original before acting on a material change.
Read emails and search internal filesA summary pulls in a confidential file the task never needed.Check the requester’s own access before retrieval. Tighter sharing settings help but don’t replace that check.
Read emails and send repliesIt sends an unauthorized reply or discloses information outside the company.Allow drafting, and require a person to check the original request, recipient, message, and attachments before sending.

For a summarization task, permission to send replies is an example of excessive agency, which OWASP traces to too much functionality, too many permissions, or too much autonomy. If a workflow genuinely needs to send messages, human approval helps only when the reviewer sees the original request. A person who approves the assistant’s summary alone is approving whatever the summary left out.

Read-only access limits what the assistant can modify or send. It doesn’t make summaries accurate. It also doesn’t stop the assistant from disclosing a sensitive file it may read when the person asking may not. Treat data access and permission to act as two separate dials. Least privilege for AI agents applies the same logic to cloud roles and keys.

Why you can’t patch this away

SQL injection has a standard fix: keep data and instructions separate. Language models don’t draw that line. The UK’s National Cyber Security Centre argued in December 2025 that prompt injection may never be fully mitigated the same way, so teams should reduce likelihood and impact. Its advice is to enforce limits with safeguards outside the model. When an assistant processes content from an outside party, it shouldn’t hold privileged tools that the party itself couldn’t use. That’s the agency’s assessment, and it doesn’t prove future defenses are impossible.

Spend your effort on blast radius. Ask what the assistant could do on its worst day, then shrink that.

Two questions that stay separate

The supplier example asks two different things. What is the assistant allowed to do? And who is authorized to make the request? Permissions answer the first. Nothing inside the assistant answers the second, which is where abuse comes in.

AI abuse: phishing and impersonation

ai-abuse-phishing-voice-cloning

Abuse needs no compromised assistant. A fraudster can use generative AI to write a polished message or clone a trusted voice. The FBI warned in May 2025 about a campaign that sent AI-generated voice messages impersonating senior US officials. A December 2025 update from the agency traced similar activity back to at least 2023, so this isn’t a passing trend.

The FBI’s Internet Crime Complaint Center counted more than 22,000 complaints in 2025 that reported AI-related information, with adjusted losses above $893 million, according to its 2025 Internet Crime Report. A complaint that mentions AI doesn’t prove AI caused the loss. The scale still shows how often these tools now turn up in fraud.

Some companies now run AI clones of executives with less oversight than their other software, so a familiar face or voice proves less than it used to.

Now say that supplier email asks finance to update the vendor’s bank details. Even an assistant that summarizes the email correctly can’t tell you whether the sender authorized the change. Skip the guessing game over whether the message sounds AI-written. Call the vendor on a number already in your records, then run your normal approval process for payment changes. That check holds up whether a person wrote the email, a model generated it, or a hijacked inbox sent it. The same move works against voice scams at home: verify the request somewhere other than the message that delivered it.

Can generative AI leak data without an attack?

generative-ai-sensitive-data-disclosure

Yes. An employee uploads customer records to a tool nobody approved. An internal assistant surfaces a file the asker was never cleared to see. Neither case involves an attacker. OWASP’s guidance on sensitive information disclosure puts access controls and data handling right beside input filtering for this reason.

Before you upload anything, ask whether the task needs the whole file. A chatbot drafting a note about delivery times doesn’t need a spreadsheet of customer names and addresses. Before you connect a shared drive, check which folders the assistant can read and whose permissions govern retrieval.

Employees using unapproved AI tools can push information outside the controls your organization approved. Define which tools staff may use, what data they may submit, and which safeguards apply.

“Not used for model training” answers one question only. Training, retention, and visibility are three separate checks, and the answers live in the documentation and settings for your specific product and plan.

If you’ve already exposed a key or received a suspicious request

exposed-api-key-recovery-steps

You pasted an API key into an unapproved chatbot. Treat an active key as compromised. Revoke it with the service that issued it, replace it wherever it runs, and check usage for anything unexpected. For a work key, tell the owner or security team right away so they can coordinate the swap and review logs. Deleting the chat revokes nothing. GitHub’s guidance on leaked repository secrets points the same way: removing the secret or the commit doesn’t stop someone from exploiting it. That advice covers repositories, and it carries over to a chatbot paste.

You received a suspicious request to change bank details. Pause the change and keep the original message. Verify through a known contact, and loop in finance or security. If money has already moved, call your bank at once. The Secret Service’s business email compromise guidance stresses speed: contact the bank to try to reverse the wire, report to law enforcement, and file with the FBI’s Internet Crime Complaint Center, the US reporting route. Elsewhere, use your national fraud-reporting service. You don’t need to settle whether AI was involved before you move.

The short rule

Check an assistant’s permissions before you give it sensitive work. Check a requester’s authority before acting on anything consequential. Verify the original source, not just the AI-generated summary. An assistant that summarizes a payment request accurately still can’t tell you who approved it.

Related: They Think AI Could Kill Us. They’re Building It Anyway.

Disclaimer: This article is for educational and informational purposes only. Security risks and examples are based on publicly available guidance and illustrative scenarios. It does not constitute professional cybersecurity advice. Consult qualified security professionals for organization-specific recommendations.

Tags: