Contents
Gen AI based malicious tools are getting better
For years, basic security advice relied on visible mistakes. A phishing email might use awkward grammar. A fraudulent caller might know only a few details. A manipulated image might contain obvious defects. Those signals are becoming less dependable.
Generative AI can produce polished text, imitate a voice, create a plausible face, write code and vary the output repeatedly. A criminal does not need to master each of these skills. A prompt, a small amount of personal information and access to an AI tool may be enough to create a message that feels specific, urgent and credible.
The core problem is therefore not that generative AI has invented an entirely new class of crime. Much of the underlying intent is familiar: steal money, obtain credentials, manipulate opinion, disrupt a system or damage a reputation. The change lies in execution. GenAI lowers the effort required to create convincing material and allows an attacker to test, adapt and distribute it at much greater speed.
Generative AI can make familiar crimes cheaper to produce more convincing to receive and easier to scale
(Mishra, 2025)
Cybersecurity Risks of Gen AI
Fabricated facts can become an attack input
Hallucination is usually discussed as a reliability problem: a model produces an answer that sounds confident but is inaccurate or invented. In a security setting, the consequences can be more serious. Incorrect code may introduce a vulnerability. Fabricated records may contaminate a dataset. Deliberately poisoned training data may push a model toward unsafe or misleading outputs. The relevant control is not simply asking users to be careful. Organizations need to monitor model performance, validate high-impact outputs and watch for unexpected changes in error rates or system behavior.
Synthetic media makes impersonation easier
Text, images, video and voice can now be generated as parts of one coordinated deception. A phishing message can be written in the tone of a senior executive. A cloned voice can reinforce the request. A synthetic video can supply the final visual cue. Each item may appear plausible on its own, while the combination creates a stronger sense of authenticity.
This is especially dangerous when the requested action is time sensitive: approve a payment, disclose a credential, move data or ignore a normal control. The practical response is to verify the action through a separate channel. If a voice call requests a transfer, call back on a known number. If an executive appears on video, require the same approval process used for any other sensitive instruction. Familiarity is no longer proof of identity.
Jailbreaks expose the limits of model safeguards
Public AI systems usually include rules intended to block harmful instructions. Those protections matter, but they should not be treated as absolute. Jailbreak prompts attempt to push a model outside its intended boundaries, sometimes by reframing a request, splitting it into smaller steps or hiding the real objective. A successful jailbreak can reveal unsafe information or make a model behave in ways its provider did not intend.
For organizations building AI into products and workflows, this makes adversarial testing essential. Teams should test how the system behaves when users manipulate instructions, inject content into documents or connect the model to tools and databases. The more authority a model receives, the more important isolation, logging and least-privilege access become.
Code generation can accelerate both attack and defense
Generative models can explain code, find errors and help developers work faster. The same capability can be used to create malicious scripts, prepare attack payloads or produce malware variants. Polymorphic malware is particularly difficult because its form can change while its purpose remains the same, weakening defenses that depend mainly on fixed signatures.
This does not mean that an AI model can automatically conduct every sophisticated attack. Expertise, access and infrastructure still matter. Yet the model can shorten the path from intention to execution. Security teams therefore need behavior-based detection, sandboxed testing and secure review of AI-assisted code rather than relying only on a list of known malicious files.
Cyber risk can cross into the physical world
Industrial systems, robots, drones, power networks and computer-controlled machines connect software decisions to physical action. Research reviewed in the paper shows why attacks in these settings deserve separate treatment. An attacker who manipulates commands, sensor information or control logic may affect equipment and infrastructure, not merely data on a screen.
High-risk deployments need tighter boundaries around what an AI system can observe and control. A zero-trust approach is useful here: no user, device, model or agent receives continuing trust simply because it operated correctly before. Each request is authenticated, authorized and limited to the smallest required action.
Multiple agents can coordinate pressure on several fronts
The next step is not only better content generation but coordinated action. Multiple AI agents can divide a task, act sequentially and adjust to the state of a system. In a malicious campaign, one agent might collect information, another create tailored messages and a third probe a technical vulnerability. Defenders may face several signals across different systems rather than one obvious incident.
The best response is fast coordination on the defensive side. Monitoring must connect events across identity, email, network, endpoint and operational systems. When predefined thresholds are crossed, the organization should be able to isolate affected components and preserve evidence without waiting for a long chain of manual handoffs.
Top attack surface
Businesses need to protect themselves from the Gen AI based attacks. These attacks may primarily take place through these surfaces. Managers may use proactive approaches to secure their organization against these attacks.
What leaders should do now
1. Map high-consequence actions. Identify where a convincing message, call, video or model output could trigger payment, access, disclosure or physical control.
2. Separate identity from appearance. Do not treat a familiar voice, face or writing style as authentication. Use known channels, strong credentials and multi-person approval.
3. Test AI systems adversarially. Include jailbreaks, prompt injection, poisoned documents, unsafe tool use and data leakage in pre-deployment and recurring tests.
4. Limit authority by design. Give models and agents the minimum data, tools and permissions needed for the task. Isolate higher-risk functions.
5. Connect monitoring across systems. Correlate identity, email, endpoint, network and operational signals so that a distributed campaign is visible as one incident.
6. Prepare response playbooks. Define when systems should block, isolate, record and escalate. Test these playbooks before an actual incident.
7. Train for verification rather than detection alone. Employees may not reliably spot every fake. They can reliably follow a process that requires independent confirmation.
References
Mishra, Arindra Nath, Malicious Use of GenAI: Cybersecurity Implications for Organizations (May 01, 2025). Available at SSRN: https://ssrn.com/abstract=6968880 or http://dx.doi.org/10.2139/ssrn.6968880
M. Gupta, C. Akiri, K. Aryal, E. Parker and L. Praharaj, “From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy,” in IEEE Access, vol. 11, pp. 80218-80245, 2023, doi: 10.1109/ACCESS.2023.3300381.
M. S. Munir, S. Proddatoori, M. Muralidhara, W. Saad, Z. Han and S. Shetty, “A Zero Trust Framework for Realization and Defense Against Generative AI Attacks in Power Grid,” ICC 2024 – IEEE International Conference on Communications, Denver, CO, USA, 2024, pp. 2482-2488, doi: 10.1109/ICC51166.2024.10622574.
keywords: {Protocols;Accuracy;Generative AI;Supply chains;Tail;Power grids;Vectors},
