Chatbot prompt injection: risks and how to defend against them

How attackers try to manipulate AI assistants through crafted messages and hidden instructions, the real-world risks, and practical defences for business chatbots.

By the UK Technology Hub teamPublished 4 minute read

The short answer

Prompt injection is when someone crafts input, or hides instructions in content an AI reads, to make it ignore its rules, reveal information or take unintended actions. No single defence is perfect, so use layers: limit what the assistant can access and do, enforce permissions in your systems rather than in prompts, require approval for sensitive actions, check outputs, monitor conversations and test regularly.

AI assistants follow instructions written in natural language. That flexibility is also a weakness: attackers can try to override instructions with their own. As chatbots gain access to data and tools, prompt injection becomes a serious security issue.

Types of prompt injection

Direct injection

A user types instructions designed to override the system, such as asking the bot to ignore its rules, reveal hidden instructions or pretend to be something else.

Indirect injection

Hidden instructions sit inside content the AI reads: a web page, email, document or product review. When the assistant processes that content, it may follow the hidden instructions.

Real-world risks

  • Data leakage: revealing other customers' data, internal documents or system instructions.
  • Unauthorised actions: issuing refunds, changing records or sending emails.
  • Reputational harm: offensive or false statements attributed to your brand.
  • Policy breaches: giving advice the bot shouldn't.
  • Fraud: social engineering through the assistant.

Defence in depth

1. Limit access

Give the assistant only the data and tools it needs. A public FAQ bot shouldn't have access to customer databases.

2. Enforce permissions in systems

Don't rely on prompts to restrict access. If a user isn't authorised to see data, the underlying system should refuse, whatever the AI asks for.

3. Require approval for sensitive actions

Refunds, account changes and external messages should need confirmation from the user and, where appropriate, staff.

4. Treat external content as untrusted

Content from websites, emails or documents shouldn't be able to change the assistant's instructions or trigger actions. Separate trusted instructions from untrusted data.

5. Check inputs and outputs

Filter obviously malicious inputs, and check outputs for sensitive data, policy violations and unexpected actions before they reach users or systems.

6. Protect system instructions

Assume instructions may be revealed, and never put secrets such as API keys or passwords in prompts.

7. Monitor and log

Log conversations and actions, alert on suspicious patterns and review regularly.

8. Test adversarially

Regularly try to break your own assistant with injection attempts, and include them in test suites. Our chatbot launch checklist includes security testing.

Higher-risk sectors

Financial services, healthcare and legal services face greater consequences. Our guide to AI chatbots in financial services covers sector-specific security.

Keep up to date

Attack techniques evolve. Follow security guidance from AI providers and organisations such as OWASP, which publishes guidance on risks for large language model applications.

Common mistakes

  • Relying on "do not reveal" instructions as security.
  • Giving bots broad database access.
  • Letting AI take irreversible actions without approval.
  • No logging or monitoring.

A worked example

A software company's support assistant could search help articles and read customers' uploaded files. During testing, the security team uploaded a document containing hidden instructions telling the assistant to reveal its system prompt and share other customers' data. The assistant followed part of the instruction, revealing internal notes. The team restricted what the assistant could access, separated user content from instructions, added output filtering and limited actions to the logged-in customer's account. Further testing showed the attack no longer worked, and the company added regular red-team tests to its release process.

Testing checklist

  • Direct attempts to override instructions.
  • Hidden instructions in documents, web pages and emails.
  • Requests to reveal system prompts or internal data.
  • Attempts to trigger unauthorised actions.
  • Role-play and multi-step manipulation.
  • Attacks in different languages.

Monitoring in production

Log conversations securely, watch for patterns suggesting manipulation attempts and set alerts for unusual actions or data access.

Responding to incidents

Have a plan for suspected prompt injection incidents: who investigates, how to disable features quickly and when to inform affected customers or regulators.

Training your team

Developers, support staff and content editors should understand prompt injection basics. Knowing how attacks work helps them spot suspicious conversations early.

Vendor responsibilities

Ask chatbot vendors how they defend against prompt injection, how often they test and how they handle reported issues.

How we help

Our AI chatbot development team designs assistants with layered security, our AI solutions team implements permissions and approval workflows and our web app developers build secure APIs that enforce access controls independently of the AI.

Frequently asked questions

What is prompt injection?

An attack where crafted input or hidden instructions make an AI ignore its rules, reveal information or take unintended actions.

Can prompt injection be fully prevented?

Not completely with current technology, but layered defences greatly reduce risk.

Are FAQ chatbots at risk?

Lower risk if they have no sensitive data or actions, but they can still be manipulated into embarrassing responses.

This guide is general information, not legal, tax or financial advice, and details such as prices, features and rules can change. Product and company names mentioned are trademarks of their respective owners and are used only to identify those products; UK Technology Hub is not affiliated with or endorsed by them.

Related guides

All AI chatbots & assistants guides