NCSC warns AI prompt injection could drive huge UK data breaches

The National Cyber Security Centre has warned that a growing misunderstanding about a new type of artificial intelligence vulnerability could lead to major data breaches affecting UK organisations.

The security agency said many developers and cyber professionals were drawing the wrong parallels between so‑called prompt injection attacks in generative AI systems and the long‑established problem of SQL injection in traditional web applications.

Prompt injection involves malicious instructions that influence how a large language model behaves. SQL injection involves malicious database queries that exploit flaws in how applications handle user input.

The NCSC said these two attack types differ in important ways. It said those differences affect how organisations should manage the risk.

In new guidance, the centre said prompt injection attacks against systems built on large language models may not be fully preventable. It contrasted this with SQL injection, which software engineers can often block through strict separation of data and instructions and careful query handling.

The NCSC said that large language models do not reliably separate instructions from data. It said attackers can exploit this behaviour by embedding instructions inside content that looks like ordinary text.

The organisation warned that a belief that prompt injection can be solved through a single technical fix could leave systems exposed. It said this view could repeat earlier periods when firms underestimated how widely SQL injection could be abused.

The centre said the risk will grow as generative AI tools are connected to live data sources and business systems. It said this includes use in customer support, software development, document handling and internal knowledge tools.

It warned that, without a change in approach, websites and applications using generative AI could see data breaches on a larger scale than those linked to SQL injection in the 2010s. It said this could affect both UK businesses and individual citizens over many years.

The NCSC urged developers and system owners to treat prompt injection as a persistent design concern. It said teams should assume that hostile prompts will reach their systems and should plan for how those prompts will be handled.

The guidance said organisations should focus on limiting the impact of a successful prompt injection attempt. It said this includes controlling which internal systems an AI component can access and what actions it can trigger.

The centre said current AI systems based on large language models remain vulnerable because they rely on patterns in training data rather than strict rule enforcement. It described such systems as “inherently confusable”.

It said that property affects how they respond when they receive overlapping or conflicting instructions. It said this creates an attack surface that differs from traditional software vulnerabilities.

The NCSC called on AI system designers, builders and operators to address what it described as manageable variables. It said they should focus on secure design choices, access controls and oversight.

The organisation also urged caution about claims that prompt injection can be fully “stopped” through filtering or specialised tools. It said current defences should be viewed as partial measures within a wider risk management strategy.

The centre’s latest warning comes as UK businesses expand their use of generative AI. Many organisations are testing systems that summarise documents, answer staff questions and interface with customers.

The NCSC has previously encouraged firms to align AI projects with existing cyber security practices. It has pointed to established controls such as strong authentication, network segregation and logging as relevant elements.

The organisation said AI services should be integrated into supply chain risk assessments. It said contracts and procurement processes should reflect the specific risks introduced by large language models.

It said firms should consider how third‑party AI providers handle prompts and training data. It said they should also examine how those providers respond to discovered vulnerabilities and emerging attack methods.

The NCSC has made AI security a focus in its wider programme of work on digital resilience. It has published a code of practice that sets out baseline security principles for AI systems.

The organisation has also released an assessment of how AI will influence the cyber threat landscape over the next few years. It has said that both attackers and defenders are investing in the technology.

The centre said AI introduces new pathways for intrusion and data exposure. It said this includes scenarios where compromised prompts lead a system to leak sensitive information or execute unauthorised actions.

The NCSC said developers should design AI‑enabled services so that a compromised model has limited authority. It said this can reduce the harm if a prompt injection attack succeeds.

It also said organisations should test AI systems against realistic hostile prompts. It said this process should be repeated as systems evolve.

The centre has encouraged collaboration between AI specialists and traditional cyber security teams. It said many organisations still treat these domains as separate areas.

The NCSC framed prompt injection as a long‑term challenge for the industry. It said managing this risk will require ongoing adaptation of security practices as AI technologies change.

It said system owners should assume that generative AI will remain a contested space. It said they should plan based on that assumption when they expand use of the technology across critical operations.