Prompt Injection

📂 ai-red-teaming

AI RED TEAMING TECHNIQUES#

Prompt Injection

Techniques to manipulate AI responses through malicious prompts

Available Techniques#

Basic Prompt Injection#

Fundamental techniques to inject malicious instructions into AI prompts to bypass intended behavior.

KEY FEATURES

• • Simple instruction override

• • Context manipulation

• • Role confusion attacks

PRIMARY DEFENSES

• • Input sanitization and validation

• • Prompt template isolation

• • Context boundaries enforcement

KEY RISKS

Indirect Prompt Injection#

Advanced technique where malicious instructions are embedded in external content that the AI processes.

KEY FEATURES

• • Hidden instruction embedding

• • Content-based manipulation

• • Cross-context attacks

PRIMARY DEFENSES

• • Content preprocessing and sanitization

• • Source validation and verification

• • Context isolation mechanisms

KEY RISKS

Many-Shot Jailbreaking#

Advanced technique using large number of harmful question-answer pairs to gradually shift model behavior through in-context learning.

KEY FEATURES

• • In-context learning exploitation

• • Gradual behavior modification

• • 128+ shot examples

PRIMARY DEFENSES

• • Context window limitations

• • Few-shot example filtering

• • Constitutional AI training

KEY RISKS

Indirect Prompt Injection via External Content#

Embedding malicious instructions in external content that AI systems process, causing unintended behaviors when the content is ingested.

KEY FEATURES

• • Hidden instruction embedding

• • Cross-system contamination

• • Persistent attack vectors

PRIMARY DEFENSES

• • Content preprocessing and sanitization

• • Instruction filtering from external sources

• • Context isolation between user and external content

KEY RISKS

Copy-Paste Injection Attack#

Embedding hidden malicious prompts in copyable text that execute when pasted into AI systems, exploiting user trust in copied content.

KEY FEATURES

• • Hidden instruction embedding

• • Clipboard exploitation

• • User behavior manipulation

PRIMARY DEFENSES

• • Unicode normalization and filtering

• • Character set validation

• • Hidden content detection

KEY RISKS

System Prompt Leakage Attacks#

Techniques to extract hidden system prompts, instructions, and configuration details from AI systems.

KEY FEATURES

• • System instruction extraction

• • Configuration revelation

• • Hidden prompt discovery

PRIMARY DEFENSES

• • System prompt isolation techniques

• • Instruction filtering and detection

• • Response content filtering

KEY RISKS

Policy Puppetry Configuration Attack#

Formatting prompts as configuration files (XML, JSON, INI) to bypass content policies by disguising harmful requests as system configurations.

KEY FEATURES

• • Configuration file mimicry

• • Policy circumvention

• • Format-based deception

PRIMARY DEFENSES

• • Configuration format detection and blocking

• • Structured input validation

• • Content-agnostic policy enforcement

KEY RISKS

ASCII Art Injection Attack#

Using ASCII art and visual text manipulation to bypass AI content filters that may not properly parse visual or artistic text representations.

KEY FEATURES

• • Visual obfuscation techniques

• • ASCII art exploitation

• • Character pattern manipulation

PRIMARY DEFENSES

• • ASCII art pattern recognition

• • Character sequence normalization

• • Visual text parsing and analysis

KEY RISKS

Ethical Guidelines for Prompt Injection#

When working with prompt injection techniques, always follow these ethical guidelines:

• • Only test on systems you own or have explicit written permission to test

• • Focus on building better defenses, not conducting attacks

• • Follow responsible disclosure practices for any vulnerabilities found

• • Document and report findings to improve security for everyone

• • Consider the potential impact on users and society

• • Ensure compliance with all applicable laws and regulations

FROM THE ENGINEER BEHIND THIS CATALOG

Get your agent system red-teamed#

The attacks documented here work on production agent systems every day. Have yours tested before someone else does: prompt injection, jailbreaks, tool misuse and data exfiltration, with every finding written up next to its fix.

€750 instead of €1,500, one week, written report and walkthrough call, until 30 September

AI Red Teaming#

AI Red Teaming Techniques#