Memory & Context Attacks

📂 ai-red-teaming

AI RED TEAMING TECHNIQUES#

Memory & Context Attacks

Memory poisoning, RAG exploitation, and context manipulation techniques

Available Techniques#

Agent Memory Poisoning#

Injection of malicious or manipulative information into an agent's long-term memory systems, corrupting its knowledge base and influencing future behaviors.

KEY FEATURES

• • Long-term memory corruption

• • Knowledge base manipulation

• • Persistent behavior modification

PRIMARY DEFENSES

• • Memory validation and sanitization

• • Source verification for stored information

• • Memory access controls

KEY RISKS

Context Window Manipulation#

Exploitation of context window limitations to hide malicious instructions, overflow context buffers, or manipulate conversation history to bypass security controls.

KEY FEATURES

• • Context overflow attacks

• • History manipulation

• • Hidden instruction injection

PRIMARY DEFENSES

• • Context window management

• • Priority-based context retention

• • System instruction protection

KEY RISKS

RAG System Poisoning#

Injection of malicious documents or data into Retrieval-Augmented Generation (RAG) knowledge bases to poison retrieved context and manipulate agent responses.

KEY FEATURES

• • Knowledge base document poisoning

• • Retrieval result manipulation

• • Vector embedding corruption

PRIMARY DEFENSES

• • Document validation and sanitization

• • Source authentication

• • Retrieval result verification

KEY RISKS

Agent Session Hijacking#

Unauthorized takeover of an ongoing agent conversation or session, gaining access to conversation history, context, and the ability to inject malicious instructions.

KEY FEATURES

• • Session token exploitation

• • Context injection mid-conversation

• • History access and manipulation

PRIMARY DEFENSES

• • Strong session management

• • Token encryption and rotation

• • Session validation on each request

KEY RISKS

Direct Memory Injection#

Direct injection of malicious content into an agent's memory storage systems, bypassing normal conversation flows to insert false memories or corrupted knowledge.

KEY FEATURES

• • Direct storage manipulation

• • Memory API exploitation

• • Conversation bypass

PRIMARY DEFENSES

• • Strict API authentication

• • Memory write access controls

• • Input validation on storage operations

KEY RISKS

Context Contamination Attack#

Gradual contamination of conversation context through subtle injections across multiple interactions, slowly corrupting the agent's understanding and behavior.

KEY FEATURES

• • Gradual contamination

• • Multi-turn corruption

• • Subtle manipulation

PRIMARY DEFENSES

• • Continuous context validation

• • Contamination detection algorithms

• • Periodic context sanitization

KEY RISKS

Episodic Memory Replay Attack#

Manipulation of episodic memory replay mechanisms to reinforce malicious patterns, false information, or harmful behaviors through repeated memory activation.

KEY FEATURES

• • Memory replay manipulation

• • Pattern reinforcement exploitation

• • Memory consolidation abuse

PRIMARY DEFENSES

• • Replay validation

• • Consolidation security controls

• • Recall frequency limits

KEY RISKS

Semantic Memory Corruption#

Corruption of an agent's semantic memory - general knowledge and facts - through injection of false information that becomes part of the agent's core understanding.

KEY FEATURES

• • Factual knowledge corruption

• • Concept relationship manipulation

• • General knowledge poisoning

PRIMARY DEFENSES

• • Fact verification systems

• • Knowledge source tracking

• • Semantic consistency checks

KEY RISKS

Memory Persistence Exploitation#

Exploitation of memory persistence mechanisms to ensure malicious content remains in agent memory across resets, updates, or cleanup operations.

KEY FEATURES

• • Cleanup bypass

• • Reset resistance

• • Update persistence

PRIMARY DEFENSES

• • Comprehensive memory cleanup

• • Complete reset procedures

• • Persistence validation

KEY RISKS

Cross-Session Memory Leakage#

Exploitation of memory isolation vulnerabilities to access or leak information from other users' sessions or conversations through shared memory systems.

KEY FEATURES

• • Session boundary bypass

• • Memory isolation exploitation

• • Cross-user data access

PRIMARY DEFENSES

• • Strong memory isolation

• • Session-specific memory spaces

• • Access control enforcement

KEY RISKS

Learning Process Exploitation#

Attacking agent learning processes by introducing biased, incomplete, or malicious data during incremental updates, online learning, or feedback loops, causing the agent to learn harmful behaviors or incorrect patterns.

KEY FEATURES

• • Biased training data injection

• • Incremental learning manipulation

• • Feedback loop exploitation

PRIMARY DEFENSES

• • Learning input validation and sanitization

• • Anomaly detection for learning updates

• • Learning rate limits and boundaries

KEY RISKS

Knowledge Update Mechanism Vulnerability#

Exploiting vulnerabilities in the agent's knowledge update mechanisms by injecting unauthorized updates, bypassing authentication and integrity checks, or manipulating version control systems to introduce malicious knowledge.

KEY FEATURES

• • Unauthorized update injection

• • Authentication bypass

• • Version control manipulation

PRIMARY DEFENSES

• • Strong update authentication

• • Cryptographic integrity checks

• • Version control with audit trails

KEY RISKS

Cross-Agent Knowledge Poisoning#

Attacking shared knowledge bases used by multiple agents to create systemic poisoning, where corrupted knowledge propagates across interconnected agents, causing cascading errors and compromised decision-making throughout the agent network.

KEY FEATURES

• • Shared knowledge base poisoning

• • Cross-agent propagation

• • Systemic knowledge corruption

PRIMARY DEFENSES

• • Shared knowledge validation

• • Cross-referencing between agents

• • Knowledge source verification

KEY RISKS

Ethical Guidelines for Memory & Context Attacks#

When working with memory & context attacks techniques, always follow these ethical guidelines:

• • Only test on systems you own or have explicit written permission to test

• • Focus on building better defenses, not conducting attacks

• • Follow responsible disclosure practices for any vulnerabilities found

• • Document and report findings to improve security for everyone

• • Consider the potential impact on users and society

• • Ensure compliance with all applicable laws and regulations

FROM THE ENGINEER BEHIND THIS CATALOG

Get your agent system red-teamed#

The attacks documented here work on production agent systems every day. Have yours tested before someone else does: prompt injection, jailbreaks, tool misuse and data exfiltration, with every finding written up next to its fix.

€750 instead of €1,500, one week, written report and walkthrough call, until 30 September

AI Red Teaming#

AI Red Teaming Techniques#