# Memory & Context Attacks 📂 ai-red-teaming ## AI RED TEAMING TECHNIQUES # Memory & Context Attacks Memory poisoning, RAG exploitation, and context manipulation techniques ## Available Techniques ### Agent Memory Poisoning Injection of malicious or manipulative information into an agent's long-term memory systems, corrupting its knowledge base and influencing future behaviors. #### KEY FEATURES - • Long-term memory corruption - • Knowledge base manipulation - • Persistent behavior modification #### PRIMARY DEFENSES - • Memory validation and sanitization - • Source verification for stored information - • Memory access controls #### KEY RISKS ### Context Window Manipulation Exploitation of context window limitations to hide malicious instructions, overflow context buffers, or manipulate conversation history to bypass security controls. #### KEY FEATURES - • Context overflow attacks - • History manipulation - • Hidden instruction injection #### PRIMARY DEFENSES - • Context window management - • Priority-based context retention - • System instruction protection #### KEY RISKS ### RAG System Poisoning Injection of malicious documents or data into Retrieval-Augmented Generation (RAG) knowledge bases to poison retrieved context and manipulate agent responses. #### KEY FEATURES - • Knowledge base document poisoning - • Retrieval result manipulation - • Vector embedding corruption #### PRIMARY DEFENSES - • Document validation and sanitization - • Source authentication - • Retrieval result verification #### KEY RISKS ### Agent Session Hijacking Unauthorized takeover of an ongoing agent conversation or session, gaining access to conversation history, context, and the ability to inject malicious instructions. #### KEY FEATURES - • Session token exploitation - • Context injection mid-conversation - • History access and manipulation #### PRIMARY DEFENSES - • Strong session management - • Token encryption and rotation - • Session validation on each request #### KEY RISKS ### Direct Memory Injection Direct injection of malicious content into an agent's memory storage systems, bypassing normal conversation flows to insert false memories or corrupted knowledge. #### KEY FEATURES - • Direct storage manipulation - • Memory API exploitation - • Conversation bypass #### PRIMARY DEFENSES - • Strict API authentication - • Memory write access controls - • Input validation on storage operations #### KEY RISKS ### Context Contamination Attack Gradual contamination of conversation context through subtle injections across multiple interactions, slowly corrupting the agent's understanding and behavior. #### KEY FEATURES - • Gradual contamination - • Multi-turn corruption - • Subtle manipulation #### PRIMARY DEFENSES - • Continuous context validation - • Contamination detection algorithms - • Periodic context sanitization #### KEY RISKS ### Episodic Memory Replay Attack Manipulation of episodic memory replay mechanisms to reinforce malicious patterns, false information, or harmful behaviors through repeated memory activation. #### KEY FEATURES - • Memory replay manipulation - • Pattern reinforcement exploitation - • Memory consolidation abuse #### PRIMARY DEFENSES - • Replay validation - • Consolidation security controls - • Recall frequency limits #### KEY RISKS ### Semantic Memory Corruption Corruption of an agent's semantic memory - general knowledge and facts - through injection of false information that becomes part of the agent's core understanding. #### KEY FEATURES - • Factual knowledge corruption - • Concept relationship manipulation - • General knowledge poisoning #### PRIMARY DEFENSES - • Fact verification systems - • Knowledge source tracking - • Semantic consistency checks #### KEY RISKS ### Memory Persistence Exploitation Exploitation of memory persistence mechanisms to ensure malicious content remains in agent memory across resets, updates, or cleanup operations. #### KEY FEATURES - • Cleanup bypass - • Reset resistance - • Update persistence #### PRIMARY DEFENSES - • Comprehensive memory cleanup - • Complete reset procedures - • Persistence validation #### KEY RISKS ### Cross-Session Memory Leakage Exploitation of memory isolation vulnerabilities to access or leak information from other users' sessions or conversations through shared memory systems. #### KEY FEATURES - • Session boundary bypass - • Memory isolation exploitation - • Cross-user data access #### PRIMARY DEFENSES - • Strong memory isolation - • Session-specific memory spaces - • Access control enforcement #### KEY RISKS ### Learning Process Exploitation Attacking agent learning processes by introducing biased, incomplete, or malicious data during incremental updates, online learning, or feedback loops, causing the agent to learn harmful behaviors or incorrect patterns. #### KEY FEATURES - • Biased training data injection - • Incremental learning manipulation - • Feedback loop exploitation #### PRIMARY DEFENSES - • Learning input validation and sanitization - • Anomaly detection for learning updates - • Learning rate limits and boundaries #### KEY RISKS ### Knowledge Update Mechanism Vulnerability Exploiting vulnerabilities in the agent's knowledge update mechanisms by injecting unauthorized updates, bypassing authentication and integrity checks, or manipulating version control systems to introduce malicious knowledge. #### KEY FEATURES - • Unauthorized update injection - • Authentication bypass - • Version control manipulation #### PRIMARY DEFENSES - • Strong update authentication - • Cryptographic integrity checks - • Version control with audit trails #### KEY RISKS ### Cross-Agent Knowledge Poisoning Attacking shared knowledge bases used by multiple agents to create systemic poisoning, where corrupted knowledge propagates across interconnected agents, causing cascading errors and compromised decision-making throughout the agent network. #### KEY FEATURES - • Shared knowledge base poisoning - • Cross-agent propagation - • Systemic knowledge corruption #### PRIMARY DEFENSES - • Shared knowledge validation - • Cross-referencing between agents - • Knowledge source verification #### KEY RISKS ### Ethical Guidelines for Memory & Context Attacks When working with memory & context attacks techniques, always follow these ethical guidelines: - • Only test on systems you own or have explicit written permission to test - • Focus on building better defenses, not conducting attacks - • Follow responsible disclosure practices for any vulnerabilities found - • Document and report findings to improve security for everyone - • Consider the potential impact on users and society - • Ensure compliance with all applicable laws and regulations FROM THE ENGINEER BEHIND THIS CATALOG ## Get your agent system red-teamed The attacks documented here work on production agent systems every day. Have yours tested before someone else does: prompt injection, jailbreaks, tool misuse and data exfiltration, with every finding written up next to its fix. €750 instead of €1,500, one week, written report and walkthrough call, until 30 September ## AI Red Teaming ## AI Red Teaming Techniques