PROMPT_SECURITY_TOOL_CALL_HIJACK_001

rules/regulations/universal/prompt_security/tool_call_hijack.yaml

Citation

Regulation (EU) 2024/1689, Article 15(5): High-risk AI systems shall be resilient against attempts by unauthorised third parties to alter their use, outputs or performance by exploiting system vulnerabilities. The technical solutions aiming to ensure the cybersecurity of high-risk AI systems shall be appropriate to the relevant circumstances and the risks. The technical solutions to address AI specific vulnerabilities shall include, where appropriate, measures to prevent, detect, respond to, resolve and control for attacks trying to manipulate the training data set (data poisoning), or pre-trained components used in training (model poisoning), inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion), confidentiality attacks or model flaws.

Condition

Covers tool/function-call hijacking: injected instructions that direct an agent to invoke tools for the attacker (forward secrets, send data to an external address, run commands). EU AI Act Article 15. OWASP LLM01 / LLM07.

Layer 1 detects

in any letter case, forward, send, post, transmit, exfiltrate, leak, upload or email, then within 90 characters (no ".", "!", "?" or line break between) "system prompt", credential(s), "API key(s)", secret(s), conversation, "chat history", password or "the above", then within 90 more such characters "to" followed by an http or https URL or by an address containing "@"; make, issue, perform or execute "a" or "an" HTTP, HTTPS, GET, POST, curl or fetch "request to" an http or https URL; and call, invoke, use or execute (optionally "the") one word followed by tool, function or API, then "to send", "to forward", "to exfiltrate", "to leak", "to delete" or "to transfer", or "with my" or "with the user's" credential(s) or secret(s). It matches wording, not meaning: it does not read whose data goes where or at whose request, so it fires on the user's own request ("Email the chat history to ana@example.com", "use the email tool to send the invoice"), and it misses a hijack worded any other way. Whether the text directs the agent to act for an attacker is judged in Layer 2.

Condition type

regex

Status in the file

approved