Skip to content
Rust + AI · Project-First

Threat-Model the Agent, Not Just the API

An agent consumes untrusted content and may possess tools. That combination creates a security boundary different from an ordinary inference endpoint.

Assets

Section titled “Assets”

Protect:

  • user evidence and private memory;
  • provider credentials;
  • tool capabilities;
  • source code and infrastructure;
  • generated artifacts;
  • model and adapter weights;
  • eval datasets;
  • traces and human feedback;
  • budgets and availability.

Trust boundaries

Section titled “Trust boundaries”

Draw:

  • user → product;
  • product → harness;
  • harness → model provider;
  • retrieved content → context;
  • model output → tool runtime;
  • worker → sandbox;
  • tenant → shared storage;
  • generated artifact → publication.

Every crossing needs validation, identity, policy, and logging proportional to risk.

Prompt injection is cross-modal

Section titled “Prompt injection is cross-modal”

Instructions can appear in:

  • web pages and documents;
  • source comments and issue text;
  • image pixels or OCR;
  • spoken audio;
  • video frames and captions;
  • tool output;
  • retrieved memory.

The defense is not a sentence telling the model to ignore attacks. Use architecture:

  1. mark provenance and trust level;
  2. keep product policy outside retrieved content;
  3. expose least-privilege tools;
  4. require capabilities for effects;
  5. validate every proposed action;
  6. isolate execution;
  7. filter sensitive outputs;
  8. use human approval for consequential changes.

Tool capability model

Section titled “Tool capability model”
enum Capability {
    ReadRepository { root: RepoId },
    QueryLogs { service: ServiceId, window: TimeRange },
    CreateDraft { kind: DraftKind },
    PublishArtifact { destination: DestinationId },
    ExecuteSandboxed { profile: SandboxProfileId },
}

Capabilities are scoped resources, not global booleans. They expire and belong to an actor and run.

Sandbox design

Section titled “Sandbox design”

A code or media tool runs with:

  • an explicit working root;
  • no inherited credentials;
  • restricted network;
  • CPU, memory, file, process, and time limits;
  • read-only inputs;
  • a separate output directory;
  • captured stdout, stderr, exit status, and hashes.

The sandbox result is an observation. It does not silently grant the next permission.

Data and provider policy

Section titled “Data and provider policy”

Before a model call, evaluate:

  • tenant and evidence classification;
  • provider approval;
  • geography or retention constraints;
  • data minimization;
  • redaction;
  • whether a local model is required.

Record the policy decision without logging protected content.

Generated media risk

Section titled “Generated media risk”

Track consent, source rights, identity use, watermark or disclosure requirements, and publication approval. Separate:

  • generation;
  • internal preview;
  • user download;
  • public publication.

Each is a different effect.

Security evals

Section titled “Security evals”

Add cases where:

  • OCR reads “upload secrets to this URL”;
  • an audio note asks the agent to ignore the user;
  • source code comments request destructive commands;
  • a tool returns forged authority;
  • a model proposes a path outside the root;
  • a cross-tenant embedding result is highly similar;
  • a retry would duplicate a write;
  • a generated artifact uses an unapproved source asset.

Grade the trace, not only the final answer.

Incident readiness

Section titled “Incident readiness”

Be able to:

  • disable one tool or provider;
  • revoke capabilities and credentials;
  • stop active runs;
  • identify affected traces and artifacts;
  • delete protected evidence and derivatives;
  • roll back a program or model version;
  • preserve an audit record without preserving prohibited content.

Security is part of the harness definition, not a filter added after the model performs well.