Sandbox / Dry-Run Mode

Run any policy in observe-only mode for a few days before flipping the switch to live enforcement.

Sandbox mode is the governance analogue of a database transaction ROLLBACK: the gateway evaluates every rule, records every would-be decision in the audit log, and applies none of them. The agent proceeds as if no policy were in effect. Once you’ve reviewed the would-be violations and tuned the policy, you cut over to live enforce mode with a one-line change.

The feature is part of the open-source core — not an enterprise add-on.

How it works

Sandbox mode is an enforcement posture, not a separate runtime. It only changes what the gateway does after a policy decision is computed:

Decision	Enforce mode (default)	Observe / dry-run mode
`Allow`	Action proceeds	Action proceeds (identical)
`Deny`	Action blocked; error returned	Action proceeds; `dry_run: true` shadow event written to the audit log
`Redact`	Payload sanitised	Unredacted payload forwarded; shadow event written
`RequiresApproval`	Action halts pending review	Action proceeds; shadow event written

Every shadow event carries the full decision context: which rule matched (shadow_decision), what the rejection reason would have been (shadow_reason), and a dry_run: true flag the audit consumer can filter on.

Quick start — 5 steps

# 1. Author a policy in observe mode (zero risk to running agents)
cat > coding-team-sandbox.yaml << 'EOF'
name: coding-team-sandbox
enforcement_mode: observe       # ← the one new field

rules:
  - action: deny
    match:
      tool_name: bash
      command_pattern: "rm -rf"
  - action: redact
    match:
      output_contains_pattern: "(AKIA|ghp_)[A-Za-z0-9]+"
EOF

# 2. Apply the policy
aasm policy apply --file coding-team-sandbox.yaml

# 3. Run an agent under observe-mode governance
aasm run --observe claude --workspace .

# 4. After a few days, review what would have been blocked
aasm audit list --dry-run-only --since 7d

# 5. Confident the policy is right? Flip to live enforcement.
sed -i 's/enforcement_mode: observe/enforcement_mode: enforce/' coding-team-sandbox.yaml
aasm policy apply --file coding-team-sandbox.yaml

Policy configuration

enforcement_mode is a top-level optional field on the policy document:

name: my-policy
enforcement_mode: observe       # "enforce" (default) | "observe" | "disabled"

rules: [ ... ]

When the field is omitted, the policy defaults to enforce — the pre-feature behaviour. Existing on-disk policies upgrade transparently.

Per-agent overrides via agent_overrides are also supported, so you can run a single experimental agent in observe mode while the rest of the team stays in live enforce:

name: coding-team-policy
enforcement_mode: enforce

agent_overrides:
  - agent_glob: "experimental-*"
    enforcement_mode: observe

Resolution order (highest priority first):

Per-agent override — agent_overrides block in the policy YAML, or enforcement_mode on the agent’s RegisterAgent RPC payload.
Policy document default — the top-level enforcement_mode field.
Server-wide default — enforce.

CLI reference

`aasm run --observe`

Launches a managed AI dev tool with observe-mode governance for the duration of the session.

# Boolean shorthand — most common case
aasm run --observe claude --workspace .

# Explicit form — interchangeable with the above
aasm run --enforcement-mode observe claude --workspace .

# Disabled mode — only valid in hermetic test environments
aasm run --enforcement-mode disabled codex --workspace .

# Combine with --dry-run to preview the launch without executing the tool
aasm run --observe --dry-run claude --workspace .

When observe mode is active, a one-time banner prints to stderr ahead of any tool output:

⚠️  [AAASM] Running in sandbox/observe mode.
    Policy decisions are recorded but NOT enforced.
    Review captured events: aa audit list --dry-run-only

The child process inherits AA_ENFORCEMENT_MODE=observe in its environment so tools that env-sniff (or downstream wrappers) can surface their own observe-mode badge.

--observe and --enforcement-mode are mutually exclusive — passing both fails fast at clap-parse time.

`aasm audit list --dry-run-only`

Filters the audit log to shadow events only:

# Show shadow events from the last 24h
aasm audit list --dry-run-only --since 24h

# Compose with other filters
aasm audit list --dry-run-only --since 7d --agent "codex-*"

# Machine-readable output for CI gates
aasm audit list --dry-run-only --format json

The flag is exclusive: by default aasm audit list HIDES shadow events so operators don’t see them mixed with live decisions; --dry-run-only flips that to show ONLY shadow events.

SDK usage

All three SDKs expose the same posture surface. Pass an enforcement_mode (Python / Go) or enforcementMode (Node.js) at agent registration:

Python

from agent_assembly import init_assembly

ctx = init_assembly(
    gateway_url="http://localhost:8080",
    api_key="...",
    agent_id="experimental-agent-001",
    enforcement_mode="observe",   # "enforce" | "observe" | "disabled"
)

The parameter is keyword-only; the type is Literal["enforce", "observe", "disabled"]. Omitting it preserves the pre-feature wire shape (the gateway applies its server-side enforce default).

Node.js / TypeScript

import { initAssembly, type EnforcementMode } from "@agent-assembly/sdk";

const ctx = await initAssembly({
  gatewayUrl: "http://localhost:8080",
  apiKey: "...",
  agentId: "experimental-agent-001",
  enforcementMode: "observe",   // 'enforce' | 'observe' | 'disabled'
});

The EnforcementMode union narrows at compile time; runtime validation catches typos from JS / JSON-config / dynamic-input callers with a RangeError.

Go

import "github.com/agent-assembly/go-sdk/assembly"

a, err := assembly.Init(ctx,
    assembly.WithGatewayURL("http://localhost:8080"),
    assembly.WithAPIKey("..."),
    assembly.WithSelfAgentID("experimental-agent-001"),
    assembly.WithEnforcementMode(assembly.EnforcementModeObserve),
)

assembly.EnforcementMode is a string-typed alias; the empty zero value omits the field from the registration body, preserving pre-feature wire shape.

CI integration — the policy-regression gate

A common observe-mode use case: gate every PR on “would my policy change block any existing agent workflow?”

# .github/workflows/policy-regression.yml
jobs:
  policy-regression:
    steps:
      - name: Run agent under observe-mode governance
        run: aasm run --observe codex -- codex "refactor src/auth.py"

      - name: Fail the PR on any would-be deny
        run: |
          BLOCKS=$(aasm audit list --dry-run-only --format json \
                   | jq '[.[] | select(.shadow_decision == "deny")] | length')
          if [ "$BLOCKS" -gt 0 ]; then
            echo "Policy regression: $BLOCKS actions would be blocked"
            aasm audit list --dry-run-only --format table
            exit 1
          fi

The exclusive-filter semantic of --dry-run-only means this gate doesn’t pick up unrelated live-enforcement events from other agents on the same gateway.

Dashboard

The dashboard exposes a SandboxSummaryCard component that renders the per-policy observe-mode aggregates:

┌─ SANDBOX SUMMARY ────────────────────────────────┐
│ coding-team-sandbox (last 24h)                    │
│                                                   │
│  47        12         3                           │
│  Would-be  Would-be   Would-be                    │
│  denies    redactions pending approvals           │
│                                                   │
│  Top matched rule: block-bash-rm-rf (31×)         │
│                                                   │
│  [View all events]  [Export CSV]  [Enable live →] │
└───────────────────────────────────────────────────┘

The amber colour is intentional — it visually contrasts with the dashboard’s red (live-deny) and green (live-allow) tokens so an operator can tell at a glance whether they’re looking at observe-mode aggregates or live enforcement data.

Status (2026-05): the card primitive is shipped (AAASM-1563). The full integration — wiring it into Policy detail, the audit-log toggle, the amber row badge, and the “Enable live enforcement” action — is tracked under AAASM-1911 and depends on aa-api surface changes that aren’t in this release.

Graduating to live enforcement

Once you’ve reviewed the shadow events and tuned the policy:

Inspect the most-common would-be violations:

aasm audit list --dry-run-only --since 7d --format json \
  | jq 'group_by(.shadow_decision) | map({decision: .[0].shadow_decision, count: length})'

Adjust the policy — tighten matchers that fired too eagerly, relax ones that blocked legitimate work.
Re-apply in observe mode for another short window to confirm the tuned policy behaves as expected.

Flip to enforce:

enforcement_mode: enforce

aasm policy apply --file my-policy.yaml

The cutover is instantaneous from the next CheckAction call onward — no agent restart required. Already-in-flight actions evaluated before the swap keep their original posture.

FAQ

Does observe mode affect performance? No measurable difference. The rule pipeline runs identically; the only added work is writing the shadow audit event when a non-Allow decision would have fired. That’s the same audit-write path live enforcement already uses, so the per-request cost is dominated by the rule evaluation itself.

Are redacted payloads ever stored in observe mode? No. The redact decision in observe mode forwards the unredacted payload to the agent (that’s the whole point — “what would have happened if we’d enforced”). The shadow audit event records that a redact rule matched, but neither the would-be redacted version nor the raw payload is persisted as a separate artefact. The audit pipeline’s existing PII-scanner pass still applies before any event is written.

Can I set observe mode per-agent without changing the policy? Yes — three ways:

CLI: aasm run --observe <tool> for the duration of that session.
SDK: pass enforcement_mode="observe" (Python / Go) or enforcementMode: "observe" (Node.js) at initAssembly.
Policy YAML: agent_overrides block targeting an agent_glob.

The per-agent override always wins over the policy document’s default.

What happens to an agent that’s mid-action when I flip from observe to enforce? The action that’s already through CheckAction keeps its observe-mode disposition (allowed). The very next CheckAction call sees the new posture and starts enforcing. There’s no in-flight rollback.

Does the SDK have any guard against accidentally registering in observe mode? The SDK doesn’t second-guess the operator — observe mode is a deliberate posture. What the SDK does is:

Reject typos (e.g. "obesrve") with a clear error at init time
Default to “no opinion” (omits the field from the registration body) so a pre-feature SDK call gets the gateway’s server-side enforce default — only operators who explicitly opt in get observe mode

Can I use observe mode in production for a long-running agent? That’s the recommended pattern for new policies — run them in observe mode for a week, review the shadow events, then cut over. The audit log retention follows your normal retention policy, so the shadow events are queryable for as long as live events.

Keyboard shortcuts

agent-assembly