In-Flight Ops Registry — Architecture

Status: Active design — PR-A landed (AAASM-1422). Scope: Gateway-side tracking of agent operations from CheckActionRequest ingestion through to terminal Completing/Terminated states, the IPC protocol that lets the dashboard observe and control those operations, and the SDK return-channel that propagates control signals back to running agents.

1 — Why this exists

The original audit pipeline records what already happened (AuditEvent is post-facto and immutable). The Live Ops dashboard (AAASM-1326, AAASM-1334) needs a live view of operations currently in flight: which agents are running right now, which are paused, which were just terminated. None of that existed before AAASM-1525 / AAASM-1422.

AAASM-1415 shipped the POST /api/v1/ops/{id}/{pause,resume,terminate} route shells as stubs that return 202 + log so the dashboard’s row-action menu could be wired without 404-ing. AAASM-1525 added the OpsRegistry skeleton in aa-api with a 3-state machine (Running / Paused / Terminated) and a client-driven POST /api/v1/ops registration endpoint. AAASM-1422 closes the remaining gap: gateway-side ingestion from the policy-check path, a 5-state model that distinguishes pre-allow from post-completion, and a sub-task plan for the IPC protocol and SDK enforcement.

2 — Decisions recorded for this iteration

3 — Data model

#![allow(unused)]
fn main() {
// aa-gateway/src/ops/mod.rs

pub enum OpState {
    Pending,     // ingested, awaiting policy decision
    Running,     // policy allowed; agent is actively executing
    Paused,      // operator paused via POST /api/v1/ops/{id}/pause
    Completing,  // action signalled complete, draining
    Terminated,  // operator terminated, or policy denied
}

pub struct OpRecord {
    pub op_id: String,        // "{trace_id}:{span_id}"
    pub state: OpState,
    pub registered_at: String,// RFC 3339 — first time the op id was seen
    pub updated_at: String,   // RFC 3339 — most recent transition
}

pub enum OpsError {
    NotFound,
    InvalidTransition,
}

pub struct OpsRegistry { /* DashMap<String, OpRecord> */ }
}

4 — State machine

stateDiagram-v2
    [*] --> Pending: ingest()
    Pending --> Running: allow()
    Pending --> Terminated: deny() / terminate()
    Running --> Paused: pause()
    Paused --> Running: resume()
    Running --> Completing: complete()
    Running --> Terminated: terminate()
    Paused --> Terminated: terminate()
    Completing --> [*]: (sweep — PR-H)
    Terminated --> [*]: (sweep — PR-H)

Transition rules:

The registry remains idempotent on terminal states: calling terminate on an already-Terminated op returns the existing record without erroring.

5 — Ingestion path

agent ──gRPC──▶ PolicyServiceImpl::check_action(req)
                  │
                  ├─▶ ops_registry.ingest("{trace_id}:{span_id}")
                  │     // entry created in `Pending`
                  │
                  ├─▶ engine.evaluate(req)  ─▶  EvaluationResult
                  │
                  ├─▶ if Allow:
                  │      ops_registry.allow(op_id)   // Pending → Running
                  │   if Deny:
                  │      ops_registry.terminate(op_id) // Pending → Terminated  (PR-H)
                  │
                  └─▶ Response { decision, reason, ... }

This means: by the time the SDK receives the CheckActionResponse, the gateway-side registry has the op recorded and the dashboard sees it in the correct state via the WebSocket stream (PR-B).

PR-A ships the ingest() + allow() call sites. The terminate() on Deny is deferred to PR-H so PR-A keeps a small surface area.

6 — IPC sketch (PR-D)

Today the gateway → SDK channel is request/response only (CheckActionRequest → CheckActionResponse). For real pause / terminate enforcement, the SDK must learn about state changes while the action is in flight.

Two viable shapes:

1. Server-streaming OpControlStream — SDK opens a long-lived stream on register_agent. Gateway pushes {op_id, signal: pause|resume|terminate} messages. SDK acknowledges via a separate unary RPC. (Recommended in PR-D.)
2. Bidirectional OperationChannel — replace per-action CheckAction with a single bidi stream. Heavier protocol churn; deferred.

The SDK then cooperatively yields on pause, resumes on resume, and fast-fails on terminate. Each SDK (Python / Node / Go) ships its own enforcement layer in PR-E / PR-F / PR-G.

7 — Dashboard correlation (PR-C)

Today the dashboard’s useLiveOpsStream hook builds an in-memory map keyed by GovernanceEvent.id (monotonic, unique per event). Two events for the same op therefore can’t be correlated — the override-clear logic in LiveOpsPage never sees its target id again.

After PR-B/PR-C, the WebSocket emits a new OpStateChanged payload variant:

{
  "event_type": "ops_change",
  "agent_id": "agent-7",
  "payload": {
    "op_id": "trace-abc:span-1",   // stable across the op's lifetime
    "state": "running",            // OpState serialized snake_case
    "updated_at": "2026-05-20T09:32:20.822Z"
  }
}

The dashboard then keys its map by payload.op_id. The override-clear logic matches on the same key, so a pause followed by the server’s confirming paused event auto-clears the optimistic state without manual intervention.

8 — Sub-task plan

9 — Out of scope for this Task (AAASM-1422)

• Persistence across gateway restarts (registry is in-memory; restart re-empties it and the dashboard reconciles via the existing WS reconnect).
• Multi-gateway cluster coordination (sharded by agent_id-affinity in a later release; not on the roadmap for v0.0.1).
• Cross-team aggregation views beyond what the existing Live Ops page surfaces.

10 — References

• AAASM-1422 — this Task
• AAASM-1415 — stub /ops/{id}/{pause,resume,terminate} endpoints
• AAASM-1525 — OpsRegistry skeleton with 3-state machine
• AAASM-1326 / AAASM-1334 — Live Ops dashboard design + row actions

Last updated: 2026-05-21 by Chisanan232