Oracle PackX Round-Robin Productization Receipt

2026-07-16 active-worker preflight correction

Observed false failure

Correction and proof

2026-07-15 cross-process FIFO and bounded-preflight requalification

Oracle recommendation and implementation

Deterministic and live proof

Immutable release

Deployment recovery evidence

2026-07-15 four-slot concurrency and capacity-wait requalification

Behavior and authority

Implementation

Immutable release and deterministic proof

Live four-worker and wait proof

Final local skill digests

2026-07-15 Cloudflare false-positive requalification

Failure and root cause

Source and deterministic proof

Immutable green release

No-submit and live qualification

Demonstrated source case

Product decision

Invariants and parameters

Invariant Why stable Verification
Browser mode, Pro, Extended User/runtime requirement Runner-owned CLI flags
Automatic profile selection Required for persistent server round robin Runner never passes --remote-profile; durable metadata records assignment
Sticky identity after allocation Prevents cross-account replay Protocol-v2 journal/allocation tests and live reattach proof
Production untouched Critical safety boundary PID/start identity checked before and after each invocation
Exact green release/client match Prevents protocol drift Build SHA and local/remote file SHA-256 comparisons
Isolated sessions Prevents production session/config mixing ORACLE_HOME_DIR=~/.oracle-round-robin
Exact cleanup only Prevents collateral process termination Tunnel PID command verification before SIGTERM
Environment detail Parameter/default Secret or safety handling
SSH target pinned tv Batch-mode SSH; no sudo
Green endpoint pinned remote 127.0.0.1:19474 Temporary local SSH forward on 29474
Green token remote mode-0600 token file Read into process memory; never printed or persisted locally
Local matching client ~/dev/oracle-wt/dual-profile-isolation-scaffold Dist file hashes must match pinned release
Mutable run state ~/.oracle-round-robin mode-700 home and atomic run lock

Qualification case

Release boundary

Optional rollout handoff

Decision

# Oracle PackX Round-Robin Productization Receipt

## 2026-07-16 active-worker preflight correction

### Observed false failure

- observed failure: `run_round_robin.sh --check` repeatedly reported `one or
  more workers are unhealthy` while one qualified worker was serving an active
  Oracle consultation.
- authoritative `/health` evidence showed three idle profiles in `healthy` state
  and `profile-b-2` in the expected `busy` state with `activeRun: true`;
  capacity was `active=1`, `maximum=4`.
- the runner incorrectly required every profile state to equal `healthy` before
  it evaluated the already-present active-run and availability consistency
  checks. This contradicted the server's intentional `busy` state for an active
  worker and the skill's documented allowance for busy workers.
- all four Chrome root processes and CDP listeners were stable from the prior
  service start. The visible browser activity was renderer/tab activity from the
  active run, not root-Chrome crash/relaunch churn.

### Correction and proof

- the runner now requires exact state correspondence: an active worker must be
  `busy`; an inactive worker must be `healthy`. Truly unhealthy workers, a busy
  worker without an active run, and a healthy worker claiming an active run
  remain rejected.
- shell syntax and synthetic healthy/busy/unhealthy fixtures passed.
- the public `--check` passed while `profile-b-2` remained actively busy:
  `round_robin_check=ok`, pinned build unchanged, four slots, capacity waiting
  enabled, protocol v1 disabled, production listener PID `15286` unchanged.
- no green service restart, Chrome termination, prompt submission, production
  configuration change, or source-release change was required.
- current local digests:
  - `SKILL.md`: `bcaec7ed0f30d1cf1af5caed5e29e6497776ebcef3c366cc68d410c62a201d95`
  - `scripts/run_round_robin.sh`:
    `88b7ecaf4adcf4fb18b3bfa07556cff167f23a4c6bf5350ab2208ebad91805bb`

## 2026-07-15 cross-process FIFO and bounded-preflight requalification

### Oracle recommendation and implementation

- Oracle session: `round-robin-queue-final`.
- chosen boundary: the server remains authoritative for one active run per
  isolated worker, two per explicit account family, and four total; cooperating
  local CLIs use a filesystem FIFO only to decide which fresh request may probe
  allocation capacity next.
- added `src/remote/localAdmissionQueue.ts`: hashed stable namespace, atomic JSON
  state under the existing file lock, strict ordinal FIFO, mode-0700 queue root,
  exact process-birth identity, dead/PID-reused waiter recovery, and release
  immediately after allocation rather than after the full Oracle response.
- `src/remote/client.ts` wraps only fresh protocol-v2 allocation. Existing routed
  sessions bypass the queue and preserve sticky reattachment.
- the runner passes one stable service namespace and queue root beneath
  `~/.oracle-round-robin/admission`; dynamic tunnel ports, slugs, tokens, and
  build hashes are excluded from queue identity.
- the runner health preflight now has a 60-second overall deadline, five-second
  request bounds, bounded SSH establishment, and a safe last-failure diagnostic.
- the allocator has an explicit production policy mapping `profile-a` and
  `profile-a-2` to account family A, and `profile-b` and `profile-b-2` to family
  B, with two active leases per family and four total.

### Deterministic and live proof

- targeted queue/client/allocator checks passed after the final PID-reuse and
  mode-0700 changes: 27 tests.
- a real multi-process test launched independent Node processes against one queue
  root and proved the second process could not enter before the first released.
- full isolation suite: 172 tests passed.
- TypeScript typecheck, production build, targeted formatting, shell syntax, and
  local/remote artifact digest comparison passed.
- no-submit live qualification allocated all four distinct workers with exactly
  two from each account family; a fifth request returned only structured
  `409/no_capacity` with `retryAfterMs: 2000`; all four allocations were then
  abandoned and active capacity returned to zero.
- the public `--check` passed on the deployed build with capacity four, protocol
  v1 disabled, all workers healthy, and production listener PID `15286` with
  start identity `Tue Jul 14 12:27:33 2026` unchanged.
- no prompt-bearing qualification was submitted.

### Immutable release

- qualified source archive/build:
  `4a5f05a1d46857396cda46dab5913f7d57c303d8467167404a5b3e9c0d227c7a`.
- matching local/remote artifacts:
  - CLI: `4e3ae1d769b14872e6be114220c21ef9eef65e1b6beb428a8b9385fbb8fa8064`
  - protocol-v2 client: `7c764509723f49e7cb959b2f45379af96447fe6ad95d020fdaf5309c43bc0916`
  - browser navigation: `e251bd0ce316653fbae6c99bdff20231a889fc9756ddf8051c3063482d2c7127`
  - local admission queue: `e0c0fa70c44d761aa2b3ed31170129b3677156f80e7445584b06c4a2a49d871e`
- final skill digests:
  - `SKILL.md`: `bcaec7ed0f30d1cf1af5caed5e29e6497776ebcef3c366cc68d410c62a201d95`
  - `scripts/run_round_robin.sh`:
    `67cd2bae1bc8b9f2a15cc245ac0fae57424be8a7b63385778014a95104decfd5`

### Deployment recovery evidence

- staging restart waited for an unrelated active Oracle run to become terminal;
  no active allocation was interrupted.
- two initial `launchctl bootstrap` attempts hit macOS error 5 while bootout and
  bootstrap occurred in one SSH command. Rollback configuration was restored and
  production remained unchanged. The cause was launchd deregistration timing,
  compounded on the first attempt by a duplicate-label rollback plist under
  `~/Library/LaunchAgents`.
- the corrected deployment keeps candidates and rollback plists under the green
  service home, leaves only the canonical plist in `~/Library/LaunchAgents`, uses
  mode `0644`, waits after bootout, and retries exact bootstrap without broad
  process termination. The new green release then started healthy.

## 2026-07-15 four-slot concurrency and capacity-wait requalification

### Behavior and authority

- requested behavior: two concurrent Oracle workers per account, four total;
  callers beyond capacity wait instead of returning a capacity blocker.
- safe design: four isolated Chrome workers, not two automations sharing one
  Chrome data directory.
- canonical runtime authority: protocol-v2 allocator/client plus the managed
  worker pool in `~/dev/oracle-wt/dual-profile-isolation-scaffold`.
- dependent surfaces synchronized: green deployment config/release, runner,
  `SKILL.md`, managed-staging documentation, tests, and this receipt.

### Implementation

- profile-pool configuration and allocation now accept any non-empty worker list
  while retaining exactly one active lease per isolated worker.
- green topology is `profile-a:19222`, `profile-b:19223`,
  `profile-a-2:19224`, and `profile-b-2:19225`.
- capacity exhaustion is the structured response
  `409 {error:"no_capacity", retryAfterMs:2000}`. The client retries only that
  response, indefinitely by default, with the identical request and affinity
  identity. Upload and submission cannot begin before allocation.
- runner serialization changed from one global lock and fixed port `29474` to
  per-slug locks plus invocation-private dynamically allocated SSH forwards.
  Normal invocations use a cheap immutable topology/build check so four callers
  do not serialize on expensive readiness probes. `--check` retains the full
  four-worker readiness check.
- concurrent model/thinking-time hydration was slower than sequential use and
  initially caused `Thinking chip` failures. Thinking-time selection now retries
  for up to ten one-second intervals. Four-way live concurrency passed afterward.
- the two new empty profiles were logged out. Per explicit user direction, the
  service was stopped and the already-qualified isolated staging states were
  cold-cloned `profile-a -> profile-a-2` and `profile-b -> profile-b-2`, excluding
  manager identity markers, singleton files, and DevTools state. No production
  Chrome state or production cookies were read or copied.

### Immutable release and deterministic proof

- qualified source archive/build:
  `8bff63d9681ab90fa4997467e1fddc80dd22de097676c469792e12ffa33ef5ff`.
- matching local/remote artifacts:
  - CLI: `4b74dbc5b1d2f6166c52f15db4c7d5283179358f42d42e69eca87d3cb25e1ddd`
  - protocol-v2 client: `b31e59079a0b2c3e5a10b64e6b98278853c75e9dee23329929318e67e2e53f97`
  - browser navigation: `e251bd0ce316653fbae6c99bdff20231a889fc9756ddf8051c3063482d2c7127`
- targeted implementation proof: 49 profile/config/client tests passed; the later
  browser/concurrency retry slice passed 70 tests with 1 intentional skip;
  TypeScript typecheck and vendor build passed.
- runner boundary proof: four concurrent synthetic dry-runs used four distinct
  local loopback forwards and removed every per-slug lock.
- live allocator proof: four held leases occupied four different workers; a
  fifth allocation returned only structured `no_capacity`; release made capacity
  reusable.

### Live four-worker and wait proof

- sequential live markers passed once on every worker:
  - `profile-b`: `SEQUENTIAL-ONE-QUALIFIED-20260715`
  - `profile-a-2`: `SEQUENTIAL-TWO-QUALIFIED-20260715`
  - `profile-b-2`: `SEQUENTIAL-THREE-QUALIFIED-20260715`
  - `profile-a`: `SEQUENTIAL-FOUR-QUALIFIED-20260715`
- final four-way concurrent live markers all passed in one wave:
  `RETRY-FINAL-ONE/ TWO/ THREE/ FOUR-QUALIFIED-20260715`, routed across all
  four worker IDs with durable protocol-v2 receipts.
- final five-caller wave passed all five markers. Allocation IDs `00000049`
  through `00000052` occupied the four slots. Allocation `00000053` reused
  `profile-b-2` only after its prior `00000051` run became terminal, proving the
  fifth caller waited rather than failing or sharing an active worker.
- final health: `capacity.active=0`, `capacity.maximum=4`; all workers healthy
  and available.
- production listener PID `15286` and start identity remained unchanged.

### Final local skill digests

- `SKILL.md`: `6ce51dd614a3694187ab3777b97970ef38d8157bb3bc1ab4883f1dfab858641d`
- `scripts/run_round_robin.sh`:
  `98e3df8edfe30a3e44f248f907045e7fe8ee4726646b3711ceb5eb0dee9036d1`

## 2026-07-15 Cloudflare false-positive requalification

### Failure and root cause

- observed failure: protocol-v2 runs on both managed profiles reported a Cloudflare
  challenge immediately after navigation, while ordinary `oracle-packx` remained healthy.
- transport comparison: ordinary `oracle-packx` reaches `tv:7333` directly over
  Tailscale and attaches to the long-lived production Chrome; round-robin tunnels
  control traffic over SSH to `tv:19474` and attaches to service-owned Chrome on
  `19222`/`19223`. Both browsers access ChatGPT from `tv`, so the tunnel was not
  visible to Cloudflare and was not the cause.
- real-DOM evidence: both managed profiles showed title `ChatGPT`, a recognized
  ChatGPT app shell, and `script[src*="/challenge-platform/"]`. The old detector
  treated that normal bot-management script as an interstitial by itself.
- correction: app-shell evidence now overrides the script; real title/widget or
  short verification-page evidence remains strong; script-only weak evidence must
  persist through a 12-second hydration grace before classification.

### Source and deterministic proof

- source: `src/browser/actions/navigation.ts` in
  `~/dev/oracle-wt/dual-profile-isolation-scaffold`.
- tests: `tests/browser/pageActions.test.ts` covers app-shell + script, content-rich
  pages, true title/widget/short-page challenges, and weak-evidence hydration.
- targeted result: 49 passed, 1 skipped across `pageActions.test.ts` and
  `navigation.test.ts`.
- repository result: isolation and lint gates passed; full suite 793 passed,
  4 skipped; TypeScript/vendor build passed; `git diff --check` passed.

### Immutable green release

- superseded rollback build:
  `e56d7cccf7fdedc75189e8cc0f7ae3101684e02242254e8e85029f9bbc900e3b`.
- qualified source archive/build:
  `2850837d0731411a8ebde3f9a2d65b27b107bcbbd97129477b647394e5cf508d`.
- matching local/remote artifacts:
  - CLI: `4b74dbc5b1d2f6166c52f15db4c7d5283179358f42d42e69eca87d3cb25e1ddd`
  - protocol-v2 client: `1ae5d15d0959dcac966cde24807bb5f173fac66a9da27a4ce782c262ce79467a`
  - browser navigation: `e251bd0ce316653fbae6c99bdff20231a889fc9756ddf8051c3063482d2c7127`
- green config/plist were atomically repointed; only the exact green LaunchAgent
  was booted out/bootstrap-loaded. Profile-instance identities persisted.

### No-submit and live qualification

- both live profiles returned title `ChatGPT`, app-shell `true`,
  challenge-script `true`, and patched classification `not-blocked`.
- automatic no-submit allocation covered profile-a then profile-b; replay of the
  first request preserved run/profile identity; both allocations were abandoned;
  capacity returned to zero.
- protocol-v1 GET and POST remained `404 unsupported_protocol`.
- profile-a attachment response:
  `PROFILE-A-CLOUDFLARE-FIX-QUALIFIED-20260715`.
- profile-b attachment response:
  `PROFILE-B-CLOUDFLARE-FIX-QUALIFIED-20260715`.
- terminal sticky reattachment returned the identical marker on each original
  profile without resubmission.
- final health: both profiles healthy and available, zero active runs. The runner's
  authenticated health timeout is 30 seconds so active no-submit readiness probes
  cannot spuriously fail the five-second transport budget.
- isolation: no green listener/Chrome process opened protected production paths;
  production listener PID `15286` and start identity
  `Tue Jul 14 12:27:33 2026` remained unchanged.

## Demonstrated source case

- case: isolated dual-ChatGPT-account Oracle green deployment on `tv`.
- successful output: two healthy persistent workers, automatic A/B allocation,
  sticky protocol-v2 routing, attachment handoff, restart persistence, and one
  successful Pro Extended attachment response plus terminal reattach per account.
- proof/receipt: green `/health`; allocation order `profile-a`, `profile-b`;
  stable profile-instance IDs across restart; qualification answers
  `PROFILE-A-QUALIFIED-20260715` and `PROFILE-B-QUALIFIED-20260715` with identical
  reattach results; production PID `15286` and start identity unchanged.
- failures and corrections mined: macOS `ps` command parsing lost executable paths
  containing spaces; CDP reports `Browser` with uppercase B; concurrent Chrome
  startup raced; login controls can flash before cookie restoration; profile B's
  first login did not persist; protocol-v1 routes were initially still readable.

## Product decision

- chosen form: layered skill plus deterministic runner.
- target runtime and install location:
  `~/.agents/skills/oracle-packx-round-robin`.
- existing owner or new surface: new specialized transport surface composing the
  existing `oracle-packx` prompt/bundling owner.
- public boundary: `scripts/run_round_robin.sh --check` and one prompt-file/bundle
  invocation.
- why expected reuse justifies the product: avoids repeatedly reconstructing SSH
  tunneling, secret handling, release matching, round-robin semantics, isolated
  session roots, and production guards for every expensive Oracle consult.
- existing pattern reused: `oracle-packx` bundle/prompt/output contract and the
  existing `~/.agents/skills/*/SKILL.md` format.

## Invariants and parameters

| Invariant | Why stable | Verification |
| --- | --- | --- |
| Browser mode, Pro, Extended | User/runtime requirement | Runner-owned CLI flags |
| Automatic profile selection | Required for persistent server round robin | Runner never passes `--remote-profile`; durable metadata records assignment |
| Sticky identity after allocation | Prevents cross-account replay | Protocol-v2 journal/allocation tests and live reattach proof |
| Production untouched | Critical safety boundary | PID/start identity checked before and after each invocation |
| Exact green release/client match | Prevents protocol drift | Build SHA and local/remote file SHA-256 comparisons |
| Isolated sessions | Prevents production session/config mixing | `ORACLE_HOME_DIR=~/.oracle-round-robin` |
| Exact cleanup only | Prevents collateral process termination | Tunnel PID command verification before SIGTERM |

| Environment detail | Parameter/default | Secret or safety handling |
| --- | --- | --- |
| SSH target | pinned `tv` | Batch-mode SSH; no sudo |
| Green endpoint | pinned remote `127.0.0.1:19474` | Temporary local SSH forward on `29474` |
| Green token | remote mode-0600 token file | Read into process memory; never printed or persisted locally |
| Local matching client | `~/dev/oracle-wt/dual-profile-isolation-scaffold` | Dist file hashes must match pinned release |
| Mutable run state | `~/.oracle-round-robin` | mode-700 home and atomic run lock |

## Qualification case

- case: clean public-boundary preflight plus synthetic dry-run and occupied-port
  failure fixture.
- material differences from source case: tiny synthetic JavaScript-shaped bundle
  instead of the live qualification attachment; dry-run rather than paid browser
  submission; disposable `/tmp` files; forced local tunnel collision failure.
- isolation or sandbox: disposable `mktemp` directory; isolated client home; exact
  synthetic Python listener PID cleaned after the failure test.
- fresh HOME/profile/container or scrubbed-equivalent evidence: no browser prompt
  submitted; no new Chrome profile; synthetic inputs removed; production config
  was never loaded because the runner injects an isolated Oracle home and remote
  environment.
- relevant PATH, binaries, config, credentials, caches, and prior state excluded:
  production `~/.oracle/config.json`, production sessions, production cookies,
  and production Chrome data were excluded; matching pinned CLI/client binaries
  were intentionally retained as the qualified runtime.
- qualification state: passed.
- approval for any live or paid target: source-case live qualification was already
  explicitly scoped to at most two prompt submissions; product boundary tests
  submitted no additional prompts.
- cleanup plan and result: exact SSH tunnel PID stopped; synthetic fixture files,
  listener PID, and lock removed; cleanup passed.
- tested artifact digest:
  - `SKILL.md`: `457e2c8e5889c97b4d964d333d1f4831bd6ba0fe964a1af2f1dc70f129537df9`
  - `scripts/run_round_robin.sh`: `24e7e18f1d9450059bac584f052505de44500506b9583b74bb655d0878e2a65f`
- command/invocation: `run_round_robin.sh --check`; synthetic `--dry-run` with
  prompt and bundle; `--check` while an exact synthetic listener occupied port
  `29474`.
- success proof: preflight printed pinned build, two profiles, protocol v1
  disabled, and production PID; dry-run selected browser `gpt-5.6-pro`, detected
  remote host `127.0.0.1:29474`, and previewed one attachment without submission.
- failure-path proof: exited `1` with `local tunnel port 29474 is already occupied`.
- revisions required: initial remote SHA command incorrectly expanded shell `$1`;
  replaced with remote `cut`; added v1-disabled and post-check production guards;
  all boundary checks rerun afterward.

## Release boundary

- final tested artifact digest: the two SHA-256 values above.
- install/init: files live directly under the requested global `~/.agents/skills`
  location; no production Oracle configuration changes.
- help/schema/API/skill surface: YAML frontmatter, `SKILL.md`, runner `--help`,
  `--check`, live invocation, optional repeated `--file`, and `--dry-run`.
- runtime state location: `~/.oracle-round-robin`; remote mutable state remains
  beneath `~/.oracle-staging/dual-profile-green` on `tv`.
- publication URL or package: none; local skill only.
- independent boundary check: shell syntax, help, authenticated topology check,
  synthetic dry-run, occupied-port failure, and secret-string scan.
- proof rerun after final behavior change: required and completed.

## Optional rollout handoff

- rollout requested: no production cutover requested.
- explicitly approved targets and actions: isolated green staging only.
- credentials, branch, and public-boundary constraints: token remains remote and
  ephemeral locally; worktree remains uncommitted; no publication.
- cleanup and rollback: stop exact runner tunnel; green LaunchAgent can be stopped
  only by its exact label if separately requested.
- orchestration proof gates: not applicable.
- unapproved targets intentionally excluded: production Oracle routing/config,
  production Chrome state, and any commit.

## Decision

- reusable product because: the successful workflow has stable safety/routing
  invariants and the runner removes repeated high-risk manual setup.
- intentionally not generalized: host, label, ports, build SHA, release, worktree,
  and profile IDs remain pinned; arbitrary targets would weaken fail-closed safety.
- owner and maintenance trigger: update this skill, runner, and receipt together
  only after a new immutable green release passes both-account qualification.