security: contain physical miner commands in Stage A

Disable physical control through legacy HTTP/M30 producers, typed admission,
queue dispatch and upgraded Windows executors. Restrict diagnostic transports
to exact read-only operations, with ordinal PowerShell validation before sockets.

Retire unsupported pending/claimed work with atomic, idempotent security audits.
Preserve prior delivery evidence and report unknown outcomes after a prior claim,
including explicitly repaired pending rows. Include containment, cancellation,
concurrency and malformed-payload regressions and the old-Agent cutover runbook.

Validation: independent final application-security revalidation PASS. Two unchanged
baseline source-assertion failures are documented as exceptions. Production release
still requires native Windows validation and verified fencing/stop/drain/upgrade;
server cancellation alone cannot revoke a cached command held by an old Agent.

docs/operations/STAGE_A_OLD_AGENT_CUTOVER.txt

@@ -0,0 +1,248 @@
+MinerOps Blocker #3 Stage A: old-Agent containment and upgrade runbook
+Status: review candidate. This procedure has NOT been executed in production.
+Scope: contain physical control; no Stages B-E protocol or control re-enablement.
+
+The independent audit demonstrated that an old Agent can execute a previously
+delivered command after the Hub marks it cancelled. Cancelling a database row,
+revoking a Hub credential, advertising control unavailable, or replacing script
+files does not revoke a cached command or loaded PowerShell functions. Execution
+of any previously claimed command remains UNKNOWN until separately investigated.
+
+Production approval requires independent security revalidation, an approved change
+window, verified network enforcement, and the native Windows checks below. A new
+database migration or production network infrastructure change requires explicit
+approval. This runbook authorizes no production actions by itself.
+
+Ownership and evidence
+
+Assign an incident/change owner, a network operator, a Hub/database operator, and a
+Windows operator. Record their identities, UTC timestamps, change ticket, and every
+gate's evidence in a protected location. Keep enrollment tokens and Agent/miner
+credentials out of ordinary logs and this checklist. Preserve required backups
+and recovery state with their existing ACLs; do not overwrite agent-config.json.
+
+Maintain one ledger containing ALL registered Agents, every distinct queue Agent
+(including deleted registry entries), offline/unreachable Agents, alternative
+installations and manually running executors, legacy shared identities, all Hub
+replicas/workers, scheduled jobs, task hosts, physical networks, and IPv4/IPv6 paths.
+For each Agent record its Site/Zone assignment, last check-in, host/network identity,
+network fence evidence, process/task evidence, installed-file hashes, verified
+artifact identifier, and release decision. Keep unverified entries quarantined.
+
+Gate 1: restrict physical write access BEFORE stopping/upgrading anything
+
+1. Arrange an approved maintenance window and identify the actual network control
+   that can keep old Agents from reaching ALL physical miners. Layer-4 rules cannot
+   distinguish a legacy read from a write on the same miner API port. Default to
+   denying Agent-to-miner traffic, including ports 4028 and 4433, alternative miner
+   management interfaces/ports, IPv6, alternate NICs/VPNs, and established sessions.
+   Telemetry may be interrupted. Do not substitute an unreviewed 'read-only' proxy.
+2. Apply only separately approved network controls through the responsible operator.
+   A stopped process or a host-local rule alone is insufficient for an unreachable
+   machine that can later reboot/reconnect. Use enforcement that already applies
+   to that machine's traffic when it returns. Include unknown/retired installations
+   and aliases; trusting an IP address alone is insufficient when addresses can change.
+3. Verify deny counters and session termination/conntrack handling. Use approved
+   synthetic targets/canaries, never a real write command to a physical miner.
+   A rule that blocks only NEW connections leaves a pre-existing API session exposed.
+   Check both the intended route and alternate routes. Keep the fence if evidence
+   is unavailable, ambiguous, or an offline Agent cannot be located.
+
+Exit gate: verified deny enforcement covers every unverified/legacy executor,
+including future reconnects and existing sessions. Failure means STOP; keep fences.
+
+Gate 2: stop old executors and automatic restarts
+
+1. Stop command production and processing on every old Hub/worker version, including
+   old schedule materializers and running requests that could still insert/deliver
+   commands. Hold ingress and scale/restart mechanisms as necessary under the
+   approved change procedure. Inventory the actual versions of every instance.
+2. On each reachable Windows host disable/hold all relevant Scheduled Task triggers,
+   retry/restart policies, service recovery, watchdogs, startup/logon entries, GUI
+   launch actions, manual jobs, and external orchestration that can launch MinerOps.
+   Record original definitions and intended running/stopped state for recovery.
+3. Stop MinerOpsAgent and any alternative/manual executor, then verify task instances
+   AND their PowerShell/child process trees have exited. Record command paths,
+   process start times and identities. An in-memory old module is still executable
+   even after its source file changes. Do not gracefully 'drain' by executing writes.
+4. Keep pool/enrichment tasks held during runtime replacement and verify no alternate
+   old code copy can auto-start. Never run uninstall or erase config/state to stop it.
+
+Exit gate: every reachable old executor is stopped and restart-held; every
+unreachable/unverified Agent remains network-fenced. Do not remove a fence to
+diagnose a failed stop. Escalate failures with evidence and keep maintenance holds.
+
+Gate 3: update ALL Hub instances with no mixed-version execution
+
+1. Capture protected queue/audit evidence and the relevant schedule/Agent inventory
+   before changes. Retain original request/result payloads, timestamps and errors.
+   Secure these exports because historical payloads may contain sensitive data.
+2. Install only the independently reviewed Stage A artifact on every Hub instance
+   while executor fences/holds remain. Stop old replicas and workers fully, including
+   rolling-update stragglers and old connections/requests; do not allow a mixture of
+   active baseline and Stage A schedulers/queue consumers.
+3. Verify each instance's actual artifact hashes and that physical admission is
+   fixed disabled: legacy write/diagnostic POSTs return 410 after authorization,
+   canonical typed controls return 409 physical_control_disabled_stage_a, and queue
+   SQL dispatches only the fixed read types. Do not use runtime flags to enable writes.
+4. Run reviewed schema compatibility checks as appropriate. Stage A remediation adds
+   NO migration. Stop and request approval if the deployed schema needs one.
+
+Exit gate: only reviewed containment Hub code is active; all old producers are
+stopped. Keep Agents held. If any instance cannot be verified, keep ingress/worker
+holds and physical fences rather than continuing a partial rolling deployment.
+
+Gate 4: audited cancellation across ALL Agents, including offline/queue-only IDs
+
+1. Use the canonical reviewed code and a real accountable operator users.id for
+   manual cutover retirement. Existing authenticated Agent polling/result rejection
+   is system-attributed by source and Agent ID; manual cutover supplies actor_user_id.
+   Do not impersonate another user, create invented execution evidence, or log secrets.
+2. Inventory IDs using the UNION of registry and queue Agent IDs. Also reconcile the
+   fleet ledger, historical/retired identities, and old shared Agent ID. Do not use
+   only Agents currently checking in. A useful read-only inventory is:
+
+     SELECT agent_id FROM minerops_site_agents
+     UNION SELECT agent_id FROM minerops_agent_command_queue
+     ORDER BY agent_id;
+
+3. In a separately approved administrative process configured for the intended DB,
+   call the reviewed repository operation for each ID, in committed bounded batches:
+
+     repository = AgentCommandRepository()
+     count = repository.retire_unsupported(agent_id, actor_user_id=operator_user_id)
+
+   Repeat for all IDs while progress is made, under an explicit time/batch limit.
+   This method never claims read work; do NOT substitute claim_pending or an Agent
+   HTTP poll as an administrative drain. Each transition and rejection audit commits
+   atomically. Database/audit failure rolls back and keeps the change window closed.
+4. A batch is bounded at 1000 and uses SKIP LOCKED. A zero count may mean locked rows,
+   not completion. Resolve/retry approved lock holders with time limits; never
+   silently declare the drain finished. Query ALL Agents to verify a zero live count:
+
+     SELECT agent_id, command_type, command_status, count(*)
+     FROM minerops_agent_command_queue
+     WHERE command_status IN ('pending', 'claimed')
+       AND command_type NOT IN
+           ('poll_known_miners_now', 'discover_new_miners', 'enrich_match_discovery')
+     GROUP BY agent_id, command_type, command_status;
+
+5. Reconcile newly cancelled rows to agent.command.rejected records. Check command ID,
+   Agent ID, prior state, immutable rejection reason, source and operator attribution.
+   Current pending status does not establish that a command was never delivered.
+   Current claimed status, any retained claimed_at, completion/result/error evidence,
+   linked legacy result or non-creation audit history, an unlinked legacy result
+   for that Agent, or an Agent result submission
+   requires execution_outcome=unknown. Missing or ambiguous evidence is unknown.
+   Only a pending row with a complete pre-cancellation check finding no delivery
+   evidence is classified not_dispatched by the queue. Historical claim evidence
+   survives cancellation; an explicitly repaired/requeued pending row is still
+   unknown. No ordinary production requeue mechanism is introduced by Stage A.
+   Neither classification proves absence of out-of-band historical physical writes.
+   Repeated retirement must add no duplicate audit and must not rewrite history.
+6. Schedule cancel/delete uses the stored schedule's Agent ID and the same rejection
+   audit policy; delete retains the schedule. This is NOT a fleet drain: orphaned or
+   foreign-Agent rows are handled by the per-Agent cutover sweep above. Never delete
+   queue/audit rows, reset claimed commands to pending, or rewrite terminal results.
+   Existing pre-remediation cancelled rows missing an audit must be preserved and
+   documented as a historical evidence gap. Do not invent a prior-state audit.
+
+Exit gate: the global live-unsupported query returns no rows, locks are accounted
+for, transition counts/audits reconcile, and no old producer can replenish work.
+This gate does not revoke already delivered payloads; gates 1, 2 and 5 remain mandatory.
+
+Gate 5: upgrade Windows code and verify installed bytes while stopped
+
+1. Use a verified artifact prepared from the exact independently reviewed Stage A
+   boundary. Record its digest and a trusted expected SHA-256 manifest outside the
+   target installation. VERSION is unchanged at 1.0.0-alpha and is not upgrade proof.
+2. Include all managed runtime scripts, especially MinerOpsAgentService.ps1,
+   MinerOpsControl.ps1, EnrichOneMiner.ps1, MinerOpsApi3EnrichmentPoller.ps1, the pool
+   pollers, task/GUI/tray scripts, and VERSION. Check the package's installer and
+   enrollment/credential-upgrade entry points as well. Include the manifest's origin
+   and review approval; hashes computed only from a suspect host are not a trust root.
+3. Confirm the main task was positively stopped BEFORE invoking the existing
+   installer. It can auto-restart a task that it found running. Keep the external
+   network fence and restart hold throughout installation and failure recovery.
+   Do not pass filesystem trust-reset switches unless their existing reviewed
+   recovery workflow specifically calls for them. Preserve config/state/logs/backups.
+4. Compare installed Get-FileHash -Algorithm SHA256 values to the trusted manifest.
+   Review task actions for the expected executable, canonical script path, arguments,
+   principal, and original/reviewed definition; check all aliases/duplicate tasks.
+   Verify ACLs, signer/source provenance where used, absence of old running processes,
+   and the task state consistent with the maintenance hold. Do not overwrite an
+   established task definition merely to force a convenient state.
+5. Follow the existing credential transition/enrollment recovery workflow only if
+   required. Hub credentials and local miner credentials are different trust domains;
+   rotating a Hub secret cannot recall a cached physical command. Keep credentials
+   local and protected. Never restore an old credential/config to make a test pass.
+
+Exit gate: installed bytes and process/task state match the approved ledger for
+every Agent proposed for release. An inaccessible Agent, hash mismatch, duplicate
+task, unexpected process, or trust-reset blocker remains fenced and stopped.
+
+Gate 6: native Windows validation and fresh process startup
+
+Before production release, run on a network-isolated native Windows PowerShell 5.1
+host with simulated miners only. Linux PowerShell 7 tests are not evidence of SYSTEM
+task lifecycle correctness. Validate these points and retain results:
+ - Actual SYSTEM/Scheduled Task stop, child cleanup, restart hold and reboot behavior.
+ - Installed package hash verification and fresh module loading after upgrade.
+ - Legacy/malformed/unsupported cached commands rejected with zero physical writes.
+ - Diagnostic write/parameter overrides rejected before any device connection;
+   fixed telemetry/discovery/API3 read selections remain compatible with simulators.
+ - Enrollment, credential-transition interruption/recovery, and running/stopped intent
+   retain existing protections, without credentials appearing in normal output.
+ - A full network fence prevents an old cached command from reaching a live simulator;
+   a separate unfenced positive control proves the simulator and old-executor setup.
+
+1. Start only a fresh verified Agent process, initially still fenced, after all
+   preceding gates are satisfied. Verify executable/script paths, start time after
+   publication, loaded source provenance, per-Agent authentication, Zone assignment,
+   and unavailable control advertisement. The advertisement alone is not proof.
+2. Recheck the unsupported queue and audit reconciliation after startup. Do not
+   restore read connectivity for a host with any unverified old executor.
+3. Release the fence only for individually verified upgraded hosts under the approved
+   network change. Re-enable only reviewed telemetry schedules. Physical control
+   stays disabled on Hub AND Agent. Retain independent monitor/rollback evidence.
+
+Exit gate: every released Agent has complete positive verification. Fleet-wide
+containment cannot be claimed while unverified devices have an unfenced route.
+
+Gate 7: returning offline Agents and rollback/recovery
+
+Maintain a default quarantine for every offline, unknown, historical or unverified
+Agent when it reconnects. A timestamp, IP address, same VERSION, successful Hub
+authentication, or control_available=false claim cannot release it. Repeat stop,
+hash/task verification, upgrade, fresh-start and relevant gates before restoring
+miner connectivity. Keep retired Agent credentials retired; do not reopen legacy
+shared-secret fallback for convenience.
+
+If ANY gate fails:
+ - Keep or reinstate physical network fencing immediately. Stop/restart-hold Agent
+   executors and prevent old Hub producers from running. Preserve evidence and notify
+   the change owner; do not automatically resume the previous running state.
+ - Database/audit failure: leave rolled-back rows as evidence, resolve the error,
+   retry audited retirement, and re-run the global zero-live check. Never repair
+   audit integrity by deleting history or fabricating execution outcomes.
+ - Installer/recovery failure: preserve trust blockers, recovery tokens and protected
+   backups according to existing recovery instructions. Verify state before retry.
+ - If restoring old Hub or Agent binaries is unavoidable, those binaries stay stopped
+   and all affected miner routes stay fenced. An operational rollback does not restore
+   security containment. Do not reconnect until the reviewed Stage A bytes are again
+   installed and every failed gate is repeated successfully.
+ - If verification cannot be completed within the change window, leave the affected
+   host quarantined with a named owner. Partial success is not authority to un-fence it.
+
+Disposable simulation available for independent review
+
+agents/windows/tests/Test-MinerOpsOldAgentFence.ps1 loads AST functions from a
+separate git archive of 2d89e80 and the audit's cached suspend payload. It requires
+MINEROPS_STAGE_A_SIMULATION=1 and an explicitly provided simulated target. The
+remediation artifact directory contains simulate_fence.py and simulated_miner.py
+which create a temporary Docker INTERNAL network and fake API3 miner, then run the
+old executor first with --network none (zero simulated writes) and separately on
+the internal network (one simulated service-stop write). No host ports are exposed.
+Only those created resources are removed. This demonstrates the coarse fence's
+effect, not the adequacy of any production firewall/NAC design. Native Windows
+evidence and actual fleet/network verification remain separate release gates.