MinerOps Blocker #3 Stage A: old-Agent containment and upgrade runbook
Status: review candidate. This procedure has NOT been executed in production.
Scope: contain physical control; no Stages B-E protocol or control re-enablement.

The independent audit demonstrated that an old Agent can execute a previously
delivered command after the Hub marks it cancelled. Cancelling a database row,
revoking a Hub credential, advertising control unavailable, or replacing script
files does not revoke a cached command or loaded PowerShell functions. Execution
of any previously claimed command remains UNKNOWN until separately investigated.

Production approval requires independent security revalidation, an approved change
window, verified network enforcement, and the native Windows checks below. A new
database migration or production network infrastructure change requires explicit
approval. This runbook authorizes no production actions by itself.

Ownership and evidence

Assign an incident/change owner, a network operator, a Hub/database operator, and a
Windows operator. Record their identities, UTC timestamps, change ticket, and every
gate's evidence in a protected location. Keep enrollment tokens and Agent/miner
credentials out of ordinary logs and this checklist. Preserve required backups
and recovery state with their existing ACLs; do not overwrite agent-config.json.

Maintain one ledger containing ALL registered Agents, every distinct queue Agent
(including deleted registry entries), offline/unreachable Agents, alternative
installations and manually running executors, legacy shared identities, all Hub
replicas/workers, scheduled jobs, task hosts, physical networks, and IPv4/IPv6 paths.
For each Agent record its Site/Zone assignment, last check-in, host/network identity,
network fence evidence, process/task evidence, installed-file hashes, verified
artifact identifier, and release decision. Keep unverified entries quarantined.

Gate 1: restrict physical write access BEFORE stopping/upgrading anything

1. Arrange an approved maintenance window and identify the actual network control
   that can keep old Agents from reaching ALL physical miners. Layer-4 rules cannot
   distinguish a legacy read from a write on the same miner API port. Default to
   denying Agent-to-miner traffic, including ports 4028 and 4433, alternative miner
   management interfaces/ports, IPv6, alternate NICs/VPNs, and established sessions.
   Telemetry may be interrupted. Do not substitute an unreviewed 'read-only' proxy.
2. Apply only separately approved network controls through the responsible operator.
   A stopped process or a host-local rule alone is insufficient for an unreachable
   machine that can later reboot/reconnect. Use enforcement that already applies
   to that machine's traffic when it returns. Include unknown/retired installations
   and aliases; trusting an IP address alone is insufficient when addresses can change.
3. Verify deny counters and session termination/conntrack handling. Use approved
   synthetic targets/canaries, never a real write command to a physical miner.
   A rule that blocks only NEW connections leaves a pre-existing API session exposed.
   Check both the intended route and alternate routes. Keep the fence if evidence
   is unavailable, ambiguous, or an offline Agent cannot be located.

Exit gate: verified deny enforcement covers every unverified/legacy executor,
including future reconnects and existing sessions. Failure means STOP; keep fences.

Gate 2: stop old executors and automatic restarts

1. Stop command production and processing on every old Hub/worker version, including
   old schedule materializers and running requests that could still insert/deliver
   commands. Hold ingress and scale/restart mechanisms as necessary under the
   approved change procedure. Inventory the actual versions of every instance.
2. On each reachable Windows host disable/hold all relevant Scheduled Task triggers,
   retry/restart policies, service recovery, watchdogs, startup/logon entries, GUI
   launch actions, manual jobs, and external orchestration that can launch MinerOps.
   Record original definitions and intended running/stopped state for recovery.
3. Stop MinerOpsAgent and any alternative/manual executor, then verify task instances
   AND their PowerShell/child process trees have exited. Record command paths,
   process start times and identities. An in-memory old module is still executable
   even after its source file changes. Do not gracefully 'drain' by executing writes.
4. Keep pool/enrichment tasks held during runtime replacement and verify no alternate
   old code copy can auto-start. Never run uninstall or erase config/state to stop it.

Exit gate: every reachable old executor is stopped and restart-held; every
unreachable/unverified Agent remains network-fenced. Do not remove a fence to
diagnose a failed stop. Escalate failures with evidence and keep maintenance holds.

Gate 3: update ALL Hub instances with no mixed-version execution

1. Capture protected queue/audit evidence and the relevant schedule/Agent inventory
   before changes. Retain original request/result payloads, timestamps and errors.
   Secure these exports because historical payloads may contain sensitive data.
2. Install only the independently reviewed Stage A artifact on every Hub instance
   while executor fences/holds remain. Stop old replicas and workers fully, including
   rolling-update stragglers and old connections/requests; do not allow a mixture of
   active baseline and Stage A schedulers/queue consumers.
3. Verify each instance's actual artifact hashes and that physical admission is
   fixed disabled: legacy write/diagnostic POSTs return 410 after authorization,
   canonical typed controls return 409 physical_control_disabled_stage_a, and queue
   SQL dispatches only the fixed read types. Do not use runtime flags to enable writes.
4. Run reviewed schema compatibility checks as appropriate. Stage A remediation adds
   NO migration. Stop and request approval if the deployed schema needs one.

Exit gate: only reviewed containment Hub code is active; all old producers are
stopped. Keep Agents held. If any instance cannot be verified, keep ingress/worker
holds and physical fences rather than continuing a partial rolling deployment.

Gate 4: audited cancellation across ALL Agents, including offline/queue-only IDs

1. Use the canonical reviewed code and a real accountable operator users.id for
   manual cutover retirement. Existing authenticated Agent polling/result rejection
   is system-attributed by source and Agent ID; manual cutover supplies actor_user_id.
   Do not impersonate another user, create invented execution evidence, or log secrets.
2. Inventory IDs using the UNION of registry and queue Agent IDs. Also reconcile the
   fleet ledger, historical/retired identities, and old shared Agent ID. Do not use
   only Agents currently checking in. A useful read-only inventory is:

     SELECT agent_id FROM minerops_site_agents
     UNION SELECT agent_id FROM minerops_agent_command_queue
     ORDER BY agent_id;

3. In a separately approved administrative process configured for the intended DB,
   call the reviewed repository operation for each ID, in committed bounded batches:

     repository = AgentCommandRepository()
     count = repository.retire_unsupported(agent_id, actor_user_id=operator_user_id)

   Repeat for all IDs while progress is made, under an explicit time/batch limit.
   This method never claims read work; do NOT substitute claim_pending or an Agent
   HTTP poll as an administrative drain. Each transition and rejection audit commits
   atomically. Database/audit failure rolls back and keeps the change window closed.
4. A batch is bounded at 1000 and uses SKIP LOCKED. A zero count may mean locked rows,
   not completion. Resolve/retry approved lock holders with time limits; never
   silently declare the drain finished. Query ALL Agents to verify a zero live count:

     SELECT agent_id, command_type, command_status, count(*)
     FROM minerops_agent_command_queue
     WHERE command_status IN ('pending', 'claimed')
       AND command_type NOT IN
           ('poll_known_miners_now', 'discover_new_miners', 'enrich_match_discovery')
     GROUP BY agent_id, command_type, command_status;

5. Reconcile newly cancelled rows to agent.command.rejected records. Check command ID,
   Agent ID, prior state, immutable rejection reason, source and operator attribution.
   Current pending status does not establish that a command was never delivered.
   Current claimed status, any retained claimed_at, completion/result/error evidence,
   linked legacy result or non-creation audit history, an unlinked legacy result
   for that Agent, or an Agent result submission
   requires execution_outcome=unknown. Missing or ambiguous evidence is unknown.
   Only a pending row with a complete pre-cancellation check finding no delivery
   evidence is classified not_dispatched by the queue. Historical claim evidence
   survives cancellation; an explicitly repaired/requeued pending row is still
   unknown. No ordinary production requeue mechanism is introduced by Stage A.
   Neither classification proves absence of out-of-band historical physical writes.
   Repeated retirement must add no duplicate audit and must not rewrite history.
6. Schedule cancel/delete uses the stored schedule's Agent ID and the same rejection
   audit policy; delete retains the schedule. This is NOT a fleet drain: orphaned or
   foreign-Agent rows are handled by the per-Agent cutover sweep above. Never delete
   queue/audit rows, reset claimed commands to pending, or rewrite terminal results.
   Existing pre-remediation cancelled rows missing an audit must be preserved and
   documented as a historical evidence gap. Do not invent a prior-state audit.

Exit gate: the global live-unsupported query returns no rows, locks are accounted
for, transition counts/audits reconcile, and no old producer can replenish work.
This gate does not revoke already delivered payloads; gates 1, 2 and 5 remain mandatory.

Gate 5: upgrade Windows code and verify installed bytes while stopped

1. Use a verified artifact prepared from the exact independently reviewed Stage A
   boundary. Record its digest and a trusted expected SHA-256 manifest outside the
   target installation. VERSION is unchanged at 1.0.0-alpha and is not upgrade proof.
2. Include all managed runtime scripts, especially MinerOpsAgentService.ps1,
   MinerOpsControl.ps1, EnrichOneMiner.ps1, MinerOpsApi3EnrichmentPoller.ps1, the pool
   pollers, task/GUI/tray scripts, and VERSION. Check the package's installer and
   enrollment/credential-upgrade entry points as well. Include the manifest's origin
   and review approval; hashes computed only from a suspect host are not a trust root.
3. Confirm the main task was positively stopped BEFORE invoking the existing
   installer. It can auto-restart a task that it found running. Keep the external
   network fence and restart hold throughout installation and failure recovery.
   Do not pass filesystem trust-reset switches unless their existing reviewed
   recovery workflow specifically calls for them. Preserve config/state/logs/backups.
4. Compare installed Get-FileHash -Algorithm SHA256 values to the trusted manifest.
   Review task actions for the expected executable, canonical script path, arguments,
   principal, and original/reviewed definition; check all aliases/duplicate tasks.
   Verify ACLs, signer/source provenance where used, absence of old running processes,
   and the task state consistent with the maintenance hold. Do not overwrite an
   established task definition merely to force a convenient state.
5. Follow the existing credential transition/enrollment recovery workflow only if
   required. Hub credentials and local miner credentials are different trust domains;
   rotating a Hub secret cannot recall a cached physical command. Keep credentials
   local and protected. Never restore an old credential/config to make a test pass.

Exit gate: installed bytes and process/task state match the approved ledger for
every Agent proposed for release. An inaccessible Agent, hash mismatch, duplicate
task, unexpected process, or trust-reset blocker remains fenced and stopped.

Gate 6: native Windows validation and fresh process startup

Before production release, run on a network-isolated native Windows PowerShell 5.1
host with simulated miners only. Linux PowerShell 7 tests are not evidence of SYSTEM
task lifecycle correctness. Validate these points and retain results:
 - Actual SYSTEM/Scheduled Task stop, child cleanup, restart hold and reboot behavior.
 - Installed package hash verification and fresh module loading after upgrade.
 - Legacy/malformed/unsupported cached commands rejected with zero physical writes.
 - Diagnostic write/parameter overrides rejected before any device connection;
   fixed telemetry/discovery/API3 read selections remain compatible with simulators.
 - Enrollment, credential-transition interruption/recovery, and running/stopped intent
   retain existing protections, without credentials appearing in normal output.
 - A full network fence prevents an old cached command from reaching a live simulator;
   a separate unfenced positive control proves the simulator and old-executor setup.

1. Start only a fresh verified Agent process, initially still fenced, after all
   preceding gates are satisfied. Verify executable/script paths, start time after
   publication, loaded source provenance, per-Agent authentication, Zone assignment,
   and unavailable control advertisement. The advertisement alone is not proof.
2. Recheck the unsupported queue and audit reconciliation after startup. Do not
   restore read connectivity for a host with any unverified old executor.
3. Release the fence only for individually verified upgraded hosts under the approved
   network change. Re-enable only reviewed telemetry schedules. Physical control
   stays disabled on Hub AND Agent. Retain independent monitor/rollback evidence.

Exit gate: every released Agent has complete positive verification. Fleet-wide
containment cannot be claimed while unverified devices have an unfenced route.

Gate 7: returning offline Agents and rollback/recovery

Maintain a default quarantine for every offline, unknown, historical or unverified
Agent when it reconnects. A timestamp, IP address, same VERSION, successful Hub
authentication, or control_available=false claim cannot release it. Repeat stop,
hash/task verification, upgrade, fresh-start and relevant gates before restoring
miner connectivity. Keep retired Agent credentials retired; do not reopen legacy
shared-secret fallback for convenience.

If ANY gate fails:
 - Keep or reinstate physical network fencing immediately. Stop/restart-hold Agent
   executors and prevent old Hub producers from running. Preserve evidence and notify
   the change owner; do not automatically resume the previous running state.
 - Database/audit failure: leave rolled-back rows as evidence, resolve the error,
   retry audited retirement, and re-run the global zero-live check. Never repair
   audit integrity by deleting history or fabricating execution outcomes.
 - Installer/recovery failure: preserve trust blockers, recovery tokens and protected
   backups according to existing recovery instructions. Verify state before retry.
 - If restoring old Hub or Agent binaries is unavoidable, those binaries stay stopped
   and all affected miner routes stay fenced. An operational rollback does not restore
   security containment. Do not reconnect until the reviewed Stage A bytes are again
   installed and every failed gate is repeated successfully.
 - If verification cannot be completed within the change window, leave the affected
   host quarantined with a named owner. Partial success is not authority to un-fence it.

Disposable simulation available for independent review

agents/windows/tests/Test-MinerOpsOldAgentFence.ps1 loads AST functions from a
separate git archive of 2d89e80 and the audit's cached suspend payload. It requires
MINEROPS_STAGE_A_SIMULATION=1 and an explicitly provided simulated target. The
remediation artifact directory contains simulate_fence.py and simulated_miner.py
which create a temporary Docker INTERNAL network and fake API3 miner, then run the
old executor first with --network none (zero simulated writes) and separately on
the internal network (one simulated service-stop write). No host ports are exposed.
Only those created resources are removed. This demonstrates the coarse fence's
effect, not the adequacy of any production firewall/NAC design. Native Windows
evidence and actual fleet/network verification remain separate release gates.
