138 lines
4.0 KiB
Markdown
138 lines
4.0 KiB
Markdown
# MinerOps Operations Runbook
|
|
|
|
This file contains common day-to-day commands and safe procedures for operating MinerOps.
|
|
|
|
## Quick health check
|
|
|
|
Run from the Proxmox host:
|
|
|
|
pct exec 306 -- bash -lc "curl -s http://127.0.0.1:8000/api/health"
|
|
|
|
Expected result should include:
|
|
|
|
status: ok
|
|
database: ok
|
|
redis: ok
|
|
|
|
## Check Git status before changes
|
|
|
|
pct exec 306 -- bash -lc "cd /opt/minerops && git status --short"
|
|
|
|
If there is output, review it before changing files.
|
|
|
|
## Commit and push a stable change
|
|
|
|
pct exec 306 -- bash -lc "cd /opt/minerops && git add README.md CHANGELOG.md OPERATIONS.md backend scripts sql docker-compose.yml .gitignore && git commit -m 'Update MinerOps' && git push"
|
|
|
|
## Rebuild backend
|
|
|
|
Use this after changing backend/app/main.py or backend files.
|
|
|
|
pct exec 306 -- bash -lc "cd /opt/minerops && docker compose build backend || docker-compose build backend"
|
|
pct exec 306 -- bash -lc "cd /opt/minerops && docker compose up -d --force-recreate backend || docker-compose up -d --force-recreate backend"
|
|
|
|
Then check:
|
|
|
|
pct exec 306 -- bash -lc "docker exec minerops-backend python -m py_compile /app/app/main.py"
|
|
pct exec 306 -- bash -lc "curl -s http://127.0.0.1:8000/api/health"
|
|
|
|
## Check backend logs
|
|
|
|
pct exec 306 -- bash -lc "docker logs minerops-backend --since 30m"
|
|
|
|
For only errors:
|
|
|
|
pct exec 306 -- bash -lc "docker logs minerops-backend --since 30m 2>&1 | grep -iE 'error|exception|traceback|500|programmingerror|keyerror|typeerror|warning' | tail -n 120 || true"
|
|
|
|
## Main web pages
|
|
|
|
Dashboard:
|
|
https://plexus.lussier.cc/dashboard
|
|
|
|
Repair Queue:
|
|
https://plexus.lussier.cc/repairs
|
|
|
|
Locations:
|
|
https://plexus.lussier.cc/locations
|
|
|
|
System Health:
|
|
https://plexus.lussier.cc/system-health
|
|
|
|
Gitea repo:
|
|
https://gitea.lussier.cc/lussierj/minerops
|
|
|
|
## Repair workflow
|
|
|
|
1. Open /repairs.
|
|
2. Search by MAC, IP, model, serial, or error.
|
|
3. Add a repair note.
|
|
4. Select part used and quantity if applicable.
|
|
5. Set repair status.
|
|
6. Use History to review, edit, or delete repair entries.
|
|
7. Mark Ready to Shelf when repaired.
|
|
8. Use Move to Shelf.
|
|
9. Pick an open shelf position on /locations.
|
|
|
|
## Retention check
|
|
|
|
Open /system-health and review the Retention section.
|
|
|
|
Also run:
|
|
|
|
pct exec 306 -- bash -lc "tail -n 120 /opt/minerops/logs/status-retention.log || true"
|
|
|
|
Check recent retention runs:
|
|
|
|
pct exec 306 -- bash -lc "docker exec -i minerops-postgres psql -U minerops -d minerops -c \"SELECT mode, raw_status_rows_before, raw_status_rows_eligible_delete, raw_status_rows_deleted, hourly_rollup_rows_after, daily_rollup_rows_after, started_at, finished_at FROM minerops_retention_runs ORDER BY started_at DESC LIMIT 5;\""
|
|
|
|
Retention is currently dry-run only. Do not switch to delete/apply mode until the dry-run has been reviewed.
|
|
|
|
## Roll back to last Git version
|
|
|
|
Check current status first:
|
|
|
|
pct exec 306 -- bash -lc "cd /opt/minerops && git status --short"
|
|
|
|
To discard uncommitted file changes:
|
|
|
|
pct exec 306 -- bash -lc "cd /opt/minerops && git restore ."
|
|
|
|
To reset to the last committed version:
|
|
|
|
pct exec 306 -- bash -lc "cd /opt/minerops && git reset --hard HEAD"
|
|
|
|
Then rebuild backend if backend files were involved.
|
|
|
|
## Important safety rules
|
|
|
|
- Do not commit .env files.
|
|
- Do not commit database dumps.
|
|
- Do not commit backups.
|
|
- Do not commit logs.
|
|
- Do not commit imports.
|
|
- Check Git status before and after every patch.
|
|
- Commit and push after every stable working change.
|
|
- Use /system-health after major changes.
|
|
|
|
## Linux read-only miner scanner
|
|
|
|
A standalone Linux scanner is included at:
|
|
|
|
scripts/linux-miner-scanner.py
|
|
|
|
Run it from a Linux machine that can reach the miner networks. CT 306 may not have routes to the miner VLANs, so run this on the correct network.
|
|
|
|
Example:
|
|
|
|
python3 scripts/linux-miner-scanner.py --cidr 10.30.0.0/22 --cidr 10.80.0.0/22 --csv miner-scan.csv --json miner-scan.json
|
|
|
|
Single miner test:
|
|
|
|
python3 scripts/linux-miner-scanner.py --ip 10.30.0.138
|
|
|
|
Port check only:
|
|
|
|
python3 scripts/linux-miner-scanner.py --cidr 10.30.0.0/22 --no-query
|
|
|
|
This scanner is read-only. It checks legacy port 4028 and API3 port 4433.
|