minerops/docs/ARCHITECTURE.md

MinerOps Architecture

Purpose

This document is the primary engineering architecture guide for MinerOps. It explains the codebase's current structure and the rules contributors must follow when adding or changing features.

MinerOps is self-hosted software with operational data and workflows that must remain understandable as the system grows. Maintainability, testability, and clear boundaries are primary architectural goals. A feature is not complete merely because it works; its responsibilities and dependencies must also be in the correct layer.

Current implementation: The repository is transitional. Most existing HTTP routes, persistence, and server-rendered assets are concentrated in backend/app/main.py. Identity has been separated into a FastAPI router. Inventory is the first domain with explicit models, exceptions, a service, a repository, and focused tests.

Engineering rule: New domain code follows the layered architecture in this document. Existing integrated code is compatibility context, not a pattern to copy.

Core Principles

Separation of Concerns

Transport, business rules, persistence, and presentation are different responsibilities. Keep them separate so each can change and be tested without requiring unrelated changes.

Single Responsibility

Each module, class, and function should have one clear reason to change. A router handles HTTP. A service handles domain behavior. A repository handles persistence. Combining these responsibilities makes ownership ambiguous and tests expensive.

Thin HTTP Layer

Routers accept and validate transport input, invoke services, and map results to HTTP responses. They do not implement business workflows and do not access the database directly.

Business Logic Belongs in Services

Services own normalization, domain validation, decisions, and orchestration. They must not depend on FastAPI request or response objects, route decorators, or HTTP status codes.

Persistence Belongs in Repositories

Repositories own queries, writes, database-facing filters, transaction mechanics, and persistence mapping. They contain no HTTP behavior, permission rules, or product decisions.

Explicit Dependencies

Dependencies are visible at construction or call boundaries. The current InventoryService receives its repository through its constructor, allowing a substitute repository in service tests and a PostgreSQL repository in integration tests.

Incremental Evolution

MinerOps has working behavior that must remain usable while responsibilities are separated. Improve boundaries in small, verified changes. Do not require a large rewrite before a domain can become clearer.

Testability

Architecture should make the correct test easy to write. Business rules should be testable without FastAPI or PostgreSQL. Repository behavior should be testable against a disposable PostgreSQL database. Cross-application behavior should be testable through the disposable development stack.

High-Level Architecture

The required dependency flow for domain-backed application features is:

Browser
  |
  v
FastAPI Routers
  |
  v
Services
  |
  v
Repositories
  |
  v
PostgreSQL

• Browser: Presents the interface and sends authenticated HTTP requests. The current interface is served by FastAPI using HTML, CSS, and JavaScript.
• FastAPI routers: Define the HTTP contract, enforce transport-level validation and authorization dependencies, call services, and serialize responses.
• Services: Implement use cases and domain rules without knowing how the request arrived.
• Repositories: Perform database operations and expose persistence behavior needed by services.
• PostgreSQL: Stores authoritative operational records and enforces durable constraints.

Redis and Nginx are current infrastructure components, but they do not change this domain dependency flow. Redis is configured as an application service and included in health checks. Nginx is the gateway in the development Compose stack.

Repository Organization

The repository is organized by runtime responsibility and supporting material:

• backend/: Python application code, FastAPI composition, domain packages, Alembic migrations, backend dependencies, tests, and the backend container definition.
• frontend/ (planned): A future standalone frontend may live here after a frontend technology and migration plan are selected. No frontend/ directory or standalone frontend application exists today.
• docs/: Engineering and product documentation that applies across domains.
• branding/: Product identity guidelines and source vector assets. It has no runtime application responsibility.
• scripts/: Development setup, disposable test runners, schema checks, retention helpers, and operational utilities.

Root Compose files define containerized services for the backend, PostgreSQL, and Redis. The development configuration also defines an Nginx gateway and runs Alembic before starting Uvicorn.

Backend Layers

Routers

Routers are the HTTP boundary. A domain router should be an APIRouter included by the application entry point.

Router responsibilities:

• Declare paths, HTTP methods, parameters, and response status codes.
• Parse path, query, header, and body values.
• Apply authentication and permission dependencies.
• Construct domain input models from transport input.
• Call services.
• Translate domain exceptions into consistent HTTP errors.
• Serialize service results into documented response models.

Routers must never contain:

• SQL or database connection management.
• Business rules, normalization policy, or lifecycle decisions.
• Multi-step persistence workflows or transaction control.
• Device protocol implementations.
• Copies of logic already owned by a service or repository.

A router may reject structurally invalid HTTP input. It must delegate questions such as whether an Asset ID is valid for a use case, whether a lifecycle change is allowed, or whether a record conflicts with existing state to the service.

Current implementation: backend/app/main.py creates the FastAPI application and includes the router from backend/app/identity.py. Identity is the current example of router extraction. Most other routes remain registered directly on app and often contain SQL or business logic. Those routes predate the current layering rules and must not be used as templates for new domains. The Inventory package currently has no router and registers no HTTP endpoints.

Services

Services implement application use cases for a domain. A service coordinates repositories and applies rules that must remain consistent regardless of the transport calling it.

Service responsibilities:

• Normalize values according to domain semantics.
• Validate business rules and allowed state transitions.
• Detect conflicts meaningful to the domain.
• Coordinate multiple repository operations as one use case.
• Establish required transaction scope through repository abstractions.
• Return domain models or domain-specific results.
• Raise domain exceptions rather than HTTP exceptions.

The current InventoryService demonstrates these responsibilities. It:

• Normalizes Asset IDs and optional text.
• Normalizes and deduplicates tags.
• Validates active, spare, and retired lifecycle states.
• Detects duplicate Asset IDs.
• Defines never-seen timestamps for manually created miners.
• Coordinates miner creation or update with tag associations.
• Maps database-shaped records to Inventory summary and detail models.

The service imports no FastAPI components. Its extension hooks are domain-level hooks, not route callbacks. A service must not inspect cookies, headers, request objects, or response objects.

The current Inventory service catches a PostgreSQL uniqueness exception to translate a race into DuplicateAssetID. This is a known persistence coupling. New code should prevent database-specific exceptions from spreading further through the service layer, preferably by translating them at an explicit repository boundary.

Repositories

Repositories are the persistence boundary for a domain. They expose operations in terms useful to services while keeping SQL and database mechanics out of higher layers.

Repository responsibilities:

• Execute parameterized reads and writes.
• Implement database-facing search, filtering, ordering, and pagination.
• Manage connections, commits, rollbacks, and transaction-bound instances.
• Maintain associations owned by the domain.
• Map database values into persistence records.
• Translate low-level persistence failures when an owned abstraction exists.

Repositories contain database access only. They must not:

• Import FastAPI or router modules.
• Choose HTTP responses or status codes.
• Check the current user's permissions.
• Normalize values according to product policy.
• Decide whether a lifecycle transition or operational action is allowed.

InventoryRepository is the current reference implementation. It uses psycopg, accepts an optional connection factory, supports explicit transaction scope, and owns miner and tag CRUD, search, and associations. Its methods return database-shaped dictionaries; InventoryService maps those records to domain models.

Dynamic SQL must be restricted to trusted identifiers or fixed fragments. User-provided values always use database parameters. Repositories must not accept raw SQL from routers or services.

Database

PostgreSQL is the authoritative persistent store for MinerOps. Application rules that must hold for every writer are also represented as database constraints where practical, including uniqueness, allowed values, and referential integrity.

Alembic is the authority for new persistent schema changes. Migrations must state how existing data is handled and provide a downgrade when it can be done without ambiguity or data loss.

Current implementation: Alembic migrations cover the foundational, scanning, audit, identity, role, and Inventory schemas. Some legacy code in main.py still creates or alters tables at runtime for compatibility. Do not add new runtime DDL. Move persistent schema changes through reviewed Alembic migrations.

The repository also contains a SQLAlchemy engine and session factory, while most current operational persistence and the Inventory repository use direct psycopg; some existing agent paths use asyncpg. A domain must choose and encapsulate its persistence mechanism inside its repository rather than expose driver details to routers.

Domains

A domain package groups one operational capability behind an explicit public interface. Inventory is the current reference domain:

backend/app/inventory/
    __init__.py
    exceptions.py
    models.py
    repository.py
    service.py

• models.py defines typed request, update, search, summary, detail, and tag contracts.
• exceptions.py defines errors meaningful to Inventory callers.
• service.py defines Inventory rules and use cases.
• repository.py defines Inventory persistence.
• __init__.py exposes the package's intended public surface.

Inventory's internal asset identity is the miner UUID. Product-facing Asset IDs are normalized and checked independently of mutable network fields. The domain maps product terminology such as asset_id and psu_serial to existing database columns at its service and persistence boundary.

Future domain packages should follow the same separation when it fits their use cases. Identity, Repairs, Monitoring, Operations, Scheduling, and Locations are examples of possible domain boundaries, not a fixed package list. A domain boundary must be based on ownership of rules and data, not on menu pages or route prefixes alone.

A domain may call another domain only through an explicit service contract. It must not import another domain's repository to update tables it does not own. Shared read requirements should use a clearly owned query service or read model rather than duplicating cross-domain SQL.

Dependency Rules

Dependencies flow downward:

Routers
  |
  v
Services
  |
  v
Repositories
  |
  v
Database

Allowed dependencies:

• The application entry point may include routers and install application-wide middleware.
• Routers may depend on domain models, services, and exceptions.
• Services may depend on their domain's models, exceptions, and explicit repository contracts.
• Repositories may depend on configuration, database drivers, and persistence-specific utilities.
• Tests may replace repositories or connection factories at defined dependency boundaries.

Examples of allowed flow:

• An Inventory router validates a UUID, calls InventoryService.get_miner, and translates MinerNotFound into an HTTP 404 response.
• InventoryService.create_miner normalizes an Asset ID and coordinates miner and tag writes through an InventoryRepository transaction.
• InventoryRepository.search_miners builds parameterized filters and executes them against PostgreSQL.

Prohibited dependencies:

• Repositories importing services, routers, or app.main.
• Services importing FastAPI request or response classes.
• Routers opening database connections or issuing SQL.
• Domain code importing globals from app.main.
• A lower layer choosing permissions or HTTP status codes.
• A domain bypassing another domain's service to write its tables.
• Circular imports between domains or layers.

Examples of prohibited flow:

• A route updating miners directly after checking request JSON.
• A service returning JSONResponse or raising HTTPException.
• A repository checking whether the current user is an administrator.
• A Repairs service importing InventoryRepository to mutate miner ownership without an Inventory service contract.

Dependency direction is an architectural constraint, not a naming convention. Putting SQL in a file named service.py does not make it service logic.

Database Philosophy

• Alembic-managed schema: New persistent tables, columns, constraints, and indexes are introduced through versioned migrations.
• Explicit migrations: Every migration explains existing-data behavior, upgrade ordering, and downgrade behavior. Migrations do not silently merge, rename, or discard operational records.
• Stable identifiers: Internal identity uses immutable identifiers rather than mutable IP addresses, MAC addresses, names, or locations.
• Repository ownership: Each domain has one clear persistence boundary for its owned records. Other layers do not duplicate its SQL.
• Database enforcement: PostgreSQL constraints protect invariants that must hold independently of application code.
• Transactional consistency: Operations that must succeed or fail together execute in one explicit transaction.
• Data preservation: Schema evolution preserves existing records unless a separately reviewed product decision explicitly requires otherwise.

Migration tests should verify upgrade, downgrade, constraints, and existing-data behavior. Runtime schema changes are legacy compatibility behavior and are not permitted for new work.

API Philosophy

MinerOps APIs use resource-oriented HTTP semantics. New or replaced domain APIs follow these conventions:

• Use nouns for resources and standard HTTP methods for operations.
• Validate path, query, and body structure at the router boundary.
• Apply business validation in the service.
• Use Pydantic models for request and response contracts.
• Return predictable status codes and response shapes.
• Translate domain exceptions into consistent, non-leaking API errors.
• Bound collection endpoints with pagination.
• Make supported filtering, searching, and ordering explicit.
• Preserve stable identifiers in URLs and response contracts.
• Enforce authentication and permissions on the server.
• Avoid duplicate endpoints for the same use case.

Current implementation: Many routes return the common data, meta, and error envelope defined in main.py. Pydantic validation, pagination, and error shapes are not uniform across all existing routes. Identity supplies current authentication and role-aware dependencies. New endpoints should follow the conventions above while preserving compatibility for any route they replace.

Pagination metadata should be sufficient for callers to understand the returned window. Filters must be parameterized in repositories, not interpolated in routers. Database exceptions and internal details must not be exposed to API clients.

Testing Philosophy

Tests follow the same boundaries as production code.

• Unit tests: Exercise service rules with substituted dependencies. They should not require FastAPI or a database when the behavior is purely domain logic.
• Repository tests: Exercise real queries, constraints, associations, and transactions against a disposable PostgreSQL database.
• Migration tests: Verify schema upgrade, downgrade, constraint behavior, and existing-data preservation.
• Integration tests: Exercise application startup, migrations, authentication, authorization, safety behavior, and cross-layer workflows in a disposable Compose stack.
• Architecture tests: Protect dependency rules that can be checked mechanically, such as the Inventory package containing no FastAPI imports or router registration.

Current Inventory tests use a fake repository for service rules and PostgreSQL for repository behavior. The focused runner migrates a disposable database through Alembic before executing the suite. The development baseline starts an isolated Compose project and checks schema, login gating, roles, and safety interlocks.

Before committing, run validation appropriate to the change. This commonly includes Python compilation, shell syntax checks, focused unit or repository tests, migration tests for schema work, the disposable development baseline for cross-application changes, and git diff --check.

Keep commits small and reviewable. A commit should express one coherent change, include focused tests when behavior changes, and avoid unrelated formatting or refactoring. Broad integration coverage does not replace a focused test for a new rule.

Future Evolution

MinerOps will evolve incrementally as existing responsibilities are moved into clear domain boundaries. New infrastructure or domain packages must preserve the router-to-service-to-repository dependency direction, explicit ownership, and testability rules in this document. Architectural changes should clarify those rules rather than bypass them.

When In Doubt

When a feature does not fit naturally into an existing module, refine the architecture instead of weakening architectural boundaries. Clear boundaries are easier to maintain than clever shortcuts.