diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..fed3a08 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,457 @@ +# MinerOps Architecture + +## Purpose + +This document is the primary engineering architecture guide for MinerOps. It +explains the codebase's current structure and the rules contributors must follow +when adding or changing features. + +MinerOps is self-hosted software with operational data and workflows that must +remain understandable as the system grows. Maintainability, testability, and +clear boundaries are primary architectural goals. A feature is not complete +merely because it works; its responsibilities and dependencies must also be in +the correct layer. + +**Current implementation:** The repository is transitional. Most existing HTTP +routes, persistence, and server-rendered assets are concentrated in +`backend/app/main.py`. Identity has been separated into a FastAPI router. +Inventory is the first domain with explicit models, exceptions, a service, a +repository, and focused tests. + +**Engineering rule:** New domain code follows the layered architecture in this +document. Existing integrated code is compatibility context, not a pattern to +copy. + +## Core Principles + +### Separation of Concerns + +Transport, business rules, persistence, and presentation are different +responsibilities. Keep them separate so each can change and be tested without +requiring unrelated changes. + +### Single Responsibility + +Each module, class, and function should have one clear reason to change. A +router handles HTTP. A service handles domain behavior. A repository handles +persistence. Combining these responsibilities makes ownership ambiguous and +tests expensive. + +### Thin HTTP Layer + +Routers accept and validate transport input, invoke services, and map results to +HTTP responses. They do not implement business workflows and do not access the +database directly. + +### Business Logic Belongs in Services + +Services own normalization, domain validation, decisions, and orchestration. +They must not depend on FastAPI request or response objects, route decorators, +or HTTP status codes. + +### Persistence Belongs in Repositories + +Repositories own queries, writes, database-facing filters, transaction +mechanics, and persistence mapping. They contain no HTTP behavior, permission +rules, or product decisions. + +### Explicit Dependencies + +Dependencies are visible at construction or call boundaries. The current +`InventoryService` receives its repository through its constructor, allowing a +substitute repository in service tests and a PostgreSQL repository in +integration tests. + +### Incremental Evolution + +MinerOps has working behavior that must remain usable while responsibilities +are separated. Improve boundaries in small, verified changes. Do not require a +large rewrite before a domain can become clearer. + +### Testability + +Architecture should make the correct test easy to write. Business rules should +be testable without FastAPI or PostgreSQL. Repository behavior should be +testable against a disposable PostgreSQL database. Cross-application behavior +should be testable through the disposable development stack. + +## High-Level Architecture + +The required dependency flow for domain-backed application features is: + +```text +Browser + | + v +FastAPI Routers + | + v +Services + | + v +Repositories + | + v +PostgreSQL +``` + +- **Browser:** Presents the interface and sends authenticated HTTP requests. The + current interface is served by FastAPI using HTML, CSS, and JavaScript. +- **FastAPI routers:** Define the HTTP contract, enforce transport-level + validation and authorization dependencies, call services, and serialize + responses. +- **Services:** Implement use cases and domain rules without knowing how the + request arrived. +- **Repositories:** Perform database operations and expose persistence behavior + needed by services. +- **PostgreSQL:** Stores authoritative operational records and enforces durable + constraints. + +Redis and Nginx are current infrastructure components, but they do not change +this domain dependency flow. Redis is configured as an application service and +included in health checks. Nginx is the gateway in the development Compose +stack. + +## Repository Organization + +The repository is organized by runtime responsibility and supporting material: + +- **`backend/`:** Python application code, FastAPI composition, domain packages, + Alembic migrations, backend dependencies, tests, and the backend container + definition. +- **`frontend/` (planned):** A future standalone frontend may live here after a + frontend technology and migration plan are selected. No `frontend/` directory + or standalone frontend application exists today. +- **`docs/`:** Engineering and product documentation that applies across + domains. +- **`branding/`:** Product identity guidelines and source vector assets. It has + no runtime application responsibility. +- **`scripts/`:** Development setup, disposable test runners, schema checks, + retention helpers, and operational utilities. + +Root Compose files define containerized services for the backend, PostgreSQL, +and Redis. The development configuration also defines an Nginx gateway and runs +Alembic before starting Uvicorn. + +## Backend Layers + +### Routers + +Routers are the HTTP boundary. A domain router should be an `APIRouter` included +by the application entry point. + +Router responsibilities: + +- Declare paths, HTTP methods, parameters, and response status codes. +- Parse path, query, header, and body values. +- Apply authentication and permission dependencies. +- Construct domain input models from transport input. +- Call services. +- Translate domain exceptions into consistent HTTP errors. +- Serialize service results into documented response models. + +Routers must never contain: + +- SQL or database connection management. +- Business rules, normalization policy, or lifecycle decisions. +- Multi-step persistence workflows or transaction control. +- Device protocol implementations. +- Copies of logic already owned by a service or repository. + +A router may reject structurally invalid HTTP input. It must delegate questions +such as whether an Asset ID is valid for a use case, whether a lifecycle change +is allowed, or whether a record conflicts with existing state to the service. + +**Current implementation:** `backend/app/main.py` creates the `FastAPI` +application and includes the router from `backend/app/identity.py`. Identity is +the current example of router extraction. Most other routes remain registered +directly on `app` and often contain SQL or business logic. Those routes predate +the current layering rules and must not be used as templates for new domains. +The Inventory package currently has no router and registers no HTTP endpoints. + +### Services + +Services implement application use cases for a domain. A service coordinates +repositories and applies rules that must remain consistent regardless of the +transport calling it. + +Service responsibilities: + +- Normalize values according to domain semantics. +- Validate business rules and allowed state transitions. +- Detect conflicts meaningful to the domain. +- Coordinate multiple repository operations as one use case. +- Establish required transaction scope through repository abstractions. +- Return domain models or domain-specific results. +- Raise domain exceptions rather than HTTP exceptions. + +The current `InventoryService` demonstrates these responsibilities. It: + +- Normalizes Asset IDs and optional text. +- Normalizes and deduplicates tags. +- Validates `active`, `spare`, and `retired` lifecycle states. +- Detects duplicate Asset IDs. +- Defines never-seen timestamps for manually created miners. +- Coordinates miner creation or update with tag associations. +- Maps database-shaped records to Inventory summary and detail models. + +The service imports no FastAPI components. Its extension hooks are domain-level +hooks, not route callbacks. A service must not inspect cookies, headers, request +objects, or response objects. + +The current Inventory service catches a PostgreSQL uniqueness exception to +translate a race into `DuplicateAssetID`. This is a known persistence coupling. +New code should prevent database-specific exceptions from spreading further +through the service layer, preferably by translating them at an explicit +repository boundary. + +### Repositories + +Repositories are the persistence boundary for a domain. They expose operations +in terms useful to services while keeping SQL and database mechanics out of +higher layers. + +Repository responsibilities: + +- Execute parameterized reads and writes. +- Implement database-facing search, filtering, ordering, and pagination. +- Manage connections, commits, rollbacks, and transaction-bound instances. +- Maintain associations owned by the domain. +- Map database values into persistence records. +- Translate low-level persistence failures when an owned abstraction exists. + +Repositories contain database access only. They must not: + +- Import FastAPI or router modules. +- Choose HTTP responses or status codes. +- Check the current user's permissions. +- Normalize values according to product policy. +- Decide whether a lifecycle transition or operational action is allowed. + +`InventoryRepository` is the current reference implementation. It uses +`psycopg`, accepts an optional connection factory, supports explicit transaction +scope, and owns miner and tag CRUD, search, and associations. Its methods return +database-shaped dictionaries; `InventoryService` maps those records to domain +models. + +Dynamic SQL must be restricted to trusted identifiers or fixed fragments. +User-provided values always use database parameters. Repositories must not accept +raw SQL from routers or services. + +### Database + +PostgreSQL is the authoritative persistent store for MinerOps. Application rules +that must hold for every writer are also represented as database constraints +where practical, including uniqueness, allowed values, and referential +integrity. + +Alembic is the authority for new persistent schema changes. Migrations must +state how existing data is handled and provide a downgrade when it can be done +without ambiguity or data loss. + +**Current implementation:** Alembic migrations cover the foundational, +scanning, audit, identity, role, and Inventory schemas. Some legacy code in +`main.py` still creates or alters tables at runtime for compatibility. Do not add +new runtime DDL. Move persistent schema changes through reviewed Alembic +migrations. + +The repository also contains a SQLAlchemy engine and session factory, while +most current operational persistence and the Inventory repository use direct +`psycopg`; some existing agent paths use `asyncpg`. A domain must choose and +encapsulate its persistence mechanism inside its repository rather than expose +driver details to routers. + +## Domains + +A domain package groups one operational capability behind an explicit public +interface. Inventory is the current reference domain: + +```text +backend/app/inventory/ + __init__.py + exceptions.py + models.py + repository.py + service.py +``` + +- `models.py` defines typed request, update, search, summary, detail, and tag + contracts. +- `exceptions.py` defines errors meaningful to Inventory callers. +- `service.py` defines Inventory rules and use cases. +- `repository.py` defines Inventory persistence. +- `__init__.py` exposes the package's intended public surface. + +Inventory's internal asset identity is the miner UUID. Product-facing Asset IDs +are normalized and checked independently of mutable network fields. The domain +maps product terminology such as `asset_id` and `psu_serial` to existing database +columns at its service and persistence boundary. + +Future domain packages should follow the same separation when it fits their use +cases. Identity, Repairs, Monitoring, Operations, Scheduling, and Locations are +examples of possible domain boundaries, not a fixed package list. A domain +boundary must be based on ownership of rules and data, not on menu pages or +route prefixes alone. + +A domain may call another domain only through an explicit service contract. It +must not import another domain's repository to update tables it does not own. +Shared read requirements should use a clearly owned query service or read model +rather than duplicating cross-domain SQL. + +## Dependency Rules + +Dependencies flow downward: + +```text +Routers + | + v +Services + | + v +Repositories + | + v +Database +``` + +Allowed dependencies: + +- The application entry point may include routers and install application-wide + middleware. +- Routers may depend on domain models, services, and exceptions. +- Services may depend on their domain's models, exceptions, and explicit + repository contracts. +- Repositories may depend on configuration, database drivers, and + persistence-specific utilities. +- Tests may replace repositories or connection factories at defined dependency + boundaries. + +Examples of allowed flow: + +- An Inventory router validates a UUID, calls `InventoryService.get_miner`, and + translates `MinerNotFound` into an HTTP 404 response. +- `InventoryService.create_miner` normalizes an Asset ID and coordinates miner + and tag writes through an `InventoryRepository` transaction. +- `InventoryRepository.search_miners` builds parameterized filters and executes + them against PostgreSQL. + +Prohibited dependencies: + +- Repositories importing services, routers, or `app.main`. +- Services importing FastAPI request or response classes. +- Routers opening database connections or issuing SQL. +- Domain code importing globals from `app.main`. +- A lower layer choosing permissions or HTTP status codes. +- A domain bypassing another domain's service to write its tables. +- Circular imports between domains or layers. + +Examples of prohibited flow: + +- A route updating `miners` directly after checking request JSON. +- A service returning `JSONResponse` or raising `HTTPException`. +- A repository checking whether the current user is an administrator. +- A Repairs service importing `InventoryRepository` to mutate miner ownership + without an Inventory service contract. + +Dependency direction is an architectural constraint, not a naming convention. +Putting SQL in a file named `service.py` does not make it service logic. + +## Database Philosophy + +- **Alembic-managed schema:** New persistent tables, columns, constraints, and + indexes are introduced through versioned migrations. +- **Explicit migrations:** Every migration explains existing-data behavior, + upgrade ordering, and downgrade behavior. Migrations do not silently merge, + rename, or discard operational records. +- **Stable identifiers:** Internal identity uses immutable identifiers rather + than mutable IP addresses, MAC addresses, names, or locations. +- **Repository ownership:** Each domain has one clear persistence boundary for + its owned records. Other layers do not duplicate its SQL. +- **Database enforcement:** PostgreSQL constraints protect invariants that must + hold independently of application code. +- **Transactional consistency:** Operations that must succeed or fail together + execute in one explicit transaction. +- **Data preservation:** Schema evolution preserves existing records unless a + separately reviewed product decision explicitly requires otherwise. + +Migration tests should verify upgrade, downgrade, constraints, and existing-data +behavior. Runtime schema changes are legacy compatibility behavior and are not +permitted for new work. + +## API Philosophy + +MinerOps APIs use resource-oriented HTTP semantics. New or replaced domain APIs +follow these conventions: + +- Use nouns for resources and standard HTTP methods for operations. +- Validate path, query, and body structure at the router boundary. +- Apply business validation in the service. +- Use Pydantic models for request and response contracts. +- Return predictable status codes and response shapes. +- Translate domain exceptions into consistent, non-leaking API errors. +- Bound collection endpoints with pagination. +- Make supported filtering, searching, and ordering explicit. +- Preserve stable identifiers in URLs and response contracts. +- Enforce authentication and permissions on the server. +- Avoid duplicate endpoints for the same use case. + +**Current implementation:** Many routes return the common `data`, `meta`, and +`error` envelope defined in `main.py`. Pydantic validation, pagination, and +error shapes are not uniform across all existing routes. Identity supplies +current authentication and role-aware dependencies. New endpoints should follow +the conventions above while preserving compatibility for any route they +replace. + +Pagination metadata should be sufficient for callers to understand the returned +window. Filters must be parameterized in repositories, not interpolated in +routers. Database exceptions and internal details must not be exposed to API +clients. + +## Testing Philosophy + +Tests follow the same boundaries as production code. + +- **Unit tests:** Exercise service rules with substituted dependencies. They + should not require FastAPI or a database when the behavior is purely domain + logic. +- **Repository tests:** Exercise real queries, constraints, associations, and + transactions against a disposable PostgreSQL database. +- **Migration tests:** Verify schema upgrade, downgrade, constraint behavior, + and existing-data preservation. +- **Integration tests:** Exercise application startup, migrations, + authentication, authorization, safety behavior, and cross-layer workflows in + a disposable Compose stack. +- **Architecture tests:** Protect dependency rules that can be checked + mechanically, such as the Inventory package containing no FastAPI imports or + router registration. + +Current Inventory tests use a fake repository for service rules and PostgreSQL +for repository behavior. The focused runner migrates a disposable database +through Alembic before executing the suite. The development baseline starts an +isolated Compose project and checks schema, login gating, roles, and safety +interlocks. + +Before committing, run validation appropriate to the change. This commonly +includes Python compilation, shell syntax checks, focused unit or repository +tests, migration tests for schema work, the disposable development baseline for +cross-application changes, and `git diff --check`. + +Keep commits small and reviewable. A commit should express one coherent change, +include focused tests when behavior changes, and avoid unrelated formatting or +refactoring. Broad integration coverage does not replace a focused test for a +new rule. + +## Future Evolution + +MinerOps will evolve incrementally as existing responsibilities are moved into +clear domain boundaries. New infrastructure or domain packages must preserve +the router-to-service-to-repository dependency direction, explicit ownership, +and testability rules in this document. Architectural changes should clarify +those rules rather than bypass them. + +## When In Doubt + +When a feature does not fit naturally into an existing module, refine the +architecture instead of weakening architectural boundaries. Clear boundaries +are easier to maintain than clever shortcuts.