Skip to content

agents.md

Latest commit

 

History

History
214 lines (180 loc) · 15 KB

agents.md

File metadata and controls

214 lines (180 loc) · 15 KB
Raw

Shard Core

The software running on each individual Freeshard. Manages installed apps, terminal pairing, peer communication, backups, and serves as the control plane for a user's personal cloud computer.

Tech Stack

  • Framework: FastAPI with uvicorn, Python 3.13+
  • Database: PostgreSQL with psycopg3 (async) + yoyo-migrations
  • Container Orchestration: Docker / Docker Compose (subprocess calls)
  • Reverse Proxy: Traefik v3 (configured dynamically per app)
  • Config: Pydantic v2 BaseSettings, loaded from config.toml + local_config.toml, env prefix FREESHARD_
  • Package Management: uv
  • Formatting/Linting: ruff + black
  • Testing: pytest + pytest-asyncio + pytest-docker

Architecture

shard_core/
  web/                → FastAPI routers, organized by auth level
    public/             No auth required (meta, health, pairing)
    protected/          JWT auth from paired terminal (apps, settings, backup, etc.)
    internal/           Same-machine auth via signatures (app→backend, app→peer calls)
    management/         Management API auth (hosted shards only)
  service/            → Business logic
    app_installation/   App install/uninstall/reinstall + background worker queue
    app_lifecycle.py    Two-tier idle control: RUNNING -> PAUSED -> STOPPED, PSI-driven LRU demotion, wake-on-request
    app_tools.py        Docker CLI wrappers (pause/unpause/stop/down); start_app is the idempotent revive primitive that decides unpause vs up from real container state, and the teardown helpers decide their unpause the same way
    memory_pressure.py  PSI parsing (/host/pressure/memory), cgroup v2 memory.reclaim page-out
    pause_metrics.py    In-memory pause-tier telemetry accumulators (transitions, latencies, PSI snapshots)
    pairing.py          Terminal pairing (JWT creation, code generation)
    owner_email.py      Owner address: candidate, confirmation token, promotion
    backup.py           Azure Blob Storage backup via rclone
    peer.py             Peer shard management
    crypto.py           RSA-4096 key generation, signing, verification (PSS padding)
    portal_controller.py  API calls to management portal
    freeshard_controller.py  API calls to controller
    telemetry.py        Usage metrics reporting
    disk.py             Disk space monitoring
    migration.py        Database schema migrations
    websocket.py        WebSocket connection lifecycle
  database/           → PostgreSQL access layer (per-entity modules, conn-first-arg pattern)
  data_model/         → Pydantic v2 models
    app_meta.py         App metadata, Status enum, VMSize enum
    identity.py         Shard identity (keys, domain, short_id) — the shard's
                        public profile, published unauthenticated; holds no address
    user.py             People on the shard (owner, later members) and their address
    terminal.py         Paired device models
    peer.py             Peer shard models
    backend/            Models copied from freeshard-controller (via `just get-types`)
  util/               → Shared utilities
    async_util.py       BackgroundTask, PeriodicTask, CronTask
    signals.py          Blinker signal definitions
    subprocess.py       Async subprocess runner; app_compose_command() pins every app
                        compose call to <app_dir>/docker-compose.yml and project <app>
                        — never rely on cwd, compose walks up to the core stack

Commands

just run-dev          # FastAPI dev server on port 8080, loads local_config.toml
just cleanup          # ruff check + black formatting
just get-types        # Sync data models from freeshard-controller repo

Key Patterns

Database Access

PostgreSQL via psycopg3 async. All DB functions take conn: AsyncConnection as first arg. Callers acquire connections via async with db_conn() as conn:. Schema managed by yoyo-migrations in migrations/, named shard-core-NNNN-<slug>.sql with a -- depends: <previous> header chaining each migration to the one before it.

async with db_conn() as conn:
    app = await db_installed_apps.get_by_name(conn, "myapp")
    await db_installed_apps.update_status(conn, "myapp", Status.RUNNING)

Tables: identities, users, terminals, installed_apps, peers, backups, tours, app_usage_tracks, kv_store, oidc_clients, oidc_codes, oidc_tokens, app_secrets.

Postgres data is not part of the rclone backup set (which only syncs core//user_data/). To keep it, database/db_snapshot.py dumps all application tables to core/db_snapshot.json before each backup, and init_database() restores it on a fresh shard (before the default identity is generated, so the restored identity survives). Pre-0.38 backups are restored from TinyDB by tinydb_migration.py instead.

Authentication (4 levels)

  • /public/* — No auth
  • /protected/* — JWT in authorization cookie (from terminal pairing)
  • /internal/* — HTTP Message Signature verification, RSA_PSS_SHA512 (for app-to-shard and peer-to-shard calls)
  • /management/* — Management API auth (hosted shards only)

Background Tasks

Started at app lifespan startup, stopped at shutdown:

  • InstallationWorker — async task queue for app install/uninstall
  • PeriodicTask(control_apps, 30s) — app idle lifecycle. With apps.lifecycle.pause_enabled (default off): RUNNING pauses after idle_for_pause (cgroup freeze + page-out to swap), PAUSED stops after idle_for_stop, and high memory PSI demotes the LRU app one tier per cycle. Flag off: legacy stop-only
  • PeriodicTask(update_disk_space, 30s) — disk monitoring
  • CronTask(start_backup, "0 3 * * *") — daily backup with random delay
  • CronTask(docker_prune_images, daily) — image cleanup
  • Various telemetry and peer key refresh tasks

The Owner's Email Address

users.email is the single home for a person's address and is verified by definition; users.pending_email is an unverified candidate, at most one in flight per user. That invariant is the only reason the OIDC provider may assert email_verified: true — with no verified address it emits neither claim, and there is deliberately no synthetic fallback.

  • service/owner_email.py owns every transition. Anything that writes pending_email must retire the token with it, or the token promotes an address it was never sent to.
  • On confirmation, in this order: notify the old address, promote, mirror to shards.owner_email, notify the new one. The first step must precede the mirror — the controller's relay only ever reaches the address it currently has on file.
  • POST /public/users/confirm-email is unauthenticated by design and the token is its only credential. There is no GET: mail scanners and link prefetchers would burn a single-use token.
  • email.enabled (default true) says whether the shard can send mail at all, and is the explicit self-hosted signal — no controller means no mail, so the address is set directly. Never infer it from a failed delivery; a controller outage would silently downgrade a security control.

Signals (Event System)

Blinker-based async signals defined in util/signals.py. DB-writing handlers are async and called via await signal.send_async():

  • on_apps_update — app state changed
  • on_request_to_app — user accessing app (triggers auto-start)
  • on_terminal_auth — terminal authenticated
  • async_on_first_terminal_add — first pairing complete
  • async_on_peer_write — peer added/updated

App Installation Flow

  1. Custom-app uploads (POST /protected/apps) are validated by app_installation/app_zip.py before any state is created: the zip is buffered to a temp dir, must carry app_meta.json + docker-compose.yml.template at its root (a single top-level directory is stripped), and the app name comes from the manifest, not the filename. Invalid uploads get a 400 and leave no row, no dir, no task. Extraction goes through extract_app_zip, which refuses members resolving outside the app dir.
  2. Request queued → InstallationWorker picks it up
  3. Docker Compose template rendered with Jinja2 (shard domain, data paths, per-app secrets, etc.)
  4. Traefik dynamic config generated for the app's subdomain routing
  5. docker-compose up -d via subprocess
  6. Status updated in PostgreSQL

Reinstall tears the old stack down with docker_shutdown_app(force=True): by then the status is REINSTALLING, which no teardown gate accepts, so without the force flag the old containers survive the rmtree of the app dir and compose up --no-start collides with them (issue #199). Do not widen the gates to admit REINSTALLING instead — the same allow-lists gate the idle control tick, which must not touch an app mid-reinstall. The other half of that fix is in app_tools: the unpause both teardown helpers do first is decided from the real container state, not the stored status, which is what lets a reinstall repair a shard already stuck in ERROR with frozen containers. Reinstall then checks the containers are actually gone before it deletes the app directory — a half-failed teardown must leave the old install in place rather than destroy it.

The task queue is in-memory only, so a restart loses whatever it held. Uninstall is made crash-safe by reconcile_interrupted_uninstalls(), a lifespan step that re-enqueues an uninstall for every row still in UNINSTALLATION_QUEUED or UNINSTALLING. _uninstall_app asserts no status and tolerates missing files, so resuming it is idempotent — the row is the tombstone. The step only enqueues; the worker starts later in the lifespan, and awaiting task completion before then deadlocks. Install is made crash-safe the same way by reconcile_interrupted_installs() (run right before write_traefik_dyn_config so a row it resets to a non-routable status is excluded from the boot's Traefik config): it re-enqueues an interrupted install when its source still exists (zip on disk → "install from zip", otherwise a store/config/unknown install → re-download), resetting the status to INSTALLATION_QUEUED first so the worker's status assertion passes, and marks the row ERROR when the source is gone (a custom upload whose zip is lost). Reinstall is settled to ERROR rather than resumed: _reinstall_app deletes the app's files before re-downloading, so re-running it unattended could destroy a working install if the store is briefly unreachable at boot — the user retries it manually.

Apps in NOT_ROUTABLE_STATUS (app_installation/util.py) get no Traefik router: INSTALLATION_QUEUED, ERROR, UNINSTALLATION_QUEUED, UNINSTALLING. Their files are either not there yet or already being removed. For every other status write_traefik_dyn_config loads the app's metadata in a try/except and skips the app with a warning if it is missing — so a half-installed row can no longer raise MetadataNotFound out of the lifespan and boot-loop core.

App compose template context

render_docker_compose_template (app_installation/util.py) exposes four things to an app's docker-compose.yml.template: fs.* (host data paths), portal (the shard identity), oidc (client credentials, None unless the app declares an oidc block in its app_meta.json and the provider is enabled on the shard), and secret('<name>'). The secret helper returns an app-scoped secret, generating one (32 chars, [A-Za-z0-9]) on first reference and persisting it in the app_secrets table (database/app_secrets.py); the name is arbitrary and declared purely by referencing it. The resolver mints missing secrets synchronously during the single jinja pass (jinja is sync, the DB is not) and they are persisted afterwards. A secret is stable across re-renders, upgrades and restarts (it is reused, never rotated) and rides along in the DB snapshot backup. Secrets are kept on uninstall (so a reinstall of the same app reuses them, matching the app's retained user_data), never rotated, and never encrypted at rest. App-scoping is NOT a boundary against a hostile app author: the compose template is rendered with a non-sandboxed jinja environment (as portal/fs already are), so a malicious template can escape and read any app's secrets. Closing that gap would need a SandboxedEnvironment; it is out of scope here.

Async Fire-and-Forget

Long operations use asyncio.create_task() with done callbacks. No thread pools.

Service-to-Controller Communication

freeshard_controller.py and portal_controller.py make HTTP calls to the central platform. Requests are signed with the shard's RSA key via signed_call.py, using HTTP Message Signatures with RSA_PSS_SHA512.

File Path Conventions

Inside Docker container:

  • PostgreSQL database (configured via [db] section in config.toml)
  • /core/installed_apps/{name}/ — per-app Docker Compose files
  • /core/traefik.yml — Traefik static config
  • /user_data/app_data/{name}/ — per-app persistent data
  • /user_data/shared/ — shared data across apps

Host-side: settings().path_root_host (typically /home/shard).

Pydantic Conventions

  • BaseModel for data classes, BaseSettings for configuration
  • @field_validator for single-field, @model_validator(mode="after") for cross-field
  • @computed_field @property for derived values (e.g., Identity.domain, Identity.public_key)
  • Models in data_model/backend/ are copied from freeshard-controller — do not edit directly, use just get-types
    • The CI drift-check job re-runs get-types against controller origin/main and fails if the diff is non-empty. To fix drift: check out the controller at origin/main (a local main may be many commits behind — git fetch first), then run just SOURCE_DIR=<abs path to controller>/freeshard-controller-backend/freeshard_controller/data_model get-types from this repo and commit only the regenerated files. Never hand-edit them.

Commits

Scoped Commits: <scope>: <description>. The scope is the area of the tree the change touches, never a change type — write service: retry app install on transient docker errors, not fix(service): .... Body and trailers are optional; a change's reasoning belongs in the body, not in a code comment.

Scopes for this repo: web service data_model database migrations tests scripts ci docs meta

meta covers repo-level files (agents.md, README, justfile). For a change spanning several scopes, use a broader one, list two comma-separated, or use treewide. Merges, reverts and generated commits (set version to <v>) keep their own format. Don't generate a changelog from the commit log — release notes come from merged PRs.

Important Notes

  • The Status enum in app_meta.py covers the full app lifecycle: UNKNOWN, INSTALLATION_QUEUED, INSTALLING, STOPPED, RUNNING, DOWN, etc.
  • VMSize enum (XS, S, M, L, XL) has comparison operators — used to check if a shard is large enough for an app.
  • Docker socket is mounted read-write (/var/run/docker.sock) — shard_core manages containers directly.
  • Traefik routes app traffic via subdomains: <app-name>.<shard-domain>.
  • The project was formerly called "Portal" — some references may still use the old name.