Hofvarpnir

In Norse mythology, Hofvarpnir ("hoof-thrower") is the horse of the goddess Gná, who rides through sky and sea to fetch things from distant realms for Frigg.

A self-hosted video archival system that downloads videos from YouTube (and other platforms) via yt-dlp.

Features

• Multi-platform support: YouTube and other platforms via yt-dlp auto-detection
• Web UI: Modern web interface built with htmx and Tailwind CSS
• Terminal UI: Standalone terminal client for managing downloads and monitoring live progress
• Automatic scheduling: Per-source indexing frequency
• Resilient indexing: Age-restricted, private, and unavailable videos are skipped and the scan continues, so a single problem entry never aborts indexing of the rest of a channel/playlist
• Health monitoring: Surface sources that are enabled but persistently failing to index
• Quality presets: Configurable download quality
• Retention policies: Automatic cleanup with per-source and per-profile settings
• Deduplication: Videos downloaded once regardless of multiple source references
• Real-time progress: SSE-based live download progress in both web and TUI
• Observability: OpenTelemetry traces (Tempo), log shipping (Loki), Prometheus metrics, and Grafana dashboards
• Dark Mode: Tailwindcss darkmode
• OIDC Authentication: Single sign-on via OpenID Connect (Keycloak, Auth0, Azure AD, Google, Okta, Pocket-ID, etc.)

Planned

• Keyboard Shortcuts: Vim motions

Tech Stack

• Runtime: Tokio
• HTTP: Axum
• Actors: Kameo
• Templating: Maud + htmx
• Styling: Tailwind CSS 4
• Database: PostgreSQL 17 (via SQLx)
• API docs: OpenAPI (utoipa) + Scalar
• TUI: Ratatui

Quick Start

# Build all crates
cargo build --release

# Start the server (API + Web UI; runs pending migrations on startup)
cargo run --bin hof-server

# Run the TUI client (in another terminal)
cargo run --bin hofvarpnir-tui

Install the TUI

The TUI connects to a running Hofvarpnir server; on first launch, the setup screen lets you enter the API URL and API key. You can create an API key in the web UI under Settings.

Install script (Linux)

curl -fsSL https://raw.githubusercontent.com/Mozart409/hofvarpnir/main/scripts/install-tui.sh | bash

The script detects your platform (x86_64/aarch64; glibc or musl on Linux), downloads the latest release, verifies its SHA-256 checksum, and installs to ~/.local/bin. Environment overrides: HOFVARPNIR_VERSION=0.15.0 pins a specific version, HOFVARPNIR_INSTALL_DIR changes the install destination.

Manual install

Download hofvarpnir-tui-<target>-v<version>.tar.gz for your platform from the latest release. The archive contains a single hofvarpnir-tui binary:

tar -xzf hofvarpnir-tui-*.tar.gz
install -m 755 hofvarpnir-tui ~/.local/bin/

Available targets: x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu, x86_64-unknown-linux-musl and aarch64-unknown-linux-musl (static, for Alpine and old-glibc systems).

The server tarball keeps the plain hofvarpnir-<target>-v<version>.tar.gz name and contains only the server binary — most server deployments should use the container image instead.

Make sure ~/.local/bin is on your PATH, then start it with:

hofvarpnir-tui

Nix

With Nix flakes enabled, install the TUI into your user profile:

nix profile install github:Mozart409/hofvarpnir#hofvarpnir-tui

Then run hofvarpnir-tui. The flake builds the standalone TUI package for x86_64 Linux and aarch64 Linux.

Configure the TUI

To skip the setup screen, put the server URL and API key in $XDG_CONFIG_HOME/hofvarpnir/tui.toml (normally ~/.config/hofvarpnir/tui.toml; override with --config <PATH> or HOF_TUI_CONFIG). On first launch, press Ctrl-S on the setup screen to have the TUI write that file for you.

api_url = "https://hof.example.com"

# Exactly one token source (or none, to be asked on start):
token = "hof_sk_..."                         # file must be chmod 600
# token_file = "/run/agenix/hofvarpnir-tui"  # read and trimmed
# token_command = "pass show hofvarpnir"     # run via `sh -c`; stdout is the key

--api-url / --token win over HOF_API_URL / HOF_API_TOKEN, which win over the file. The TUI refuses an inline token in a file other users can read. With token_file or token_command the file holds no secret, so it can live somewhere read-only and world-readable, such as the Nix store (see below).

token_command runs before the TUI takes over the terminal, so commands that prompt (gpg pinentry, op read) work. A trailing newline from the file or command is trimmed.

NixOS: API key from agenix or sops-nix

Keep the key in an encrypted secret and let home-manager generate a tui.toml that only points at the decrypted file. Put just the hof_sk_... key in the secret (e.g. agenix -e secrets/hofvarpnir-tui.age).

agenix, NixOS module. The secret is decrypted to /run/agenix/<name>, owned by root with mode 0400 by default, so set owner or the TUI cannot read it:

# configuration.nix
age.secrets.hofvarpnir-tui = {
  file = ./secrets/hofvarpnir-tui.age;
  owner = "alice";
};

# home-manager, used as a NixOS module (provides `osConfig`)
xdg.configFile."hofvarpnir/tui.toml".source = (pkgs.formats.toml { }).generate "tui.toml" {
  api_url = "https://hof.example.com";
  token_file = osConfig.age.secrets.hofvarpnir-tui.path;
};

agenix, home-manager module. The secret is decrypted to $XDG_RUNTIME_DIR/agenix/<name> and owned by you, but on Linux its .path is the literal string ${XDG_RUNTIME_DIR}/agenix/<name>, expanded only at runtime. token_file takes paths literally, so use token_command, which runs through sh and expands the variable:

age.secrets.hofvarpnir-tui.file = ./secrets/hofvarpnir-tui.age;

xdg.configFile."hofvarpnir/tui.toml".source = (pkgs.formats.toml { }).generate "tui.toml" {
  api_url = "https://hof.example.com";
  token_command = "cat \"${config.age.secrets.hofvarpnir-tui.path}\"";
};

sops-nix works the same way through sops.secrets:

• NixOS module: decrypted to /run/secrets/<name>, root-owned 0400 by default. Set sops.secrets.hofvarpnir-tui.owner = "alice"; and use token_file = osConfig.sops.secrets.hofvarpnir-tui.path;.
• home-manager module: decrypted to ~/.config/sops-nix/secrets/<name> by default, a real path, so token_file = config.sops.secrets.hofvarpnir-tui.path; works. If you move it with sops.defaultSymlinkPath = "%r/secrets"; (e.g. because ~/.config is read-only), the path contains a literal %r; use token_command = "cat \"$XDG_RUNTIME_DIR/secrets/hofvarpnir-tui\""; instead.

After a rebuild, hofvarpnir-tui should open the main view directly. If it reports a token_file / token_command error, run the same cat yourself to check the secret's path and permissions.

Project Structure

crates/
├── hof-core/    Domain types, actors, database, yt-dlp process wrapper
├── hof-api/     Axum REST API + OpenAPI + SSE (JSON) + Scalar docs
├── hof-web/     Maud + htmx frontend routes + SSE (HTML partials)
└── hof-tui/     Ratatui TUI client (consumes hof-api over HTTP)

API

The REST API is documented with OpenAPI and available via Scalar at /docs when running the server.

Key endpoints:

• GET /api/v1/profiles - Manage download profiles
• GET /api/v1/sources - Manage video sources (channels, playlists)
• GET /api/v1/downloads - List and manage downloads
• GET /api/v1/downloads/progress - SSE stream for live progress (JSON)
• GET /web/v1/downloads/progress - SSE stream for live progress (HTML)
• GET /api/v1/activity - System activity log (indexing, downloads, errors)
• GET /api/v1/activity/unhealthy-sources - Sources persistently failing to index (configurable min_errors threshold)

Indexing resilience

When indexing a channel or playlist, a video that cannot be fetched (age-restricted, private, removed, or otherwise unavailable) is skipped, and indexing continues with the remaining entries. Age-restriction in particular is treated as a permanent, per-video condition — distinct from rate limiting, which pauses indexing of the source. This prevents a single age-restricted video from silently blocking discovery of every other video in the source.

Use GET /api/v1/activity/unhealthy-sources to find sources that are enabled but stuck on errors (e.g. repeated rate limiting). It reports the consecutive-error streak since each source's last successful index.

Configuration

Configuration is loaded from environment variables. For local development they live encrypted in .sops.env — a sops dotenv store (variable names visible, values ENC[...], committed) that just dev / bacon serve decrypt into the server process only; edit it with sops .sops.env. .env.example documents every variable, and the binary still loads a plain .env via dotenvy if one exists (real environment variables take precedence).

Variable	Description	Default
DATABASE_URL	PostgreSQL connection string	-
HOST	Server bind address	127.0.0.1
PORT	Server port	3000
YTDLP_PATH	Path to yt-dlp binary	yt-dlp
MAX_CONCURRENT_DOWNLOADS¹	Max simultaneous downloads	3
DOWNLOAD_TIMEOUT_HOURS	Per-download timeout (hours)	4
MAX_DOWNLOAD_ATTEMPTS	Retries before permanently-failed	5
QUALITY_WAIT_HOURS	Hours a new video may wait for higher resolutions (0 = off; per-profile override)	24
RATE_LIMIT_DELAY_SECS¹	Delay between yt-dlp invocations	5
DEFAULT_OUTPUT_DIR	Default download output directory	/var/lib/hofvarpnir/downloads
RETENTION_DAYS	Global retention policy, in days	- (no cleanup unless set)
MAX_INDEXERS_PER_TICK¹	Max sources indexed per scheduler tick	5
CHECK_INTERVAL_SECS¹	Scheduler tick interval, in seconds	60
CLEANUP_INTERVAL_SECS¹	Cleanup actor run interval, in seconds	10800 (3h)
DRAIN_TIMEOUT_SECS¹	Max time to wait for graceful shutdown	1800 (30m)
OTEL_EXPORTER_OTLP_ENDPOINT	OTLP gRPC endpoint for traces	- (disabled)
OTEL_SERVICE_NAME	Service name for traces/logs	hofvarpnir
LOKI_URL	Grafana Loki endpoint for log shipping	- (disabled)
METRICS_ENABLED	Enable Prometheus metrics endpoint	false
LOG_FORMAT	Log output format (json or default)	default

¹ Runtime-tunable. These six values can be changed at runtime via PATCH /api/v1/system/settings, without a restart. The /settings/runtime control panel displays them with their resolved provenance; editing them is currently API-only. A value stored in the database overrides the environment variable, which is itself only the fallback used when the database column is NULL — the environment variable does not "win" once a database value exists. The control panel shows a default / env / database badge next to each value indicating which layer it actually resolved from. Every other variable above is read once at startup with no runtime override. See Runtime control below.

Web session

The web UI session is a tower-sessions cookie backed by a row in the sessions table. Its lifetime is a sliding window — the session is saved and its expiry recomputed on every request — sized by:

Variable	Description	Default
SESSION_TIMEOUT_HOURS	Sliding web-session lifetime, hours	- (browser session cookie)

When SESSION_TIMEOUT_HOURS is unset — or set to 0 or a non-integer — the tower-sessions default applies: a browser-session cookie (no Max-Age, so closing the browser logs you out) whose server-side row lives two weeks past its last write. A positive value makes the cookie persistent (Max-Age = SESSION_TIMEOUT_HOURS) and extends the server-side row to match, so an active session is refreshed on every request and an idle one lapses after that many hours. The price is one write to the sessions table per request from a logged-in browser.

OIDC Authentication (Optional)

To enable OIDC single sign-on, set these environment variables:

Variable	Description	Default
OIDC_ISSUER	OIDC provider issuer URL (required to enable OIDC)	- (disabled)
OIDC_CLIENT_ID	OAuth2 client ID from your OIDC provider	-
OIDC_CLIENT_SECRET	OAuth2 client secret from your OIDC provider	-
OIDC_SCOPES	Comma-separated scopes to request	openid,profile,email
OIDC_AUTO_PROVISION	Auto-create users on first OIDC login	true
OIDC_REDIRECT_BASE_URL	Override base URL for callback (e.g., https://hof.example.com)	- (derived from request)
OIDC_LOGOUT_REDIRECT	Enable RP-initiated logout	false
OIDC_DISCOVERY_TIMEOUT	Discovery HTTP timeout in seconds	30

Runtime control

The control panel at /settings/runtime lets an operator pause indexing and/or downloads, trigger a graceful shutdown, and see every runtime value currently in force — each tunable alongside a default / env / database badge showing which layer it resolved from, plus every active timeout and countdown.

Changing the six tunable values is API-only today — PATCH /api/v1/system/settings. The panel renders them read-only; pause and shutdown are the two actions it can perform. The full endpoint set lives under /api/v1/system/ (settings, pause, shutdown, status).

• Pause is per-module: indexing and downloads are gated independently, so you can pause one without stopping the other. Choose a duration — 1h, 6h, 12h, 24h, 3d, 7d, or indefinite — and it auto-resumes on its own when the duration elapses (indefinite pauses require an explicit resume). Pause is a database column, so it survives a restart.
• Drain deliberately does not survive a restart. Draining is process-local by design: a restarted container must never come back up silently refusing all work with no visible cause. See ADR-0004.

The "Shut down" button and restart: always

Draining is a graceful shutdown, not a crash: the process stops accepting new work, finishes in-flight downloads and indexing (or gives up once DRAIN_TIMEOUT_SECS elapses), and then returns from main with exit code 0. Under a container restart policy of always or unless-stopped, the container runtime treats exit code 0 as a normal stop and starts the container straight back up — so from the operator's point of view, the "Shut down" button appears to do nothing.

This is the one place the control panel cannot do what its label implies, and the fix is deployment-side, not application-side: a deployment that wants the process to actually stay down after using this button must run with restart: on-failure instead.

Development

See GOALS.md for detailed architecture and implementation roadmap.

CI/CD

This project uses GitHub Actions for continuous integration and deployment:

• .github/workflows/ci.yml - Runs on pull requests and pushes to main branch
• .github/workflows/release.yml - Creates GitHub releases when pushing version tags (v*.*.*)

License

GPL-3.0-or-later see LICENSE