diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index b64c3e60..8adeecdc 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,75 +1,16 @@ # html2rss-web AI Agent Instructions -## Overview +This repository uses a centralized documentation structure. All AI agents (including Copilot, Gemini CLI, etc.) must follow the constraints and rules defined in the following files: -- Ruby web app that converts websites into RSS 2.0 feeds. -- Built with **Roda** backend + **Preact** frontend, using the **html2rss** gem (+ `html2rss-configs`). -- **Frontend:** Vite-built Preact UI, served alongside Ruby backend. +## Canonical Documentation -## Documentation website of core dependencies +- **Agent Constraints**: [AGENTS.md](../AGENTS.md) (Execution rules, verification, and UI principles) +- **Contributor Guide**: [docs/README.md](../docs/README.md) (Architecture, security, setup, and coding style) +- **Design System**: [docs/design-system.md](../docs/design-system.md) (Visual and CSS rules) -Search these pages before using them. Find examples, plugins, UI components, and configuration options. +## Quick Reference for Agents -### Roda - -1. https://roda.jeremyevans.net/documentation.html - -### Preact & Vite - -1. https://preactjs.com/guide/v10/getting-started/ -2. https://vite.dev/guide/ - -### html2rss - -1. If available, find source locally in: `../html2rss`. -2. source code on github: https://github.com/html2rss/html2rss - -### Test and Linters - -1. https://docs.rubocop.org/rubocop/cops.html -2. https://docs.rubocop.org/rubocop-rspec/cops_rspec.html -3. https://rspec.info/features/3-13/rspec-expectations/built-in-matchers/ -4. https://www.betterspecs.org/ - -Fix rubocop `RSpec/MultipleExpectations` adding rspec tag `:aggregate_failures`. - -## Core Rules - -- ✅ Organise Roda routes via dedicated modules (e.g. `Html2rss::Web::Routes::*`), keeping the main app class thin. -- ✅ Keep helper modules minimal: define entrypoints with `class << self` and push implementation helpers under `private`; avoid `module_function` unless mirroring existing conventions. -- ✅ Validate all inputs. Pass outbound requests through **SSRF filter**. -- ✅ Add caching headers where appropriate (`Rack::Cache`). -- ✅ Errors: friendly messages for users, detailed logging internally. -- ✅ **Frontend**: Use Preact components in `frontend/src/`. Keep it simple. -- ✅ **CSS**: Use the app-owned frontend styles in `frontend/src/styles/`. -- ✅ **Specs**: RSpec for Ruby, build tests for frontend. -- ✅ When a spec needs to tweak environment variables, wrap the example in `ClimateControl.modify` so state is restored automatically. - -## Don't - -- ❌ Don't use Ruby's URI class or addressable gem directly. Strictly use `Html2rss::Url` only. -- ❌ Don't bypass SSRF filter or weaken CSP. -- ❌ Don't add databases, ORMs, or background jobs. -- ❌ Don't leak stack traces or secrets in responses. -- ❌ Don't reach into private API with `send(...)`; expose what you need at the module level instead. -- ❌ Don't modify `frontend/dist/` - it's generated by build process. -- ❌ NEVER expose the auth token a user provides. - -## Environment - -- `RACK_ENV` – environment -- `AUTO_SOURCE_ENABLED`, `AUTO_SOURCE_USERNAME`, `AUTO_SOURCE_PASSWORD`, `AUTO_SOURCE_ALLOWED_ORIGINS` -- `HEALTH_CHECK_USERNAME`, `HEALTH_CHECK_PASSWORD` -- `SENTRY_DSN` (optional) - -### Verification Steps - -- Run `ruby -c app.rb` to check syntax -- Run `bundle exec rspec` to verify tests -- Check `bundle install` removes unused dependencies - -## Style - -- Add `# frozen_string_literal: true` -- Follow RuboCop style -- YARD doc comments for public methods +- **Environment**: All commands MUST run inside the Dev Container. +- **Verification**: Run `make ready` before any commit. +- **Security**: Follow strict [Security & Safety Rules](../docs/README.md#security--safety-rules). +- **Style**: Follow [Architectural Constraints](../docs/README.md#architectural-constraints) (YARD docs, Roda organization). diff --git a/AGENTS.md b/AGENTS.md index b7999f0b..5dae9788 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,84 +1,51 @@ -# Agent Workflow (Dev Container) +# Agent Workflow Constraints -## Start the Dev Container - -```text -docker compose -f .devcontainer/docker-compose.yml up -d -``` - -## Commands (run inside the container) - -```text -docker compose -f .devcontainer/docker-compose.yml exec -T app make setup - -docker compose -f .devcontainer/docker-compose.yml exec -T app make dev - -docker compose -f .devcontainer/docker-compose.yml exec -T app make test - -docker compose -f .devcontainer/docker-compose.yml exec -T app make ready - -docker compose -f .devcontainer/docker-compose.yml exec -T app bundle exec rubocop -F - -docker compose -f .devcontainer/docker-compose.yml exec -T app bundle exec rspec -``` - -Pre-commit gate (required): - -```text -docker compose -f .devcontainer/docker-compose.yml exec -T app make ready -``` - -If you need an interactive shell: - -```text -docker compose -f .devcontainer/docker-compose.yml exec app bash -``` - ---- +This document defines execution constraints for AI agents. For general contributor rules, setup commands, architectural constraints, and security policies, see [docs/README.md](docs/README.md). ## Collaboration Agreement (Agent ↔ User) -## Interview Answers (ID-able) + Expert Recommendations - -**DoD:** `make ready` in Dev Container; if applicable, user completes manual smoke test with agent-provided steps. -**Verification:** Always smoke Dev Container + `make ready`. -**Commits:** Group by logical unit after smoke-tested (feature / improvement / refactor). -**Responses:** Changes → Commands → Results → Next steps, ending with a concise one-line summary. -**KISS vs Refactor:** KISS by default; boy-scout refactors allowed if low-risk and simplifying. -**Ambiguity:** Proceed with safest assumption, then confirm. -**Non-negotiables:** Dev Container only; security first. - -Expert recommendation: keep workflows terminal-first and keyboard-focused (clear commands, no GUI-only steps). +- **DoD:** `make ready` in Dev Container; if applicable, user completes manual smoke test with agent-provided steps. +- **Verification:** Always smoke Dev Container + `make ready`. +- **Commits:** Group by logical unit after smoke-tested (feature / improvement / refactor). +- **Responses:** Changes → Commands → Results → Next steps, ending with a concise one-line summary. +- **KISS vs Refactor:** KISS by default; boy-scout refactors allowed if low-risk and simplifying. +- **Ambiguity:** Proceed with safest assumption, then confirm. +- **Non-negotiables:** Dev Container only; security first. -## Definition of Done - -- Run `make ready` inside the Dev Container. -- If applicable, user completes manual smoke test; agent provides clear instructions. - -## Verification Rules +## Agent-Specific Verification Rules - Always run Dev Container smoke + `make ready` for changes. +- For frontend changes, also verify in `chrome-devtools` MCP at `http://127.0.0.1:4001/` while the Dev Container is running. +- Capture a quick state check for all affected UI states (e.g., guest/member/result) to enforce state parity and avoid duplicate actions. -## Commit Granularity +### Frontend Smoke Checklist -- Group commits by logical units after they have grown and been smoke-tested (feature / improvement / refactor). +- Start the Dev Container and app (`make dev`). +- Open `http://127.0.0.1:4001/` with `chrome-devtools` MCP. +- Validate the primary user path touched by the change. +- Verify all affected states (e.g., guest/member/result) keep the same layout grammar. +- Confirm action uniqueness: one canonical control per outcome in each state. -## Response Format +## UI Execution Principles -- Default: Changes → Commands → Results → Next steps. -- End with a concise one-line summary. +See [docs/design-system.md](docs/design-system.md) for visual rules. -## KISS vs Refactor +- **Task Dominance:** Each UI state should make one user objective obvious and primary. Supporting surfaces and links must yield priority. +- **Copy Minimalism:** Remove text that repeats what the interface already communicates. Prefer action-oriented wording. +- **State Skeleton:** Adjacent UI states should read as the same frame with content changes, not as separate pages. +- **Focus Contract:** Verify browser autofocus and return-focus behavior on initial load and after transitions. +- **Support Compression:** When a user has advanced past setup, reduce the visual weight of support content. -- KISS by default. -- Boy-scout refactors are allowed when they reduce complexity and are low-risk. - -## Ambiguity Handling +## Response Format -- Proceed with the safest assumption, then confirm. +1. **Changes:** Briefly list files/symbols modified. +2. **Commands:** Show the verification commands run. +3. **Results:** Summarize the outcome. +4. **Next steps:** Propose the immediate follow-up. +5. **One-line Summary:** End with a concise summary. ## Non-Negotiables -- Security first. -- YARD docs are strict for public Ruby methods in `app/`: every public method must have a YARD docstring with typed `@param` tags (for all params) and typed `@return`. -- When touching non-public methods, add YARD docs when it improves maintenance or clarifies invariants/edge handling. +- **Security first:** No leaking secrets or insecure patterns. See [Security & Safety Rules](docs/README.md#security--safety-rules). +- **YARD docs:** Strict for public Ruby methods in `app/`. Every public method must have a YARD docstring with typed `@param` and `@return`. See [Architectural Constraints](docs/README.md#architectural-constraints). +- **No host execution:** All commands MUST run inside the Dev Container via `make` or `bundle exec`. diff --git a/README.md b/README.md index 9bf7d39b..85ec5130 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ html2rss-web converts arbitrary websites into RSS 2.0 feeds with a slim Ruby bac ## Links - Docs & feed directory: https://html2rss.github.io +- Contributor Guide: [docs/README.md](docs/README.md) - Discussions: https://github.com/orgs/html2rss/discussions - Sponsor: https://github.com/sponsors/gildesmarais @@ -15,29 +16,35 @@ html2rss-web converts arbitrary websites into RSS 2.0 feeds with a slim Ruby bac - Responsive Preact interface for demo, sign-in, conversion, and result flows. - Automatic source discovery with token-scoped permissions. - Signed public feed URLs that work in standard RSS readers. -- Built-in SSRF defences, input validation, and HMAC-protected tokens. +- Built-in URL validation, scoped feed access controls, and HMAC-protected tokens. -## Architecture +## Architecture Overview - **Backend:** Ruby + Roda, backed by the `html2rss` gem for extraction. - **Frontend:** Preact app built with Vite into `frontend/dist` and served at `/`. -- **Distribution:** Docker Compose by default; other deployments require manual wiring. -- [Project notes](docs/README.md) +- **Distribution:** Docker Compose by default. -## REST API Snapshot +For detailed architecture and internal rules, see [docs/README.md](docs/README.md). + +## Trial Run (Docker Pull And Run) + +The published image already includes a sample `config/feeds.yml`, so you can try the app without creating or mounting one first. ```bash -# List available strategies -curl -H "Authorization: Bearer " \ - "https://your-domain.com/api/v1/strategies" - -# Create a feed and capture the signed public URL -curl -X POST "https://your-domain.com/api/v1/feeds" \ - -H "Authorization: Bearer " \ - -H "Content-Type: application/json" \ - -d '{"url":"https://example.com","name":"Example Feed"}' +docker run --rm \ + -p 4000:4000 \ + -e RACK_ENV=production \ + -e HTML2RSS_SECRET_KEY=$(openssl rand -hex 32) \ + html2rss/web ``` +Then open: + +- `http://localhost:4000/` for the web UI +- `http://localhost:4000/microsoft.com/azure-products.rss` for a built-in Azure updates feed + +This trial run is intentionally minimal. Use Docker Compose for Browserless, auto-updates, or local feed overrides. + ## Deploy (Docker Compose) 1. Generate a key: `openssl rand -hex 32`. @@ -46,73 +53,12 @@ curl -X POST "https://your-domain.com/api/v1/feeds" \ UI + API run on `http://localhost:4000`. The app exits if the secret key is missing. -## Development (Dev Container) +## Development Use the repository's [Dev Container](.devcontainer/README.md) for all local development and tests. Running the app directly on the host is not supported. -Quick start inside the Dev Container: - -``` -make setup -make dev -make test -make ready -make yard-verify-public-docs -bundle exec rubocop -F -bundle exec rspec -make openapi -``` - -Dev URLs: Ruby app at `http://localhost:4000`, frontend dev server at `http://localhost:4001`. - -Backend code under the `Html2rss::Web` namespace now lives under `app/web/**`, so Zeitwerk can mirror constant paths directly instead of relying on directory-specific namespace wiring. -`make ready` also runs `rake zeitwerk:verify`, which eager-loads the app and fails on loader drift early. -For contributors and AI agents changing backend structure, follow the rules in [docs/README.md](docs/README.md). - -## Make Targets - -| Command | Purpose | -| ------------------------------ | ---------------------------------------------------------- | -| `make help` | List available shortcuts. | -| `make setup` | Install Ruby and Node dependencies. | -| `make dev` | Run Ruby (port 4000) and frontend (port 4001) dev servers. | -| `make dev-ruby` | Start only the Ruby server. | -| `make dev-frontend` | Start only the frontend dev server (port 4001). | -| `make test` | Run Ruby and frontend test suites. | -| `make test-ruby` | Run Ruby specs. | -| `make test-frontend` | Run frontend unit and contract tests. | -| `make lint` | Run all linters. | -| `make lintfix` | Auto-fix lint warnings where possible. | -| `make yard-verify-public-docs` | Enforce typed YARD docs for public methods in `app/`. | -| `make openapi` | Regenerate `public/openapi.yaml` from request specs. | -| `make openapi-verify` | Regenerate + fail if OpenAPI file is stale. | -| `make clean` | Remove build artefacts. | - -## OpenAPI Contract - -The OpenAPI file is generated from Ruby request specs only. - -- Regenerate: `make openapi` -- Verify drift (CI behavior): `make openapi-verify` - -## Frontend npm Scripts (inside Dev Container) - -| Command | Purpose | -| ----------------------- | -------------------------------------------- | -| `npm run dev` | Vite dev server with hot reload (port 4001). | -| `npm run build` | Build static assets into `frontend/dist/`. | -| `npm run test:run` | Unit tests (Vitest). | -| `npm run test:contract` | Contract tests with MSW. | - -## Testing Strategy - -| Layer | Tooling | Focus | -| ----------------- | ------------------------ | ---------------------------------------------------- | -| Ruby API | RSpec + Rack::Test | Feed creation, retrieval, auth paths. | -| Frontend unit | Vitest + Testing Library | Component rendering and hooks with mocked fetch. | -| Frontend contract | Vitest + MSW | End-to-end fetch flows against mocked API responses. | -| Docker smoke | RSpec (`:docker`) | Net::HTTP probes against the containerised service. | +See the [Contributor Guide](docs/README.md) for setup commands, testing strategy, and backend structure rules. ## Contributing diff --git a/docs/README.md b/docs/README.md index c98c8bae..d971cd39 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,44 +1,96 @@ -# Project Notes +# Project Documentation & Contributor Guide -This is the only hand-written project document in `docs/`. +Welcome! This is the canonical source of truth for contributing to `html2rss-web`. -Keep this file short, current, and operational. Do not add planning docs, migration diaries, redesign notes, or parallel architecture narratives back into this directory. +## Docs Index -The only generated artifact intentionally exposed by the app is [`public/openapi.yaml`](/Users/gil/versioned/html2rss/html2rss-web/public/openapi.yaml). +- **Start here for contributors**: This document. +- **Architecture & Request Lifecycle**: [docs/architecture.md](architecture.md) +- **UI/Design rules**: [docs/design-system.md](design-system.md) +- **Agent execution constraints**: [AGENTS.md](../AGENTS.md) +- **Generated contract artifacts**: `public/openapi.yaml` +- **Public-facing intro**: [README.md](../README.md) -## System Snapshot +--- -- Backend: Ruby + Roda under the `Html2rss::Web` namespace. -- Frontend: Preact + Vite, built into `frontend/dist` and served at `/`. -- Feed extraction: delegated to the `html2rss` gem. -- Distribution: Docker Compose / Dev Container first. +## System Snapshot & Architecture -## Source Of Truth +`html2rss-web` converts arbitrary websites into RSS 2.0 feeds. -- Runtime behavior: application code plus tests. -- HTTP contract: request specs plus generated OpenAPI. -- This file: contributor conventions and current project rules only. +- **Backend**: Ruby + Roda under the `Html2rss::Web` namespace. +- **Frontend**: Preact + Vite, built into `frontend/dist` and served at `/`. +- **Feed extraction**: Delegated to the `html2rss` gem. +- **Distribution**: Docker Compose / Dev Container first. -## Verification +### Source Of Truth -Primary local gate: +- **Runtime behavior**: Application code plus tests. +- **HTTP contract**: Request specs plus generated OpenAPI. +- **This file**: Contributor conventions and current project rules. -```text -docker compose -f .devcontainer/docker-compose.yml exec -T app make ready -``` +--- + +## Development Setup (Dev Container) + +Use the repository's [Dev Container](../.devcontainer/README.md) for all local development and tests. +Running the app directly on the host is not supported. + +### Common Commands (Inside Dev Container) + +| Command | Purpose | +| ------------------------------ | ---------------------------------------------------------- | +| `make setup` | Install Ruby and Node dependencies. | +| `make dev` | Run Ruby (port 4000) and frontend (port 4001) dev servers. | +| `make ready` | Full pre-flight check (Lint + Test + OpenAPI + Zeitwerk). | +| `make test` | Run Ruby and frontend test suites. | +| `make lint` | Run all linters. | +| `make yard-verify-public-docs` | Enforce typed YARD docs for public methods in `app/`. | +| `make openapi` | Regenerate `public/openapi.yaml` from request specs. | + +### Frontend npm Scripts + +| Command | Purpose | +| ----------------------- | -------------------------------------------- | +| `npm run dev` | Vite dev server with hot reload (port 4001). | +| `npm run build` | Build static assets into `frontend/dist/`. | +| `npm run test:run` | Unit tests (Vitest). | +| `npm run test:contract` | Contract tests with MSW. | + +--- + +## Contract-Driven Development Loop -Useful commands: +To change or add API endpoints, follow this sequence: -```text -docker compose -f .devcontainer/docker-compose.yml exec -T app make setup -docker compose -f .devcontainer/docker-compose.yml exec -T app make dev -docker compose -f .devcontainer/docker-compose.yml exec -T app bundle exec rspec -docker compose -f .devcontainer/docker-compose.yml exec -T app make openapi -docker compose -f .devcontainer/docker-compose.yml exec -T app make openapi-verify -docker compose -f .devcontainer/docker-compose.yml exec -T app make openapi-lint +1. **Ruby Request Spec**: Define the new behavior or endpoint in `spec/html2rss/web/app_integration_spec.rb` or a dedicated request spec. +2. **OpenAPI Generation**: Run `make openapi` inside the Dev Container to regenerate `public/openapi.yaml` from the spec metadata. +3. **Verify Contract**: Run `make openapi-verify` and `make openapi-lint` to ensure the generated file matches the specs and is valid. +4. **Frontend Client**: The frontend generated client in `frontend/src/api/generated` is updated by the build process. + +Always verify the contract before committing API changes. + +--- + +## Verification & Testing Strategy + +### Local Verification Gate + +Always run this before pushing or committing: + +```bash +make ready ``` -Frontend verification lives at `http://127.0.0.1:4001/` while the dev container is running. +### Testing Layers + +| Layer | Tooling | Focus | +| ----------------- | ------------------------ | ---------------------------------------------------- | +| Ruby API | RSpec + Rack::Test | Feed creation, retrieval, auth paths. | +| Frontend unit | Vitest + Testing Library | Component rendering and hooks with mocked fetch. | +| Frontend contract | Vitest + MSW | End-to-end fetch flows against mocked API responses. | +| Docker smoke | RSpec (`:docker`) | Net::HTTP probes against the containerised service. | + +--- ## Backend Structure Rules @@ -48,11 +100,13 @@ Frontend verification lives at `http://127.0.0.1:4001/` while the dev container - Keep route composition in `app/web/routes/**`. - Keep `/api/v1` contract-specific code in `app/web/api/**`. - Keep feed fetching, caching, and orchestration in `app/web/feeds/**`. -- Keep auth, token handling, SSRF strategy, and security logging in `app/web/security/**`. +- Keep auth, token handling, URL validation, and security logging in `app/web/security/**`. - Keep request-scoped context in `app/web/request/**`. - Keep boot/runtime setup in `app/web/boot/**`. - Do not create generic buckets such as `services`, `helpers`, `utils`, or `concerns`. +--- + ## API Contract Rules - `public/openapi.yaml` is generated output, not hand-edited design prose. @@ -62,15 +116,59 @@ Frontend verification lives at `http://127.0.0.1:4001/` while the dev container - Quality must fail with `make openapi-lint`. - Frontend generated client code under `frontend/src/api/generated` is machine-generated only. -## Runtime Flags +--- + +## Core Dependencies + +Search these pages for examples, plugins, and configuration options: + +- **Roda**: [roda.jeremyevans.net](https://roda.jeremyevans.net/documentation.html) +- **Preact & Vite**: [preactjs.com](https://preactjs.com/guide/v10/getting-started/) and [vite.dev](https://vite.dev/guide/) +- **html2rss**: [github.com/html2rss/html2rss](https://github.com/html2rss/html2rss) +- **Testing (Ruby)**: [rspec.info](https://rspec.info/features/3-13/rspec-expectations/built-in-matchers/), [rubocop.org](https://docs.rubocop.org/rubocop/cops.html), [betterspecs.org](https://www.betterspecs.org/) + +--- + +## Security & Safety Rules + +- **URL Handling**: Never use Ruby's `URI` class or `addressable` gem directly. Use `Html2rss::Url` for all URL logic. +- **SSRF Protection**: Delegated to the `html2rss` gem's built-in security features. Do not bypass these protections or weaken CSP. +- **Secrets**: Never leak stack traces, auth tokens, or internal secrets in HTTP responses. +- **Data Protection**: Auth tokens provided by users must never be exposed or logged. + +--- -Managed flags: +## Architectural Constraints + +- **No Persistence**: Do not add databases, ORMs, or background job systems. +- **Backend Style**: + - Keep the main `app.rb` thin; organize routes in `Html2rss::Web::Routes::*`. + - For helpers, use `class << self` and `private` methods. Avoid `module_function`. + - Use YARD doc comments for all public methods in `app/`. + - Add `# frozen_string_literal: true` to all Ruby files. + - Do not use `send(...)` to reach into private APIs; expose what is needed at the module level. +- **Frontend Style**: + - Follow visual and CSS rules in [design-system.md](design-system.md). + - Use Preact components in `frontend/src/`. + - Use shared styles in `public/shared-ui.css` or app-specific styles in `frontend/src/styles/`. + - Do not modify `frontend/dist/` directly. +- **Testing**: + - Use `ClimateControl.modify` for tests that change environment variables. + - Use `:aggregate_failures` to resolve `RSpec/MultipleExpectations` warnings. + +--- + +## Environment & Runtime Flags + +Managed flags and environment keys: | Name | Env key | Type | Default | | --------------------------------- | --------------------------------- | -------------- | ---------------------------------------- | | `auto_source_enabled` | `AUTO_SOURCE_ENABLED` | boolean | `true` in development/test, else `false` | | `async_feed_refresh_enabled` | `ASYNC_FEED_REFRESH_ENABLED` | boolean | `false` | | `async_feed_refresh_stale_factor` | `ASYNC_FEED_REFRESH_STALE_FACTOR` | integer `>= 1` | `3` | +| `health_check_token` | `HEALTH_CHECK_TOKEN` | string | `nil` | +| `sentry_dsn` | `SENTRY_DSN` | string | `nil` | Rules: @@ -78,31 +176,15 @@ Rules: - Unknown managed feature-style env keys must fail fast at boot. - Add or change flags in code, tests, and this table together. -## Observability Contract - -Canonical event fields: +--- -- `event_name` -- `schema_version` -- `request_id` -- `route_group` -- `actor` -- `outcome` - -Optional request context fields: +## Observability Contract -- `path` -- `method` -- `strategy` -- `started_at` -- `details` +Canonical event fields: `event_name`, `schema_version`, `request_id`, `route_group`, `actor`, `outcome`. -Critical-path event families: +Critical-path event families: auth, feed create, feed render, request errors. -- auth -- feed create -- feed render -- request errors +--- ## Documentation Policy diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..8c5480fd --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,51 @@ +# Architecture & Request Lifecycle + +This document provides a mental model of how `html2rss-web` processes requests. + +## High-Level Data Flow + +```mermaid +flowchart TD + user["User / RSS Reader"] --> routes["Roda App (app/web/routes)"] + security["Auth / Security (app/web/security)"] --> routes + routes --> feeds["Feeds Service (app/web/feeds)"] + cache["Cache (app/web/feeds/cache.rb)"] --> feeds + feeds --> gem["html2rss Gem"] + strategies["Request Strategies (Faraday / Browserless)"] --> gem + gem --> target["Target Website"] +``` + +## Request Lifecycle + +### 1. Routing & Auth + +Requests enter via `app.rb` and are dispatched to `app/web/routes/`. + +- **API v1**: Authenticated via `app/web/security/auth.rb`. +- **Public Feeds**: Validated via HMAC tokens in `app/web/security/feed_token.rb`. + +### 2. Resolution + +The `Html2rss::Web::Feeds::SourceResolver` determines where the feed configuration comes from: + +- **Static**: Pre-defined in `config/feeds.yml`. +- **Dynamic**: Generated on-the-fly via the `/api/v1/feeds` endpoint (AutoSource). + +### 3. Fetching & Rendering + +The `Html2rss::Web::Feeds::Service` orchestrates the extraction: + +1. Checks the `Html2rss::Web::Feeds::Cache`. +2. If stale/missing, calls the `html2rss` gem with the resolved strategy. +3. Renders the output using `RssRenderer` (XML) or `JsonRenderer`. + +## Extension Points + +### Adding a Request Strategy + +Strategies are defined by the `html2rss` gem but can be configured here. + +- **Faraday**: Default HTTP client for static HTML. +- **Browserless**: Used for JavaScript-heavy websites. + +To add or configure strategies, see `app/web/feeds/source_resolver.rb` and the `html2rss` gem documentation. diff --git a/docs/design-system.md b/docs/design-system.md new file mode 100644 index 00000000..d9b243a3 --- /dev/null +++ b/docs/design-system.md @@ -0,0 +1,235 @@ +# Design System Manifest + +This document is the visual source of truth for `html2rss-web`. + +If you change the UI, read this first. If a design decision is not reflected here, update this file in the same change. The goal is to prevent slow visual drift between the app shell, result flow, and RSS/XSL presentation. + +## Core Rule + +There is one shared design language and one shared primitive layer. + +- Shared primitives live in [public/shared-ui.css](../public/shared-ui.css). +- App-specific composition lives in [frontend/src/styles/main.css](../frontend/src/styles/main.css). +- Feed-specific composition lives in [public/rss.xsl](../public/rss.xsl). + +Do not duplicate tokens, base canvas rules, card shells, rails, stack primitives, or brand-lockup styling in `main.css` or `rss.xsl`. If app and feed both need it, it belongs in `shared-ui.css`. + +## Visual Thesis + +The UI is not a generic SaaS dashboard. It should read as: + +- stark +- editorial +- quiet but deliberate +- dark, with restrained light accents +- compact, not crowded + +The experience should feel like one product across: + +- `/` +- the token gate +- the result page +- `/example.rss` and all XSL-rendered feeds + +If a page looks like it came from a different product, the change is wrong even if the CSS is technically valid. + +## Non-Negotiable Surface Rules + +- Background must use the same dark canvas and top-light treatment defined in `shared-ui.css`. +- Shared cards must use the same border, radius, and surface treatment. +- Serif display typography is reserved for major titles and the wordmark. +- Sans UI typography is the default for controls, supporting copy, and metadata. +- Mono is reserved for URLs, tokens, and machine-like values. +- Eyebrow text is uppercase, compact, and low-noise. +- Spacing should come from the token scale only. + +Do not introduce: + +- ad hoc colors +- page-local shadows that fight the shared card elevation +- one-off radii +- extra spacing scales +- component-specific typography systems + +## Architecture + +The CSS is intentionally split by responsibility, not by page count. + +### 1. Shared Primitive Layer + +Owned by [public/shared-ui.css](../public/shared-ui.css). + +This file owns: + +- tokens +- global box sizing and canvas behavior +- global typography baseline +- link behavior +- rails +- stack primitives +- card primitives +- eyebrow primitive +- brand lockup + +This file should stay small, boring, and reusable. + +### 2. App Composition Layer + +Owned by [frontend/src/styles/main.css](../frontend/src/styles/main.css). + +This file owns: + +- page-shell composition +- workspace layout +- form behavior +- dominant-field behavior +- button behavior +- notice state behavior +- token-gate composition +- result-page composition +- utility strip behavior + +This file should not redefine shared primitives. + +### 3. Feed Composition Layer + +Owned by [public/rss.xsl](../public/rss.xsl). + +This file owns only feed-page specifics: + +- feed hero composition +- feed metadata rows +- feed list/card content styling +- feed empty/error presentation + +This file should compose shared classes rather than restyle them. + +## Approved Primitive API + +Prefer composing these primitives before inventing new classes: + +- `layout-shell` +- `layout-rail-reading` +- `layout-rail-copy` +- `layout-stack` +- `layout-stack--tight` +- `ui-card` +- `ui-card--padded` +- `ui-card--roomy` +- `ui-card--notice` +- `ui-eyebrow` +- `brand-lockup` +- `input` +- `input--lg` +- `input--minimal` +- `input--mono` +- `btn` +- `btn--primary` +- `btn--ghost` +- `btn--quiet` +- `btn--linkish` + +Semantic state should prefer attributes over extra visual variants: + +- `data-tone="error"` +- `data-tone="success"` +- `data-state="loading"` + +This is deliberate. We want a small CSS API with composable primitives, not endless component-local variants. + +## Variant Discipline + +Before adding a new class or modifier, ask: + +1. Can this be expressed by composing existing primitives? +2. Is this a reusable primitive or only page-local composition? +3. Is this visual difference actually perceptible and meaningful? +4. Does this belong to structure, modifier, or semantic state? + +Default answers: + +- New primitive: rare +- New modifier: suspicious +- New component-specific variant: usually wrong +- New semantic attribute: acceptable when behavior or tone truly changes + +Avoid returning to patterns like: + +- `input--hero` +- `input--select` +- `status-card` +- multiple near-identical surface tokens + +Those create variant creep. + +## Color And Surface Rules + +Use only the shared tokens unless there is a strong system-level reason to extend them. + +Key expectations: + +- `--surface-base` is the default card plane. +- `--surface-elevated` is for stronger inputs and interactive surfaces. +- success and error backgrounds are semantic overlays, not new card systems. +- border strength should increase only for focus or meaningful emphasis. + +If you think you need another surface token, the burden of proof is high. + +## Typography Rules + +- `--font-family-display` is for primary titles and the wordmark only. +- `--font-family-ui` is the default everywhere else. +- `--font-family-mono` is for feed URLs, tokens, and similarly mechanical strings. +- `ui-eyebrow` is the preferred pattern for small uppercase metadata labels. + +Do not create alternate display systems per page. + +## Layout Rules + +The layout language is narrow on purpose. + +- Use rails to control readable width. +- Use stack primitives for vertical rhythm. +- Keep shells centered and calm. +- Prefer composition over custom grid declarations. + +If you add `display: grid`, be able to explain why an existing stack or rail primitive was insufficient. + +## Agent Checklist + +When changing UI, an agent must verify: + +1. Does the change reuse `shared-ui.css` where appropriate? +2. Did I avoid duplicating a shared primitive in `main.css` or `rss.xsl`? +3. Does the app still match the RSS/XSL rendering in overall tone and framing? +4. Did I avoid inventing a page-local variant for something that should be a modifier or attribute? +5. If I added a token, modifier, or primitive, did I justify it in this file? + +## Drift Triggers + +These are common signs that the system is drifting: + +- app and RSS page use different canvas/background treatment +- same content type gets different card shells +- page-local spacing values appear outside the token scale +- headings start mixing unrelated type styles +- new input or card variants appear with overlapping purpose +- semantic states are encoded as a growing list of presentational classes + +If you see one of these, consolidate instead of layering more CSS. + +## Change Policy + +When changing the design system: + +- update the shared primitive first if the rule is cross-surface +- update this manifest if the rule changes +- keep the primitive API smaller, not larger, when possible +- validate both app UI and RSS/XSL output + +The right direction is brutal clarity: + +- fewer primitives +- fewer variants +- stronger shared identity +- less local exception code