feat: lawful basis builder, spec compliance check, conserver-direct delivery

.github/workflows/test.yml

@@ -18,10 +18,12 @@ jobs:
         uses: astral-sh/setup-uv@v3
 
       - name: Set up Python ${{ matrix.python-version }}
-        run: uv python install ${{ matrix.python-version }}
+        run: |
+          uv venv --python ${{ matrix.python-version }}
+          echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
 
       - name: Install
-        run: uv pip install --system -e ".[dev]"
+        run: uv pip install -e ".[dev]"
 
       - name: Lint
         run: |

.gitignore

@@ -16,3 +16,5 @@ dist/
 build/
 config.yaml
 !config.example.yaml
+.venv-scratch/
+.omc/

README.md

@@ -2,7 +2,8 @@
 
 > Adapter that converts **__SOURCE_PLATFORM__** events into [vCon](https://datatracker.ietf.org/doc/draft-ietf-vcon-vcon-core/) (Virtual Conversation) objects and delivers them to a [vCon server](https://github.com/vcon-dev/vcon-server) or other downstream consumer.
 
-**Spec target:** IETF `draft-ietf-vcon-vcon-core-02`, vCon syntax `"0.4.0"`.
+**Spec target:** IETF `draft-ietf-vcon-vcon-core-04`, vCon syntax `"0.4.0"`
+(the syntax string is deprecated in -04 but kept for parser compatibility).
 
 [![Tests](https://github.com/vcon-dev/vcon-__ADAPTER_NAME__-adapter/actions/workflows/test.yml/badge.svg)](https://github.com/vcon-dev/vcon-__ADAPTER_NAME__-adapter/actions/workflows/test.yml)
 [![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
@@ -27,14 +28,16 @@ Then delete this section.
 ## Features
 
 - Spec-compliant vCons (syntax `0.4.0`, `mediatype`, `base64url`, ISO-8601 UTC timestamps)
-- Pluggable transcription via `TranscriptionProvider` protocol (WTF format in `analysis[]`)
-- HMAC-SHA256 webhook body signing (`X-Hub-Signature-256`)
-- Idempotent delivery (`Idempotency-Key` header = vCon UUID)
-- Exponential-backoff retries with dead-letter queue
-- Optional JWS signing (RS256) of each vCon before delivery
+- `encoding: "json"` bodies are the raw JSON value, not a `json.dumps()` string (draft-ietf-vcon-vcon-core-04 §2.3.2); `json_body()` reads either shape back
+- Lawful-basis attachments (`draft-howe-vcon-lawful-basis`) via `LawfulBasisConfig` + `add_lawful_basis()`
+- Two delivery modes, selected by `delivery.mode`:
+  - `webhook`: HMAC-SHA256 body signing (`X-Hub-Signature-256`), `Idempotency-Key` header, exponential-backoff retries, dead-letter queue
+  - `conserver`: direct `POST {CONSERVER_URL}/vcon` to a vcon-server instance (`x-conserver-api-token` header, `ingress_lists` query params), same retry/backoff/DLQ machinery
 - `/healthz` + Prometheus `/metrics` endpoints
 - Configurable via YAML with `${ENV_VAR}` substitution
 
+**Not implemented in this template** (remove this note once you've decided): a `TranscriptionProvider` protocol and optional JWS (RS256) signing of outgoing vCons were previously advertised here but never built. Add them yourself if your adapter needs them, or drop the mention.
+
 ---
 
 ## Install
@@ -67,8 +70,15 @@ See [`config.example.yaml`](config.example.yaml) for all options.
 
 Required env vars:
 - `__ADAPTER_PACKAGE___API_KEY` — credentials for __SOURCE_PLATFORM__
-- `VCON_WEBHOOK_URL` — where to POST vCons
-- `VCON_WEBHOOK_HMAC_SECRET` — shared secret for body signing
+- `VCON_WEBHOOK_URL` — where to POST vCons (`delivery.mode: webhook`)
+- `VCON_WEBHOOK_HMAC_SECRET` — shared secret for body signing (`delivery.mode: webhook`)
+- `CONSERVER_URL`, `CONSERVER_API_TOKEN` — target vcon-server and its API token (`delivery.mode: conserver`)
+
+Optional lawful-basis env vars (see [USAGE.md](USAGE.md)):
+`LAWFUL_BASIS`, `LAWFUL_BASIS_PURPOSE`, `LAWFUL_BASIS_JURISDICTION`,
+`LAWFUL_BASIS_EXPIRATION`, `LAWFUL_BASIS_PROOF_MECHANISM`,
+`LAWFUL_BASIS_PROOF_DESCRIPTION`. Unset means vCons are built with no
+lawful-basis attachment — never default this in code.
 
 ## Test
 

USAGE.md

@@ -44,11 +44,71 @@ find . -name "*.bak" -delete
 2. Delete the "What this is" section at the top of `README.md`
 3. Edit `config.example.yaml` — add your platform-specific options under `source:`
 4. Implement your platform listener in `src/<package>/cli.py` (search for the `TODO: wire your source-platform listener` comment)
-5. If you don't need transcription, JWS signing, or Prometheus — delete those features and their dependencies from `pyproject.toml`
+5. Call `add_lawful_basis()` for every vCon you build. Load its config once at
+   startup with `LawfulBasisConfig.from_env()` (env vars only) or
+   `LawfulBasisConfig.resolve(yaml_block=config.vcon.get("lawful_basis"), env=...)`
+   (YAML + env, env wins per-field — this is what `config.load_config()`
+   already wires into `Config.lawful_basis`). Then, for each vCon:
+
+   ```python
+   from datetime import datetime, timezone
+   from .vcon_builder import add_lawful_basis
+
+   granted_at = datetime.now(timezone.utc).isoformat()
+   add_lawful_basis(v, config.lawful_basis, granted_at=granted_at, party=0, dialog=0)
+   ```
+
+   If `LAWFUL_BASIS` (or `vcon.lawful_basis.lawful_basis` in YAML) is unset,
+   this logs one warning per process and adds nothing — it does not raise
+   and does not invent a basis. Set an invalid value and it raises
+   `ValueError` at config-load time instead of silently building a bad vCon.
+6. Pick a delivery mode in `config.yaml`: `delivery.mode: webhook` (generic
+   HTTP + HMAC, the `webhook:` section) or `delivery.mode: conserver`
+   (direct `POST {url}/vcon` to a vcon-server instance, the `conserver:`
+   section — token header, `ingress_lists` query params). Both share the
+   same retry/backoff/dead-letter-queue implementation in
+   `webhook_delivery.py`, and both call `finalize_vcon()` on your vCon dict
+   before serializing it — see the next point.
+7. Call `finalize_vcon(v.vcon_dict)` on every finished vCon before you
+   serialize or store it *outside* `WebhookDelivery`/`ConserverDelivery`
+   (those two already call it for you inside `deliver()`). It strips empty
+   `meta`/`metadata` placeholders that vcon-lib 0.9.6 leaves on every
+   `Dialog` added via `add_dialog()` — real objects with real content are
+   left alone; only the empty `{}` vcon-lib defaults are removed. Skipping
+   this on a path that bypasses both delivery classes (e.g. writing
+   straight to local storage) will produce a vCon that fails schema
+   validation.
+8. `encoding: "json"` bodies (draft-ietf-vcon-vcon-core-04 §2.3.2): `body` is
+   the raw JSON value (object/array/number/bool/null), never a
+   `json.dumps()`'d string — `add_lawful_basis()` follows this already. Do
+   the same for any other JSON-encoded attachment or analysis body you
+   write. When *reading* a JSON body back (yours or one from an older -02
+   vCon), use `json_body(attachment)` — it returns the value as-is if it's
+   already a JSON value, or `json.loads()`s it first if it's a legacy
+   string, so you don't need to special-case either shape.
+9. If you don't need Prometheus — delete it and its dependency from `pyproject.toml`.
+   (A `TranscriptionProvider` protocol and JWS signing were previously
+   advertised in the README but never implemented — add them yourself if
+   your adapter needs them, or ignore the now-corrected README note.)
 
 ## Step 4 — Verify spec compliance
 
-The smoke tests in `tests/test_vcon_builder.py` enforce the spec compliance checklist. **Keep them green.**
+Two test modules enforce spec compliance (draft-ietf-vcon-vcon-core-04).
+**Keep them green.**
+
+- `tests/test_vcon_builder.py` — smoke tests for the builder helpers,
+  including `LawfulBasisConfig`, `add_lawful_basis()`, `finalize_vcon()`,
+  and `json_body()`.
+- `tests/test_spec_compliance.py` — validates a sample vCon (built with a
+  lawful-basis attachment) against the vendored official JSON schema at
+  `tests/schema/vcon_json_schema.json` (see `tests/schema/SOURCE.md` for
+  where it came from and how to refresh it), plus the non-negotiables the
+  schema alone doesn't fully enforce (no `mimetype`, attachments carry
+  `purpose`/`start`/`party`/`dialog`, `body` is a string unless
+  `encoding: "json"` — then it must NOT be a string, catching accidental
+  double-encoding — no empty `meta`/`metadata`/`group`/`redacted`). Its
+  `assert_spec_compliant()` function is written to be copied verbatim into
+  another vCon-writing repo's own tests.
 
 ```bash
 uv pip install -e ".[dev]"

config.example.yaml

@@ -17,13 +17,33 @@ vcon:
   extensions:
     - sip-signaling
     - lawful_basis
-  # Optional: JWS signing
-  signing:
-    enabled: false
-    private_key_path: ${VCON_SIGNING_KEY_PATH}
-    key_id: ${VCON_SIGNING_KEY_ID}
+  # Optional: JWS signing (RS256) — NOT IMPLEMENTED in this template yet.
+  # Left here as a placeholder for adapters that add it; delete if unused.
+  # signing:
+  #   enabled: false
+  #   private_key_path: ${VCON_SIGNING_KEY_PATH}
+  #   key_id: ${VCON_SIGNING_KEY_ID}
 
-# Webhook delivery
+  # Optional: lawful-basis attachment (draft-howe-vcon-lawful-basis).
+  # LAWFUL_BASIS / LAWFUL_BASIS_* env vars override these fields one-by-one;
+  # see LawfulBasisConfig.resolve() in vcon_builder.py. Leave `lawful_basis`
+  # unset (or delete this block) to build vCons with no lawful-basis
+  # attachment — never invent a basis here.
+  lawful_basis:
+    lawful_basis: ${LAWFUL_BASIS}  # consent|contract|legal_obligation|vital_interests|public_task|legitimate_interests
+    purposes:
+      - recording
+    jurisdiction: ${LAWFUL_BASIS_JURISDICTION}
+    expiration: ${LAWFUL_BASIS_EXPIRATION}
+    proof_mechanism: ${LAWFUL_BASIS_PROOF_MECHANISM}
+    proof_description: ${LAWFUL_BASIS_PROOF_DESCRIPTION}
+
+# Delivery mode: "webhook" (generic HTTP + HMAC) or "conserver" (direct
+# POST to a vcon-server instance). Only the selected section below is used.
+delivery:
+  mode: webhook
+
+# Webhook delivery (delivery.mode: webhook)
 webhook:
   endpoints:
     - url: ${VCON_WEBHOOK_URL}
@@ -35,6 +55,19 @@ webhook:
     max_backoff_seconds: 60
   dead_letter_path: /app/vcons/dlq
 
+# Conserver-direct delivery (delivery.mode: conserver)
+# POSTs to {url}/vcon with header `x-conserver-api-token: ${CONSERVER_API_TOKEN}`
+# and repeated `?ingress_lists=...` query params. Reuses the same
+# retry/backoff/DLQ implementation as webhook delivery (ConserverDelivery's
+# constructor defaults match WebhookConfig's); not yet exposed as separate
+# YAML keys here — pass overrides directly to ConserverDelivery(...) if needed.
+conserver:
+  url: ${CONSERVER_URL}
+  token: ${CONSERVER_API_TOKEN}
+  token_header: x-conserver-api-token
+  ingress_lists:
+    - default
+
 # Local filesystem storage (optional)
 storage:
   enabled: true

pyproject.toml

@@ -35,6 +35,9 @@ dev = [
     "pytest-cov>=4.0.0",
     "ruff>=0.4.0",
     "mypy>=1.0.0",
+    "jsonschema>=4.20.0",
+    "types-PyYAML>=6.0",
+    "types-aiofiles>=23.0",
 ]
 signing = ["cryptography>=42.0.0"]
 
@@ -58,3 +61,8 @@ select = ["E", "F", "I", "B", "UP", "RUF"]
 [tool.mypy]
 python_version = "3.12"
 strict = true
+
+[[tool.mypy.overrides]]
+# vcon-lib ships no type information.
+module = ["vcon", "vcon.*"]
+ignore_missing_imports = true

src/__ADAPTER_PACKAGE__/config.py

@@ -10,6 +10,8 @@
 
 import yaml
 
+from .vcon_builder import LawfulBasisConfig
+
 _ENV_RE = re.compile(r"\$\{([A-Z0-9_]+)\}")
 
 
@@ -45,6 +47,22 @@ class WebhookConfig:
     dead_letter_path: str = "./dlq"
 
 
+@dataclass
+class ConserverConfig:
+    """Config for `delivery.mode: conserver` (direct POST to a vcon-server)."""
+
+    url: str = ""
+    token: str = ""
+    token_header: str = "x-conserver-api-token"
+    ingress_lists: list[str] = field(default_factory=list)
+
+
+@dataclass
+class DeliveryConfig:
+    # "webhook" (default, WebhookDelivery) or "conserver" (ConserverDelivery)
+    mode: str = "webhook"
+
+
 @dataclass
 class ServerConfig:
     host: str = "0.0.0.0"
@@ -62,7 +80,12 @@ class Config:
     adapter: AdapterConfig = field(default_factory=AdapterConfig)
     source: dict[str, Any] = field(default_factory=dict)
     vcon: dict[str, Any] = field(default_factory=dict)
+    lawful_basis: LawfulBasisConfig = field(
+        default_factory=lambda: LawfulBasisConfig(lawful_basis=None)
+    )
+    delivery: DeliveryConfig = field(default_factory=DeliveryConfig)
     webhook: WebhookConfig = field(default_factory=WebhookConfig)
+    conserver: ConserverConfig = field(default_factory=ConserverConfig)
     storage: dict[str, Any] = field(default_factory=dict)
     server: ServerConfig = field(default_factory=ServerConfig)
     logging: LoggingConfig = field(default_factory=LoggingConfig)
@@ -72,21 +95,33 @@ def load_config(path: str | Path = "config.yaml") -> Config:
     raw = yaml.safe_load(Path(path).read_text())
     raw = _substitute(raw)
 
+    vcon_raw = raw.get("vcon", {})
     webhook_raw = raw.get("webhook", {})
     endpoints = [WebhookEndpoint(**e) for e in webhook_raw.get("endpoints", [])]
     retry = webhook_raw.get("retry", {})
+    conserver_raw = raw.get("conserver", {})
 
     return Config(
         adapter=AdapterConfig(**raw.get("adapter", {})),
         source=raw.get("source", {}),
-        vcon=raw.get("vcon", {}),
+        vcon=vcon_raw,
+        # Env vars (LAWFUL_BASIS, LAWFUL_BASIS_*) override the YAML
+        # `vcon.lawful_basis:` block field-by-field; see LawfulBasisConfig.resolve.
+        lawful_basis=LawfulBasisConfig.resolve(yaml_block=vcon_raw.get("lawful_basis")),
+        delivery=DeliveryConfig(**raw.get("delivery", {})),
         webhook=WebhookConfig(
             endpoints=endpoints,
             retry_max_attempts=retry.get("max_attempts", 5),
             retry_initial_backoff_seconds=retry.get("initial_backoff_seconds", 1.0),
             retry_max_backoff_seconds=retry.get("max_backoff_seconds", 60.0),
             dead_letter_path=webhook_raw.get("dead_letter_path", "./dlq"),
         ),
+        conserver=ConserverConfig(
+            url=conserver_raw.get("url", ""),
+            token=conserver_raw.get("token", ""),
+            token_header=conserver_raw.get("token_header", "x-conserver-api-token"),
+            ingress_lists=list(conserver_raw.get("ingress_lists", [])),
+        ),
         storage=raw.get("storage", {}),
         server=ServerConfig(**raw.get("server", {})),
         logging=LoggingConfig(**raw.get("logging", {})),