MAINT: Fail CoPyRIT deployment pipeline when initialization fails

doc/gui/0_gui.md

@@ -581,7 +581,10 @@ API clients can use administrator-only `GET /api/config/runtime`, `POST /api/con
 returns HTTP 202 and is tracked in status. Outcomes distinguish busy, unsupported, version-conflict,
 invalid-configuration, and restart-required. Authenticated non-admin clients can read readiness and generation only
 at `GET /api/runtime`.
-`GET /api/health` reports server responsiveness, not runtime readiness.
+`GET /api/health` remains HTTP 200 for server responsiveness, even when runtime initialization fails.
+It preserves `status`, `timestamp`, and `service`, and includes `ready`, `state`, and the serving ACA
+`revision` (or `null` outside ACA) for deployment readiness checks. Responses are uncached and do not
+expose configuration or exception details.
 
 ---
 

infra/README.md

@@ -102,6 +102,8 @@ Community users can deploy `infrastructure.bicep` followed by `application.bicep
 
 ## Security
 
+The existing public, compatibility-neutral `/api/health` endpoint includes readiness, runtime state, and the serving revision alongside its health fields. It remains HTTP 200 when runtime initialization fails; initialization exception details remain in access-controlled logs.
+
 - **Authentication**: [MSAL](https://learn.microsoft.com/en-us/entra/msal/) [PKCE](https://oauth.net/2/pkce/) on the frontend (`@azure/msal-browser`) and public-client device-code authentication for the PyRIT CLI, backed by Microsoft Graph middleware on the backend. Both clients send delegated Graph tokens, and the backend authenticates them through Graph `/me`. These public-client flows require no client secrets or certificates.
 - **Authorization**: Entra group checks use `allowedGroupObjectIds` for application access and `adminGroupObjectId` for backend configuration routes. Requires delegated Graph `User.Read`; the backend calls `/me/checkMemberGroups` and compares the returned transitive memberships with the configured group IDs. Each security group must also be assigned to the enterprise app (see Prerequisites §3). Authenticated deployments require at least one allowed group and fail to start without one. `/api/health`, `/api/auth/config`, and `/api/media` are intentional public exceptions; other `/api` routes require authentication when auth is enabled. Successful identity and membership results are cached in-process for 60 seconds, keyed by a SHA-256 token digest, to reduce Graph latency and throttling. Bearer tokens themselves are not stored in the cache.
 - **Identity**: `deploy_instance.py` creates its user-assigned managed identity (UAMI) and grants AcrPull and Storage Blob Data Contributor before deploying Bicep. A direct Bicep deployment can create `<appName>-identity`, but the template creates no role assignments, so its first revision can remain unhealthy until required roles are granted and the revision is restarted. A healthy one-pass direct deployment uses an existing, pre-authorized UAMI. `AZURE_CLIENT_ID` is set to the UAMI's client ID so `DefaultAzureCredential` selects the correct identity.
@@ -441,7 +443,11 @@ The scripts share scope validation, preview, and readiness helpers in `deploymen
 
 **App-only is not image-only:** it reconciles the image and Bicep-defined application configuration, including environment variables, identity attachment, ingress, and scaling. It requires the same variable groups and existing topology as infrastructure-enabled deployment. Shared infrastructure resources are referenced rather than redeployed, and the app-only preview rejects writes outside the existing Container App. The existing ACA environment public/private access mode and Front Door resources are preserved.
 
-The script verifies the exact requested revision and its access mode: direct ACA `/api/health` when public access is enabled, or Front Door `/api/health` when public access is disabled. It never falls back from a failed private path to public access. A public-mode success is **not** certification of Private Link readiness. App-only data-plane verification has a five-minute budget after revision readiness.
+The script verifies the exact requested image and its access mode: direct ACA `/api/health` when public access is enabled, or Front Door `/api/health` when public access is disabled. It never falls back from a failed private path to public access. A public-mode success is **not** certification of Private Link readiness. App-only liveness verification has a five-minute budget after ACA revision readiness.
+
+The app stage then checks the readiness fields in `/api/health` through that same entry point, with a separate five-minute budget. This uncached endpoint preserves HTTP 200 and the existing `status`, `timestamp`, and `service` fields for backend availability checks, and adds `ready`, `state`, and the serving ACA `revision`. Deployment passes only on HTTP 200 with `ready: true` from the expected revision. The script waits for initialization, transient connection failures, or an older serving revision, but fails immediately if the expected revision reports `failed`, `restart-required`, or `stopping`, even though the endpoint remains HTTP 200. Invalid responses and timeouts fail deployment rather than being accepted as healthy. Infrastructure-only stages and Front Door probes continue to check backend availability without requiring runtime readiness.
+
+On readiness failure, the pipeline prints the revision, the failure state or timeout reason, and a revision-specific `az containerapp logs show` command. Use that command or **Azure Portal > Container App > Monitoring > Log stream**, selecting the failed revision and console logs, to find `PyRIT startup failed` and its traceback. For historical startup errors, use **Monitoring > Logs** in the connected Log Analytics workspace. Exception details remain in access-controlled logs, not the public health response or pipeline output. `/api/health` and configuration recovery remain available when runtime initialization fails; repair the underlying source, permissions, or data, then restart the backend and retry deployment.
 
 App deployment does not downgrade or create a database; application startup still follows the image's normal migration behavior. App-stage failures do not invoke infrastructure, image, or database rollback because migrations may make the previous image incompatible. The previous image digest is logged for an explicit recovery decision.
 

infra/pipelines/deploy_app.sh

@@ -118,6 +118,7 @@ main() {
     --name "$deployment_name" --resource-group "$PYRIT_DEPLOYMENT_RESOURCE_GROUP" \
     --template-file "$template_file" --mode Incremental --parameters "${parameters[@]}"
   verify_readiness 300 "$expected_public_access" "$immutable_image"
+  wait_for_runtime_readiness "$health_url" "$revision" 300
   echo "Deployment healthy: $revision; verified $health_url; ACA public access: $expected_public_access; egress IPv4: $egress_ip"
 }
 

infra/pipelines/deployment_common.sh

@@ -211,6 +211,71 @@ wait_for_http_health() {
   fi
 }
 
+runtime_readiness_error() {
+  local expected_revision=$1 reason=$2
+  printf 'Inspect startup logs: az containerapp logs show --name %q --resource-group %q --revision %q --type console --tail 100\n' \
+    "$PYRIT_APP_NAME" "$PYRIT_DEPLOYMENT_RESOURCE_GROUP" "$expected_revision"
+  echo "Or use Azure Portal -> Container App $PYRIT_APP_NAME -> Monitoring -> Log stream; select revision $expected_revision and console logs."
+  echo "For older startup errors, use Monitoring -> Logs in the connected Log Analytics workspace."
+  echo "Look for 'PyRIT startup failed' and its traceback. Initialization errors remain in access-controlled container logs."
+  deployment_error "PyRIT runtime readiness failed for revision $expected_revision: $reason"
+}
+
+wait_for_runtime_readiness() {
+  local readiness_url=$1 expected_revision=$2 timeout_seconds=$3
+  local deadline=$((SECONDS + timeout_seconds)) attempt=0
+  local remaining_seconds request_timeout sleep_seconds raw_response http_status body parsed
+  local response_revision state ready last_result="No readiness response"
+  while ((SECONDS < deadline)); do
+    ((attempt += 1))
+    remaining_seconds=$((deadline - SECONDS))
+    request_timeout=$((remaining_seconds < 30 ? remaining_seconds : 30))
+    if raw_response=$(curl \
+      --silent --show-error --header 'Cache-Control: no-cache' --write-out $'\n%{http_code}' \
+      --max-time "$request_timeout" "$readiness_url"); then
+      http_status=${raw_response##*$'\n'}
+      body=${raw_response%$'\n'*}
+      if [[ "$http_status" != "200" ]]; then
+        last_result="Unexpected readiness HTTP status $http_status"
+      elif ! parsed=$(jq -ser '
+        if length != 1 then error("Invalid readiness response") else .[0] end
+        | if type != "object"
+          or (.ready | type) != "boolean"
+          or (.state | type) != "string"
+          or (.revision | type) != "string"
+        then error("Invalid readiness response")
+        elif (.revision | test("^[a-z][a-z0-9-]*--[a-z0-9-]+$") | not)
+          or (.state as $state | ["ready", "initializing", "failed", "restart-required", "stopping"] | index($state) == null)
+          or .ready != (.state == "ready")
+        then error("Inconsistent readiness response")
+        else [.revision, .state, (.ready | tostring)] | @tsv
+        end' <<< "$body" 2> /dev/null); then
+        last_result="Invalid or inconsistent readiness response"
+      else
+        IFS=$'\t' read -r response_revision state ready <<< "$parsed"
+        if [[ "$response_revision" != "$expected_revision" ]]; then
+          last_result="Waiting for revision $expected_revision; received $response_revision"
+        elif [[ "$state" == "ready" && "$ready" == "true" ]]; then
+          echo "PyRIT runtime is ready on revision $expected_revision."
+          return 0
+        elif [[ "$state" =~ ^(failed|restart-required|stopping)$ ]]; then
+          runtime_readiness_error "$expected_revision" "Runtime state is $state"
+        else
+          last_result="Runtime state is $state (HTTP $http_status)"
+        fi
+      fi
+    else
+      last_result="Readiness endpoint connection failed"
+    fi
+    echo "PyRIT runtime readiness attempt $attempt: $last_result"
+    remaining_seconds=$((deadline - SECONDS))
+    ((remaining_seconds > 0)) || break
+    sleep_seconds=$((remaining_seconds < 5 ? remaining_seconds : 5))
+    sleep "$sleep_seconds"
+  done
+  runtime_readiness_error "$expected_revision" "Timed out after ${timeout_seconds}s. $last_result"
+}
+
 verify_readiness() {
   local timeout_seconds=$1 expected_public_access=$2 expected_image=$3 unchanged_revision=${4:-}
   local public_network_access

pyrit/backend/routes/health.py

@@ -5,21 +5,15 @@
 Health check endpoints.
 """
 
+import os
 from datetime import UTC, datetime
 
-from fastapi import APIRouter, Request
+from fastapi import APIRouter, Request, Response
 
 router = APIRouter()
 
 
-@router.get("/runtime")
-async def runtime_readiness_async(request: Request) -> dict[str, str | bool]:
-    """
-    Expose readiness and generation without administrative details.
-
-    Returns:
-        dict[str, str | bool]: Lightweight runtime readiness.
-    """
+def _runtime_readiness(request: Request) -> dict[str, str | bool]:
     runtime = getattr(request.app.state, "runtime_lifecycle", None)
     if runtime is None:
         return {"ready": False, "state": "failed", "generation": ""}
@@ -30,21 +24,39 @@ async def runtime_readiness_async(request: Request) -> dict[str, str | bool]:
     }
 
 
+@router.get("/runtime")
+async def runtime_readiness_async(request: Request) -> dict[str, str | bool]:
+    """
+    Expose readiness and generation without administrative details.
+
+    Returns:
+        dict[str, str | bool]: Lightweight runtime readiness.
+    """
+    return _runtime_readiness(request)
+
+
 @router.get("/health")
-async def health_check_async() -> dict[str, str]:
+async def health_check_async(*, request: Request, response: Response) -> dict[str, str | bool | None]:
     """
     Check the health status of the backend service.
 
     This endpoint must remain lightweight, auth-free, and database-free.
     The frontend connection health monitor polls it every 60 seconds with a
     5-second timeout to detect backend availability. Adding authentication,
     database queries, or heavy computation here will break that contract.
+    HTTP 200 reports backend availability even when PyRIT initialization fails;
+    deployment checks use the readiness fields instead.
 
     Returns:
-        dict: Health status information including timestamp.
+        dict[str, str | bool | None]: Health status, timestamp, runtime readiness, and serving ACA revision.
     """
+    readiness = _runtime_readiness(request)
+    response.headers["Cache-Control"] = "no-store"
     return {
         "status": "healthy",
         "timestamp": datetime.now(UTC).isoformat(),
         "service": "pyrit-backend",
+        "ready": readiness["ready"],
+        "state": readiness["state"],
+        "revision": os.getenv("CONTAINER_APP_REVISION"),
     }

tests/unit/backend/test_compatibility.py

@@ -80,6 +80,7 @@ def graph_user() -> AuthenticatedUser:
 
 def test_openapi_documents_the_required_header_only_on_business_operations() -> None:
     schema = app.openapi()
+    assert "/api/ready" not in schema["paths"]
     neutral_paths = {"/api/health", "/api/auth/config", "/api/version", "/api/media"}
     business_operations = set()
     for path, path_item in schema["paths"].items():
@@ -236,7 +237,7 @@ def test_public_neutral_routes_bypass_guard_and_auth(guarded_client: TestClient,
     headers = {_compatibility.COMPATIBILITY_HEADER: marker} if marker is not None else {}
     with patch.object(EntraAuthMiddleware, "_authenticate_request_async", new_callable=AsyncMock) as authenticate:
         response = guarded_client.get(path, headers=headers)
-    assert response.status_code == (422 if path == "/api/media" else 200)
+    assert response.status_code == {"/api/media": 422}.get(path, 200)
     authenticate.assert_not_awaited()
 
 
@@ -246,6 +247,9 @@ def test_public_neutral_routes_bypass_guard_and_auth(guarded_client: TestClient,
         "/api",
         "/api/health/",
         "/api/health/extra",
+        "/api/ready",
+        "/api/ready/",
+        "/api/ready/extra",
         "/api/auth/config/",
         "/api/version/",
         "/api/media/",
@@ -258,6 +262,36 @@ def test_neutral_exemptions_are_exact(guarded_client: TestClient, path: str) ->
     assert response.json()["type"] == _compatibility.INVALID_COMPATIBILITY_TYPE
 
 
+@pytest.mark.parametrize("state", ["ready", "initializing", "failed", "restart-required", "stopping"])
+def test_health_is_public_without_exposing_runtime_details(*, guarded_app: FastAPI, state: str, tmp_path: Path) -> None:
+    runtime = RuntimeLifecycle(
+        app=guarded_app, source=ConfigurationFileService(config_file_value=str(tmp_path / "config.yaml"))
+    )
+    runtime.state = state
+    runtime.message = "private failure details"
+    runtime.generation = "private generation"
+    guarded_app.state.runtime_lifecycle = runtime
+    with (
+        patch.dict("os.environ", {"CONTAINER_APP_REVISION": "copyrit-test--new"}),
+        patch.object(EntraAuthMiddleware, "_authenticate_request_async", new_callable=AsyncMock) as authenticate,
+        TestClient(guarded_app) as client,
+    ):
+        response = client.get("/api/health", headers={_compatibility.COMPATIBILITY_HEADER: "invalid"})
+    assert response.status_code == 200
+    data = response.json()
+    assert isinstance(data["timestamp"], str)
+    assert data == {
+        "status": "healthy",
+        "timestamp": data["timestamp"],
+        "service": "pyrit-backend",
+        "ready": state == "ready",
+        "state": state,
+        "revision": "copyrit-test--new",
+    }
+    assert response.headers["cache-control"] == "no-store"
+    authenticate.assert_not_awaited()
+
+
 @pytest.mark.parametrize("path", ["/", "/assets/app.js", "/apikeys", "/docs"])
 def test_non_api_routes_are_not_guarded(guarded_client: TestClient, path: str) -> None:
     response = guarded_client.get(path)

tests/unit/backend/test_runtime_lifecycle.py

@@ -7,6 +7,7 @@
 import os
 from collections.abc import Generator
 from contextlib import ExitStack
+from datetime import UTC, datetime
 from pathlib import Path
 from threading import Event
 from unittest.mock import AsyncMock, MagicMock, call, patch
@@ -94,6 +95,54 @@ async def test_runtime_readiness_reports_state_async(*, runtime: RuntimeLifecycl
     }
 
 
+@pytest.mark.parametrize("state", ["initializing", "ready", "failed", "restart-required", "stopping", None])
+async def test_health_reports_readiness_and_revision_without_changing_liveness_async(
+    *, runtime: RuntimeLifecycle, state: str | None
+) -> None:
+    if state is None:
+        del runtime.app.state.runtime_lifecycle
+    else:
+        runtime.state = state
+    revision = "copyrit-test--new"
+    with patch.dict(os.environ, {"CONTAINER_APP_REVISION": revision}):
+        async with httpx.AsyncClient(transport=httpx.ASGITransport(app=runtime.app), base_url="http://test") as client:
+            response = await client.get("/api/health")
+    assert response.status_code == 200
+    assert response.headers["cache-control"] == "no-store"
+    data = response.json()
+    assert datetime.fromisoformat(data["timestamp"]).tzinfo == UTC
+    assert data == {
+        "status": "healthy",
+        "timestamp": data["timestamp"],
+        "service": "pyrit-backend",
+        "ready": state == "ready",
+        "state": state if state is not None else "failed",
+        "revision": revision,
+    }
+
+
+async def test_health_does_not_invent_a_revision_async(runtime: RuntimeLifecycle) -> None:
+    with patch.dict(os.environ, {}, clear=True):
+        async with httpx.AsyncClient(transport=httpx.ASGITransport(app=runtime.app), base_url="http://test") as client:
+            response = await client.get("/api/health")
+    assert response.status_code == 200
+    assert response.json()["revision"] is None
+
+
+async def test_failed_startup_keeps_liveness_and_configuration_recovery_async(runtime: RuntimeLifecycle) -> None:
+    with patch.object(runtime, "_load_async", AsyncMock(side_effect=ValueError("private startup error"))):
+        await runtime.startup_async()
+    async with httpx.AsyncClient(transport=httpx.ASGITransport(app=runtime.app), base_url="http://test") as client:
+        response = await client.get("/api/health")
+        saved_configuration = await client.get("/api/config")
+    assert response.status_code == 200
+    assert response.json()["status"] == "healthy"
+    assert response.json()["ready"] is False
+    assert response.json()["state"] == "restart-required"
+    assert "private startup error" not in response.text
+    assert saved_configuration.status_code == 200
+
+
 @pytest.mark.parametrize(
     "failures", [(), ("scenarios",), ("services",), ("memory",), ("scenarios", "services", "memory")]
 )

tests/unit/cli/test_api_client.py

@@ -568,10 +568,16 @@ def test_get_client_raises_when_not_opened():
 # ---------------------------------------------------------------------------
 
 
-async def test_health_check_returns_true_on_200(client, mock_httpx_client):
+@pytest.mark.parametrize(
+    "readiness_fields",
+    [{}, {"ready": False, "state": "restart-required", "revision": "copyrit-test--new"}],
+)
+async def test_health_check_returns_true_on_200(
+    *, client: PyRITApiClient, mock_httpx_client: MagicMock, readiness_fields: dict[str, str | bool | None]
+) -> None:
     mock_httpx_client.get.return_value = _make_response(
         status_code=200,
-        json_data={"status": "healthy", "service": "pyrit-backend"},
+        json_data={"status": "healthy", "service": "pyrit-backend"} | readiness_fields,
     )
     assert await client.health_check_async() is True
     mock_httpx_client.get.assert_awaited_once_with("/api/health")