pipeline fail and give insight when pyrit initialization fails

Author: jsong468

infra/README.md

@@ -102,6 +102,8 @@ Community users can deploy `infrastructure.bicep` followed by `application.bicep
 
 ## Security
 
+The deployment-only `/api/ready` endpoint is also a public, compatibility-neutral exception. It exposes only readiness, runtime state, and the serving revision; initialization exception details remain in access-controlled logs.
+
 - **Authentication**: [MSAL](https://learn.microsoft.com/en-us/entra/msal/) [PKCE](https://oauth.net/2/pkce/) on the frontend (`@azure/msal-browser`) and public-client device-code authentication for the PyRIT CLI, backed by Microsoft Graph middleware on the backend. Both clients send delegated Graph tokens, and the backend authenticates them through Graph `/me`. These public-client flows require no client secrets or certificates.
 - **Authorization**: Entra group checks use `allowedGroupObjectIds` for application access and `adminGroupObjectId` for backend configuration routes. Requires delegated Graph `User.Read`; the backend calls `/me/checkMemberGroups` and compares the returned transitive memberships with the configured group IDs. Each security group must also be assigned to the enterprise app (see Prerequisites §3). Authenticated deployments require at least one allowed group and fail to start without one. `/api/health`, `/api/auth/config`, and `/api/media` are intentional public exceptions; other `/api` routes require authentication when auth is enabled. Successful identity and membership results are cached in-process for 60 seconds, keyed by a SHA-256 token digest, to reduce Graph latency and throttling. Bearer tokens themselves are not stored in the cache.
 - **Identity**: `deploy_instance.py` creates its user-assigned managed identity (UAMI) and grants AcrPull and Storage Blob Data Contributor before deploying Bicep. A direct Bicep deployment can create `<appName>-identity`, but the template creates no role assignments, so its first revision can remain unhealthy until required roles are granted and the revision is restarted. A healthy one-pass direct deployment uses an existing, pre-authorized UAMI. `AZURE_CLIENT_ID` is set to the UAMI's client ID so `DefaultAzureCredential` selects the correct identity.
@@ -441,7 +443,11 @@ The scripts share scope validation, preview, and readiness helpers in `deploymen
 
 **App-only is not image-only:** it reconciles the image and Bicep-defined application configuration, including environment variables, identity attachment, ingress, and scaling. It requires the same variable groups and existing topology as infrastructure-enabled deployment. Shared infrastructure resources are referenced rather than redeployed, and the app-only preview rejects writes outside the existing Container App. The existing ACA environment public/private access mode and Front Door resources are preserved.
 
-The script verifies the exact requested revision and its access mode: direct ACA `/api/health` when public access is enabled, or Front Door `/api/health` when public access is disabled. It never falls back from a failed private path to public access. A public-mode success is **not** certification of Private Link readiness. App-only data-plane verification has a five-minute budget after revision readiness.
+The script verifies the exact requested image and its access mode: direct ACA `/api/health` when public access is enabled, or Front Door `/api/health` when public access is disabled. It never falls back from a failed private path to public access. A public-mode success is **not** certification of Private Link readiness. App-only liveness verification has a five-minute budget after ACA revision readiness.
+
+The app stage then checks `/api/ready` through that same entry point, with a separate five-minute budget. This minimal public endpoint returns only runtime readiness, state, and the serving ACA revision, with caching disabled. Deployment passes only on HTTP 200 with `ready: true` from the expected revision. The script waits for initialization, transient connection failures, or an older serving revision, but fails immediately if the expected revision reports `failed`, `restart-required`, or `stopping`. Invalid responses and timeouts fail deployment rather than being accepted as healthy. Infrastructure-only stages continue to use `/api/health` and do not require runtime readiness.
+
+On readiness failure, the pipeline prints the revision, the failure state or timeout reason, and a revision-specific `az containerapp logs show` command. Use that command or **Azure Portal > Container App > Monitoring > Log stream**, selecting the failed revision and console logs, to find `PyRIT startup failed` and its traceback. For historical startup errors, use **Monitoring > Logs** in the connected Log Analytics workspace. Exception details remain in access-controlled logs, not the public readiness endpoint or pipeline output. `/api/health` and configuration recovery remain available when runtime initialization fails; repair the underlying source, permissions, or data, then restart the backend and retry deployment.
 
 App deployment does not downgrade or create a database; application startup still follows the image's normal migration behavior. App-stage failures do not invoke infrastructure, image, or database rollback because migrations may make the previous image incompatible. The previous image digest is logged for an explicit recovery decision.
 

infra/pipelines/deploy_app.sh

@@ -118,6 +118,7 @@ main() {
     --name "$deployment_name" --resource-group "$PYRIT_DEPLOYMENT_RESOURCE_GROUP" \
     --template-file "$template_file" --mode Incremental --parameters "${parameters[@]}"
   verify_readiness 300 "$expected_public_access" "$immutable_image"
+  wait_for_runtime_readiness "${health_url%/api/health}/api/ready" "$revision" 300
   echo "Deployment healthy: $revision; verified $health_url; ACA public access: $expected_public_access; egress IPv4: $egress_ip"
 }
 

infra/pipelines/deployment_common.sh

@@ -211,6 +211,71 @@ wait_for_http_health() {
   fi
 }
 
+runtime_readiness_error() {
+  local expected_revision=$1 reason=$2
+  printf 'Inspect startup logs: az containerapp logs show --name %q --resource-group %q --revision %q --type console --tail 100\n' \
+    "$PYRIT_APP_NAME" "$PYRIT_DEPLOYMENT_RESOURCE_GROUP" "$expected_revision"
+  echo "Or use Azure Portal -> Container App $PYRIT_APP_NAME -> Monitoring -> Log stream; select revision $expected_revision and console logs."
+  echo "For older startup errors, use Monitoring -> Logs in the connected Log Analytics workspace."
+  echo "Look for 'PyRIT startup failed' and its traceback. Initialization errors remain in access-controlled container logs."
+  deployment_error "PyRIT runtime readiness failed for revision $expected_revision: $reason"
+}
+
+wait_for_runtime_readiness() {
+  local readiness_url=$1 expected_revision=$2 timeout_seconds=$3
+  local deadline=$((SECONDS + timeout_seconds)) attempt=0
+  local remaining_seconds request_timeout sleep_seconds raw_response http_status body parsed
+  local response_revision state ready last_result="No readiness response"
+  while ((SECONDS < deadline)); do
+    ((attempt += 1))
+    remaining_seconds=$((deadline - SECONDS))
+    request_timeout=$((remaining_seconds < 30 ? remaining_seconds : 30))
+    if raw_response=$(curl \
+      --silent --show-error --header 'Cache-Control: no-cache' --write-out $'\n%{http_code}' \
+      --max-time "$request_timeout" "$readiness_url"); then
+      http_status=${raw_response##*$'\n'}
+      body=${raw_response%$'\n'*}
+      if [[ "$http_status" != "200" && "$http_status" != "503" ]]; then
+        last_result="Unexpected readiness HTTP status $http_status"
+      elif ! parsed=$(jq -ser '
+        if length != 1 then error("Invalid readiness response") else .[0] end
+        | if type != "object"
+          or (.ready | type) != "boolean"
+          or (.state | type) != "string"
+          or (.revision | type) != "string"
+        then error("Invalid readiness response")
+        elif (.revision | test("^[a-z][a-z0-9-]*--[a-z0-9-]+$") | not)
+          or (.state as $state | ["ready", "initializing", "failed", "restart-required", "stopping"] | index($state) == null)
+          or .ready != (.state == "ready")
+        then error("Inconsistent readiness response")
+        else [.revision, .state, (.ready | tostring)] | @tsv
+        end' <<< "$body" 2> /dev/null); then
+        last_result="Invalid or inconsistent readiness response"
+      else
+        IFS=$'\t' read -r response_revision state ready <<< "$parsed"
+        if [[ "$response_revision" != "$expected_revision" ]]; then
+          last_result="Waiting for revision $expected_revision; received $response_revision"
+        elif [[ "$http_status" == "200" && "$state" == "ready" && "$ready" == "true" ]]; then
+          echo "PyRIT runtime is ready on revision $expected_revision."
+          return 0
+        elif [[ "$http_status" == "503" && "$state" =~ ^(failed|restart-required|stopping)$ ]]; then
+          runtime_readiness_error "$expected_revision" "Runtime state is $state"
+        else
+          last_result="Runtime state is $state (HTTP $http_status)"
+        fi
+      fi
+    else
+      last_result="Readiness endpoint connection failed"
+    fi
+    echo "PyRIT runtime readiness attempt $attempt: $last_result"
+    remaining_seconds=$((deadline - SECONDS))
+    ((remaining_seconds > 0)) || break
+    sleep_seconds=$((remaining_seconds < 5 ? remaining_seconds : 5))
+    sleep "$sleep_seconds"
+  done
+  runtime_readiness_error "$expected_revision" "Timed out after ${timeout_seconds}s. $last_result"
+}
+
 verify_readiness() {
   local timeout_seconds=$1 expected_public_access=$2 expected_image=$3 unchanged_revision=${4:-}
   local public_network_access

pyrit/backend/middleware/auth.py

@@ -80,6 +80,7 @@ class EntraAuthMiddleware(BaseHTTPMiddleware):
     # Paths that bypass authentication
     _PUBLIC_PATHS: ClassVar[set[str]] = {
         "/api/health",
+        "/api/ready",
         "/api/auth/config",
         "/api/media",
     }

pyrit/backend/middleware/compatibility.py

@@ -18,7 +18,7 @@ class CompatibilityMiddleware:
     """Enforce one exact packaged compatibility identity on business API requests."""
 
     _NEUTRAL_PATHS: ClassVar[frozenset[str]] = frozenset(
-        {"/api/health", "/api/auth/config", "/api/version", "/api/media"}
+        {"/api/health", "/api/ready", "/api/auth/config", "/api/version", "/api/media"}
     )
 
     def __init__(self, app: ASGIApp) -> None:

pyrit/backend/middleware/runtime.py

@@ -33,6 +33,7 @@ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
             or path
             in (
                 "/api/health",
+                "/api/ready",
                 "/api/runtime",
                 "/api/version",
                 "/api/auth/config",

pyrit/backend/routes/health.py

@@ -5,13 +5,25 @@
 Health check endpoints.
 """
 
+import os
 from datetime import UTC, datetime
 
-from fastapi import APIRouter, Request
+from fastapi import APIRouter, Request, Response
 
 router = APIRouter()
 
 
+def _runtime_readiness(request: Request) -> dict[str, str | bool]:
+    runtime = getattr(request.app.state, "runtime_lifecycle", None)
+    if runtime is None:
+        return {"ready": False, "state": "failed", "generation": ""}
+    return {
+        "ready": runtime.reported_state == "ready",
+        "state": runtime.reported_state,
+        "generation": runtime.generation,
+    }
+
+
 @router.get("/runtime")
 async def runtime_readiness_async(request: Request) -> dict[str, str | bool]:
     """
@@ -20,13 +32,24 @@ async def runtime_readiness_async(request: Request) -> dict[str, str | bool]:
     Returns:
         dict[str, str | bool]: Lightweight runtime readiness.
     """
-    runtime = getattr(request.app.state, "runtime_lifecycle", None)
-    if runtime is None:
-        return {"ready": False, "state": "failed", "generation": ""}
+    return _runtime_readiness(request)
+
+
+@router.get("/ready", responses={503: {"description": "PyRIT runtime is not ready"}})
+async def deployment_readiness_async(*, request: Request, response: Response) -> dict[str, str | bool | None]:
+    """
+    Report uncached deployment readiness without configuration or exception details.
+
+    Returns:
+        dict[str, str | bool | None]: Runtime readiness and the serving ACA revision, when available.
+    """
+    readiness = _runtime_readiness(request)
+    response.status_code = 200 if readiness["ready"] else 503
+    response.headers["Cache-Control"] = "no-store"
     return {
-        "ready": runtime.reported_state == "ready",
-        "state": runtime.reported_state,
-        "generation": runtime.generation,
+        "ready": readiness["ready"],
+        "state": readiness["state"],
+        "revision": os.getenv("CONTAINER_APP_REVISION"),
     }
 
 

tests/unit/backend/test_compatibility.py

@@ -80,7 +80,7 @@ def graph_user() -> AuthenticatedUser:
 
 def test_openapi_documents_the_required_header_only_on_business_operations() -> None:
     schema = app.openapi()
-    neutral_paths = {"/api/health", "/api/auth/config", "/api/version", "/api/media"}
+    neutral_paths = {"/api/health", "/api/ready", "/api/auth/config", "/api/version", "/api/media"}
     business_operations = set()
     for path, path_item in schema["paths"].items():
         for method, operation in path_item.items():
@@ -230,13 +230,13 @@ def test_matching_identity_does_not_bypass_group_authorization(
     assert guarded_app.state.effects == []
 
 
-@pytest.mark.parametrize("path", ["/api/health", "/api/auth/config", "/api/media"])
+@pytest.mark.parametrize("path", ["/api/health", "/api/ready", "/api/auth/config", "/api/media"])
 @pytest.mark.parametrize("marker", [None, "invalid", "0.14.0+g" + "b" * 40])
 def test_public_neutral_routes_bypass_guard_and_auth(guarded_client: TestClient, path: str, marker: str | None) -> None:
     headers = {_compatibility.COMPATIBILITY_HEADER: marker} if marker is not None else {}
     with patch.object(EntraAuthMiddleware, "_authenticate_request_async", new_callable=AsyncMock) as authenticate:
         response = guarded_client.get(path, headers=headers)
-    assert response.status_code == (422 if path == "/api/media" else 200)
+    assert response.status_code == {"/api/media": 422, "/api/ready": 503}.get(path, 200)
     authenticate.assert_not_awaited()
 
 
@@ -246,6 +246,8 @@ def test_public_neutral_routes_bypass_guard_and_auth(guarded_client: TestClient,
         "/api",
         "/api/health/",
         "/api/health/extra",
+        "/api/ready/",
+        "/api/ready/extra",
         "/api/auth/config/",
         "/api/version/",
         "/api/media/",
@@ -258,6 +260,29 @@ def test_neutral_exemptions_are_exact(guarded_client: TestClient, path: str) ->
     assert response.json()["type"] == _compatibility.INVALID_COMPATIBILITY_TYPE
 
 
+@pytest.mark.parametrize("state", ["ready", "initializing", "failed", "restart-required", "stopping"])
+def test_deployment_readiness_is_public_without_exposing_runtime_details(
+    *, guarded_app: FastAPI, state: str, tmp_path: Path
+) -> None:
+    runtime = RuntimeLifecycle(
+        app=guarded_app, source=ConfigurationFileService(config_file_value=str(tmp_path / "config.yaml"))
+    )
+    runtime.state = state
+    runtime.message = "private failure details"
+    runtime.generation = "private generation"
+    guarded_app.state.runtime_lifecycle = runtime
+    with (
+        patch.dict("os.environ", {"CONTAINER_APP_REVISION": "copyrit-test--new"}),
+        patch.object(EntraAuthMiddleware, "_authenticate_request_async", new_callable=AsyncMock) as authenticate,
+        TestClient(guarded_app) as client,
+    ):
+        response = client.get("/api/ready", headers={_compatibility.COMPATIBILITY_HEADER: "invalid"})
+    assert response.status_code == (200 if state == "ready" else 503)
+    assert response.json() == {"ready": state == "ready", "state": state, "revision": "copyrit-test--new"}
+    assert response.headers["cache-control"] == "no-store"
+    authenticate.assert_not_awaited()
+
+
 @pytest.mark.parametrize("path", ["/", "/assets/app.js", "/apikeys", "/docs"])
 def test_non_api_routes_are_not_guarded(guarded_client: TestClient, path: str) -> None:
     response = guarded_client.get(path)

tests/unit/backend/test_runtime_lifecycle.py

@@ -94,6 +94,51 @@ async def test_runtime_readiness_reports_state_async(*, runtime: RuntimeLifecycl
     }
 
 
+@pytest.mark.parametrize("state", ["initializing", "ready", "failed", "restart-required", "stopping", None])
+async def test_deployment_readiness_reports_state_and_revision_async(
+    *, runtime: RuntimeLifecycle, state: str | None
+) -> None:
+    if state is None:
+        del runtime.app.state.runtime_lifecycle
+    else:
+        runtime.state = state
+    revision = "copyrit-test--new"
+    with patch.dict(os.environ, {"CONTAINER_APP_REVISION": revision}):
+        async with httpx.AsyncClient(transport=httpx.ASGITransport(app=runtime.app), base_url="http://test") as client:
+            response = await client.get("/api/ready")
+            liveness = await client.get("/api/health")
+    assert response.status_code == (200 if state == "ready" else 503)
+    assert response.headers["cache-control"] == "no-store"
+    assert response.json() == {
+        "ready": state == "ready",
+        "state": state if state is not None else "failed",
+        "revision": revision,
+    }
+    assert liveness.status_code == 200
+
+
+async def test_deployment_readiness_does_not_invent_a_revision_async(runtime: RuntimeLifecycle) -> None:
+    with patch.dict(os.environ, {}, clear=True):
+        async with httpx.AsyncClient(transport=httpx.ASGITransport(app=runtime.app), base_url="http://test") as client:
+            response = await client.get("/api/ready")
+    assert response.status_code == 200
+    assert response.json()["revision"] is None
+
+
+async def test_failed_startup_keeps_liveness_and_configuration_recovery_async(runtime: RuntimeLifecycle) -> None:
+    with patch.object(runtime, "_load_async", AsyncMock(side_effect=ValueError("private startup error"))):
+        await runtime.startup_async()
+    async with httpx.AsyncClient(transport=httpx.ASGITransport(app=runtime.app), base_url="http://test") as client:
+        response = await client.get("/api/ready")
+        liveness = await client.get("/api/health")
+        saved_configuration = await client.get("/api/config")
+    assert response.status_code == 503
+    assert response.json()["state"] == "restart-required"
+    assert "private startup error" not in response.text
+    assert liveness.status_code == 200
+    assert saved_configuration.status_code == 200
+
+
 @pytest.mark.parametrize(
     "failures", [(), ("scenarios",), ("services",), ("memory",), ("scenarios", "services", "memory")]
 )