Skip to content

error-policy.md

Latest commit

 

History

History
170 lines (125 loc) · 13.6 KB

error-policy.md

File metadata and controls

170 lines (125 loc) · 13.6 KB
Raw

Error Policy

This document specifies the authoritative error handling model, recovery boundaries, and invariant enforcement protocols for the WebAI-to-API hardened Playwright runtime.

1. Purpose & Scope

The Error Policy defines the deterministic behavior of the runtime when encountering failures. It establishes the rules for error classification, recovery authority, and lifecycle-safe escalation.

  • Governing Semantics: This document governs how exceptions are categorized, propagated, and handled across all runtime layers.
  • Authority Boundaries: It defines which components own the right to execute recovery or initiate shutdown.
  • Relationship: This is a core runtime contract. If conflicts occur, more specialized runtime contracts (e.g., concurrency-model.md) take precedence for subsystem-specific behavior.

2. Core Error Handling Principles

The following principles are mandatory invariants for the runtime:

  • Fail-Fast Invariant Enforcement: Invariant violations (e.g., lock order, lease validity) must result in immediate failure rather than attempted workaround.
  • No Silent Recovery: Recovery actions (e.g., context recreation) must be explicitly logged and never performed silently.
  • Deterministic Failure Semantics: Similar failure conditions must result in identical, predictable state transitions.
  • Authority-Based Recovery: Only designated components may execute recovery logic. Providers identify and escalate; Sessions and Engines execute.
  • Cancellation-Safe Cleanup: Teardown logic must be protected from CancelledError via asyncio.shield to prevent resource leaks.
  • Isolation over Aggressive Reuse: Poisoned pages or corrupted states must be permanently invalidated.
  • No Resurrection after Terminal Shutdown: Once a terminal shutdown begins, all recovery attempts must fail immediately.
  • Explicit Error Classification: Failures must be categorized by their scope (Request, Tab, Session, Engine) to ensure appropriate escalation.
  • Best-Effort Cleanup: Partial cleanup failures must not suppress fatal state or prevent the release of critical resources (locks, permits).

3. Error Classification Model

Failures are formally classified into four levels of impact:

3.1 Recoverable Errors

  • Scope: Transient UI or transport instability where the browser process remains healthy.
  • Examples: Transient selector failures, navigation timeouts, temporary UI desync, authentication expiry.
  • Protocol: Providers detect and escalate to ProviderSession.ensure_healthy(). The session performs authoritative recovery (e.g., context refresh or tab purge).

3.2 Request-Scoped Terminal Errors

  • Scope: Failures that invalidate a specific request but do not poison the broader session or tab.
  • Examples: Bounded queue overflow, out-of-order stream events, invalid request-scoped bridge state, lease invalidation during an active operation.
  • Protocol: The request must fail immediately. The ManagedPage performs deterministic cleanup. Reuse eligibility of the underlying PersistentTab is determined exclusively by session-level poisoning and lease validity rules.

3.3 Session-Scoped Failures

  • Scope: Structural failures within a specific ProviderSession.
  • Examples: keepalive_page loss, persistent context corruption, unrecoverable renderer instability.
  • Protocol: The session is treated as degraded and invalidates all active leases before executing authoritative session-scoped recovery according to lifecycle ownership rules.

3.4 Engine-Scoped Fatal Errors

  • Scope: Irreversible loss of the global browser process or user interface.
  • Examples: Manual window closure, browser process disconnect, fatal BrowserEngine state corruption, generation rollover invalidation.
  • Protocol: The runtime enters an irreversible Terminal Shutdown state. All future recovery attempts and new requests MUST fail fast with terminal shutdown semantics.

3.5 Browser Resource Loss During a Request

  • BrowserDisconnectedError represents terminal browser/page/context loss for the active request. The terminal signal wakes request waits immediately and preserves the existing HTTP 502 mapping.
  • Before streaming response headers are sent, this error is mapped through the normal HTTP 502 policy. After SSE status 200 is sent, the status cannot be rewritten; the stream iterator terminates cleanly instead.
  • Post-header BrowserDisconnectedError, BrowserShuttingDownError, BrowserGenerationMismatchError, QueueOverflowError, and ConversationBusyError must not escape into Starlette/AnyIO as an ASGI application error and must not emit [DONE].
  • Uvicorn graceful-timeout cancellation (CancelledError: Task cancelled, timeout graceful shutdown exceeded) while app.shutdown is requested is expected shutdown termination, not application failure. The outermost shutdown boundary completes it as 503 pre-header or truncated streaming post-header without ERROR traceback or 500.
  • If a request already recorded a terminal BrowserDisconnectedError or BrowserShuttingDownError, that recorded terminal state takes precedence over a later raw Playwright close error caused by the same resource loss during lazy conversation registration.
  • Raw Playwright errors without recorded terminal browser-disconnect state retain existing classification, including the existing 503 mapping for a closed page/context.
  • A true timeout remains a timeout and maps through existing timeout policy; browser death is not a timeout fallback.

3.6 Conversation Contention

  • A competing request registering the same conversation_id as an already-owned active conversation is rejected with ConversationBusyError.
  • Pre-header: ConversationBusyError maps to HTTP 409 Conflict before streaming headers are sent.
  • Post-header: ConversationBusyError is an expected terminal request condition handled inside the stream iterator; the stream terminates without [DONE] and without escaping to ASGI.

4. Recovery Authority Boundaries

Ownership of recovery execution is strictly partitioned:

Component Responsibility Recovery Capability
Provider Detect & Escalate Identify failures; signal recovery needs to Session.
ProviderSession Recovery Authority Recreate contexts; invalidate tabs; restore session-scoped state.
BrowserEngine Shutdown Authority Manage process lifecycle; initiate irreversible terminal shutdown.

Invariants:

  • Providers MUST NEVER recreate browser contexts or processes directly.
  • Recovery logic MUST NEVER bypass the authoritative lifecycle ownership of the Session or Engine.
  • Shutdown authority belongs exclusively to BrowserEngine.

5. Runtime Exception Semantics

The runtime implements a dedicated, minimal, lifecycle-scoped exception hierarchy under src/app/services/browser/errors.py:

  • WebAIRuntimeError: Base exception for all browser and provider runtime failures.
    • BrowserEngineError: Base exception for BrowserEngine-scoped errors.
      • BrowserShuttingDownError: Raised when an operation is attempted while the browser engine is shutting down.
      • BrowserDisconnectedError: Raised when the underlying browser process unexpectedly disconnects.
      • BrowserGenerationMismatchError: Raised when operating on a browser resource from a previous/stale process generation.
    • SessionError: Base exception for ProviderSession-scoped errors.
      • SessionNotAliveError: Raised when a liveness probe on the context's keepalive page fails or provider authentication loss occurs.
    • RequestError: Base exception for request-scoped failures.
      • LeaseInvalidatedError: Raised when attempting to operate on an invalidated or stale tab lease.
      • QueueOverflowError: Raised when the request-scoped event buffer reaches saturation limit during bridge enqueuing.
      • ConversationBusyError: Raised when a competing request registers the same active conversation_id that already has an owner.

Invariant: Exception classification MUST reflect the runtime scope and recovery authority boundaries. Exception hierarchy is strictly scoped to lifecycle and resource-management concerns; it does not model domain-level or business-level failures.

Protocol: Generic RuntimeError usage is minimized in favor of these custom exceptions in all state-sensitive and resource-acquisition paths.

5.1 Escalation Semantics

  • Detection: Providers are responsible for detecting failures and escalating to the authoritative session layer by raising SessionNotAliveError.
  • Recovery Authority: ProviderSession catches the escalated exception and executes handle_session_failure() as the sole recovery executor, invalidating transient session contexts while preserving persistent authentication state.
  • Shutdown Authority: BrowserEngine owns terminal shutdown authority and coordinates global process teardown.
  • Isolation: Request-scoped failures (e.g., QueueOverflowError) must not silently poison the broader session or escalate into engine shutdown; they must be handled and isolated at the request level.
  • Precedence: Engine-scoped failures (e.g., BrowserShuttingDownError) invalidate all lower-level recovery paths immediately.
  • Ownership: Recovery execution must respect lifecycle ownership boundaries and follow the established lock hierarchy.

6. Cancellation & Cleanup Semantics

Cancellation safety is a core runtime invariant. The system prioritizes deterministic cleanup and strongly guarantees critical resource release under normal task cancellation, while auxiliary browser operations remain best-effort.

  • Shielded Teardown: Critical resource release paths (such as semaphore permit returns and persistent tab lease unlocking) MUST be wrapped in asyncio.shield to secure their execution during task cancellation.
  • Idempotency: Cleanup paths and ManagedPage.close() MUST be idempotent and safe for multiple concurrent executions. Failures in one step must not block subsequent resource release.
  • Non-Swallowing: asyncio.CancelledError must be propagated after executing the shielded cleanup; it must never be silently suppressed.
  • Best-Effort vs Mandatory Cleanup: The system strongly guarantees the return of active leases and semaphores under normal cancellation (mandatory release). Auxiliary Playwright cleanup operations (such as script detachment, page closure, or tab recreation) are executed on a best-effort basis and must not prevent successful lease/semaphore release.
  • Task Integrity: Request-scoped async tasks (observers, stream buffers) must be terminated before the tab's communication channels are removed.
  • Lease Ownership: During graceful drain or forced abort, BrowserEngine invokes request-owned abort callbacks and never mutates lease counters directly.

7. Logging & Observability

Runtime integrity failures must be visible and traceable.

  • Contextual Metadata: Logs MUST contain relevant identifiers: provider, request_id, generation, tab_id, and lease_id.
  • Invariant Violations: Violations of the Lock Hierarchy or Ownership Boundaries must be logged as CRITICAL.
  • No Silencing: Swallowing exceptions without a logged technical rationale is forbidden.
  • Severity Discipline:
    • DEBUG: Hot-path events (bridge events, loop iterations).
    • WARNING: Recoverable failures and successful session-scoped recovery events.
    • ERROR: Request-scoped terminal failures.
    • CRITICAL: Session corruption, invariant violations, or engine-scoped fatal errors.
    • Expected page closure during application shutdown is INFO; unexpected runtime page closure is WARNING. Expected already-disconnected cleanup is debug/info. The known Playwright transport-close race ("Connection closed while reading from the driver") may be DEBUG only when post-error browser state confirms disconnection; other unexpected or generic close failures remain WARNING/ERROR as appropriate.

8. Forbidden Error Handling Patterns

The following patterns are strictly forbidden:

  1. Silent Swallowing: Broad except Exception: pass without logging and justification.
  2. Post-Shutdown Resurrection: Attempting to re-initialize or self-heal after is_shutting_down is True.
  3. Lease Reuse: Attempting to use a tab or lease after invalidation (crash, poisoning, rollover).
  4. Autonomous Recovery: Recreating browser contexts or processes from within provider logic.
  5. Lock-Order Violations: Attempting to acquire locks outside the authoritative lock hierarchy, including during recovery or race mitigation.
  6. Unshielded Cleanup: Performing critical resource release outside of asyncio.shield.
  7. Ignoring Generations: Allowing stale page references to persist across browser process generations.

9. Relationship to Other Runtime Contracts

This document complements and is supported by the other authoritative specifications: