diff --git a/SUMMARY.md b/SUMMARY.md index 2fb58cd47..90ec27008 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -286,7 +286,7 @@ * [FHIR Schema Validator](modules/profiling-and-validation/fhir-schema-validator/README.md) * [Setup Aidbox with FHIR Schema validation engine](modules/profiling-and-validation/fhir-schema-validator/setup-aidbox-with-fhir-schema-validation-engine.md) * [Skip validation of references in resource using request header](modules/profiling-and-validation/skip-validation-of-references-in-resource-using-request-header.md) - * [Asynchronous resource validation](modules/profiling-and-validation/asynchronous-resource-validation.md) + * [Batch resource validation](modules/profiling-and-validation/batch-resource-validation.md) * [Observability](modules/observability/README.md) * [Getting started](modules/observability/getting-started/README.md) * [Run Aidbox with OpenTelemetry locally](modules/observability/getting-started/run-aidbox-with-opentelemetry-locally.md) diff --git a/docs/api/bulk-api/import-and-fhir-import.md b/docs/api/bulk-api/import-and-fhir-import.md index b4ca2d636..cb084e256 100644 --- a/docs/api/bulk-api/import-and-fhir-import.md +++ b/docs/api/bulk-api/import-and-fhir-import.md @@ -18,7 +18,7 @@ Keep in mind that $import **does not validate** inserted resources for the sake {% endhint %} {% hint style="info" %} -Please consider using [Asynchronous validation API](../../modules/profiling-and-validation/asynchronous-resource-validation.md) to validate data after $import +Please consider using [Batch validation API](../../modules/profiling-and-validation/batch-resource-validation.md) to validate data after $import {% endhint %} ## Example diff --git a/docs/api/bulk-api/load-and-fhir-load.md b/docs/api/bulk-api/load-and-fhir-load.md index 73eb274b4..edc4783d3 100644 --- a/docs/api/bulk-api/load-and-fhir-load.md +++ b/docs/api/bulk-api/load-and-fhir-load.md @@ -24,7 +24,7 @@ When loading resources with references, remember that '`/'` is {% endhint %} {% hint style="info" %} -Please consider using [Asynchronous validation API](../../modules/profiling-and-validation/asynchronous-resource-validation.md#asynchronous-batch-validation-draft) to validate data after $load +Please consider using [Batch validation API](../../modules/profiling-and-validation/batch-resource-validation.md) to validate data after $load {% endhint %} Load 100 synthea Patients to Aidbox (see [tutorial](../../tutorials/bulk-api-tutorials/synthea-by-bulk-api.md)): diff --git a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md b/docs/modules/profiling-and-validation/asynchronous-resource-validation.md deleted file mode 100644 index a518445df..000000000 --- a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md +++ /dev/null @@ -1,212 +0,0 @@ ---- -description: Use RPC to run a validation operation to check a resource conformance ---- - -# Asynchronous resource validation - -### Asynchronous Batch Validation - -It may happen that you updated your profiles when data is already in your database or you want to do efficiently load a batch of data and validate it later. API consists of 4 procedures and a couple of resources: - -* [aidbox.validation/batch-validation](asynchronous-resource-validation.md#aidbox-validation-batch-validation) **-** run validation -* [aidbox.validation/resources-batch-validation-task](asynchronous-resource-validation.md#aidbox.validation-batch-validation-result) - run validation with Aidbox Workflow -* [aidbox.validation/batch-validation-result](asynchronous-resource-validation.md#aidbox-validation-batch-validation-result) - inspect results (useful for async mode) -* [aidbox.validation/clear-batch-validation](asynchronous-resource-validation.md#aidbox-validation-clear-batch-validation) - clear validation results - -#### Prepare data - -To illustrate let's create some invalid data in Aidbox: - -```yaml -POST /Patient -content-type: text/yaml - -id: 'pt1' -birthDate: '1980-03-05' -``` - -Break data from DB Console: - -```sql -update patient -set resource = resource || '{"ups": "extra"}' -where id = 'pt1' -returning * -``` - -#### aidbox.validation/batch-validation - -You can validate your existing data with our new rpc `aidbox.validation/batch-validation`: - -```yaml -POST /rpc -content-type: text/yaml - -method: aidbox.validation/batch-validation -params: - # resourceType to validate - resource: Patient - id: pt-validation-run-1 - # you can limit number of resources to validate - limit: 100 - # you can stop process on specific number of invalid resources - errorsThreshold: 10 - # where section of resources query - filter: "resource#>>'{birthDate}' is not null" - ## run validation asynchronously - # async: true - ## specify profiles to validate - # profiles: ['profile-url-1', 'profile-url-2'] - - - -# response -result: - id: pt-validation-run-2 - valid: 0 - invalid: 1 - duration: 15 - problems: - - resource: - id: pt1 - ups: extra - meta: - createdAt: '2021-08-05T16:36:37.723008+03:00' - versionId: '1224' - lastUpdated: '2021-08-05T16:36:37.723008+03:00' - birthDate: '1980-03-05' - resourceType: Patient - errors: - - path: - - ups - message: extra property -``` - -#### aidbox.validation/resources-batch-validation-task - -You can run validation workflow with rpc method, which creates task for every resource provided in rpc's params fields `include` or `exclude`: - -
POST /rpc
-accept: text/yaml
-content-type: text/yaml
-
-
-method: aidbox.validation/resources-batch-validation-task
-params:
-  include: ['patient', 'observation']
-  error-threshold: 10000
-  
-  
-# response
-params:
-  tables:
-    - patient
-    - observation
-status: in-progress
-definition: aidbox.validation/resource-types-batch-validation-workflow
-id: >-
-  7addda33-003e-4892-a1d9-0faffbedf86d
-resourceType: AidboxWorkflow
-
- -{% hint style="info" %} -If you specify `include` param, only types you passed will be validated. - -If you specify `exclude` param, all types will be validated except the ones you passed. - -`include` and `exclude` params cannot be used together. -{% endhint %} - -You can check a progress of workflow in Aidbox UI or by rpc method: - -```yaml -POST /rpc -accept: text/yaml -content-type: text/yaml - - -method: awf.workflow/status -params: - id: 7addda33-003e-4892-a1d9-0faffbedf86d - -#response -result: - resource: - params: - tables: - - patient - - observation - result: Finished - status: done - outcome: succeeded - definition: aidbox.validation/resource-types-batch-validation-workflow - id: >- - 7addda33-003e-4892-a1d9-0faffbedf86d - resourceType: AidboxWorkflow -``` - -#### aidbox.validation/batch-validation-result - -If you run validation in async mode or aidbox.validation/resources-batch-validation-task, it will respond instantly and run validation in the background. You can get validation results with RPC `aidbox.validation/batch-validation-result` - -```yaml -POST /rpc?_format=yaml -content-type: text/yaml - -method: aidbox.validation/batch-validation-result -params: - id: pt-validation-run-1 - -# response -status: 200 -result: - valid: 1543 - invalid: 2 - duration: 3293 - problems: - - resource: {....} - errors: [{...}, {...}] -``` - -{% hint style="info" %} -`aidbox.validation/batch-validation-result` method requires `resourceType` param, which has a default value `BatchValidationRun`. -So, if you want to get the result from aidbox.validation/resources-batch-validation-task you need pass "AidboxWorkflow" to `resourceType` param. -{% endhint %} - -#### aidbox.validation/clear-batch-validation - -When you do not need results of this validation you can clean up resources with: - -```yaml -POST /rpc?_format=yaml -content-type: text/yaml - -method: aidbox.validation/clear-batch-validation -params: - id: pt-validation-run-1 -``` - -#### BatchValidationRun & BatchValidationError Resources - -When you run validation operation aidbox internally creates resource BatchValidationRun and put errors of validation in BatchValidationError. You can access these resources through standard CRUD/Search API - -```yaml -GET /BatchValidationError?.run.id=pt-validation-run-2&_format=yaml&_result=array - -# response -- run: - id: pt-validation-run-2 - resourceType: BatchValidationRun - errors: - - path: - - ups - message: extra property - resource: - id: pt1 - resourceType: Patient - id: pt-validation-run-2-Patient-pt1 -``` - -{% hint style="info" %} -If you restart Aidbox you have to start validation over -{% endhint %} diff --git a/docs/modules/profiling-and-validation/batch-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md new file mode 100644 index 000000000..803a0dad9 --- /dev/null +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -0,0 +1,284 @@ +--- +description: >- + Validate resources already stored in the database against their schemas and + FHIR profiles, at scale, via a resource-level $batch-validate operation, + synchronously or asynchronously, then drill into the offending resources. +--- + +# Batch resource validation + +{% hint style="info" %} +Available in Aidbox starting from version **2607**. +{% endhint %} + +{% hint style="warning" %} +**Breaking change.** `$batch-validate` **replaces** the previous batch-validation API, which is **removed**. The `aidbox.validation/batch-validation`, `aidbox.validation/batch-validation-result`, `aidbox.validation/clear-batch-validation`, and `aidbox.validation/resources-batch-validation-task` RPCs no longer exist, and a run no longer produces `BatchValidationRun` / `BatchValidationError` resources. Results now live in the aggregated `aidbox_batch_validation` schema (see [How results are stored](#how-results-are-stored)). Migrate to the `$batch-validate` operation described below. +{% endhint %} + +## Overview + +Batch validation checks resources **already in the database** against the active FHIR schemas and an optional set of profiles. Use it when you loaded data with validation off, or when you publish a new profile version and want to know **how many existing resources are non-compliant and why**. + +`$batch-validate` runs against **one resource type**. Aidbox hash-partitions it into a **fixed number of tasks** (`number-of-chunks`, default `12`), each task validating its `mod(hash(id), N)` slice, and aggregates the results into a compact, offender-indexed form (see [How results are stored](#how-results-are-stored)). + +It works two ways, chosen by the `Prefer` header: + +| | Trigger | Response | +| --- | --- | --- | +| **Synchronous** (default) | `POST …/$batch-validate` | blocks, returns a `Parameters` summary | +| **Asynchronous** | same + `Prefer: respond-async` | `202` + `Content-Location`; poll for the result | + +Both paths produce the same result under a **`task-id`** and persist it the same way, so you drill into a synchronous run the same as an asynchronous one. + +{% hint style="info" %} +Synchronous validation blocks the request until it finishes. That suits a type scoped by a narrow `_since`/`_until` window; for a large type, use `Prefer: respond-async`. The synchronous path runs the N tasks on a local pool sized by `scheduler-executors` (`BOX_SCHEDULER_EXECUTORS`, default `4`); the async path schedules them on the task scheduler, which spreads them across nodes. + +Each task scans the window once for its hash slice, so total scan work grows with the task count. More chunks means more parallelism (the async path spreads them across nodes) but more scans; fewer chunks means fewer scans but less parallelism. The default of `12` balances the two. Raise `number-of-chunks` to parallelize a large type further. +{% endhint %} + +{% hint style="warning" %} +**PostgreSQL connections.** Running tasks each use their own PostgreSQL connections, so a run with many parallel tasks (a high `number-of-chunks` together with a high `scheduler-executors`) raises connection use. Make sure PostgreSQL `max_connections` (and any external pooler) has the headroom, or tasks will fail to acquire a connection. +{% endhint %} + +## Start a validation + +`POST /fhir//$batch-validate` with a FHIR `Parameters` body: + +```yaml +POST /fhir/Observation/$batch-validate +content-type: application/json + +resourceType: Parameters +parameter: + # required: only resources whose meta.lastUpdated >= _since + - {name: _since, valueInstant: '2025-06-02T00:00:00Z'} + # optional upper bound (exclusive) + - {name: _until, valueInstant: '2025-06-09T00:00:00Z'} + # validate against these profiles (conjunctive, see Profiles) + - {name: profile, valueCanonical: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'} + # tuning (optional): number of parallel tasks (default 12) + - {name: number-of-chunks, valuePositiveInt: 24} + # validator options (optional): override the box's validation settings for this run + - {name: disable-terminology-validation, valueBoolean: true} +``` + +| Parameter | Type | Meaning | +| --- | --- | --- | +| `_since` **(required)** | `instant` | Only resources whose `meta.lastUpdated >= _since` (inclusive). Required so that a run declares a window instead of scanning a whole type (see [Filtering by date](#filtering-by-date)). | +| `_until` | `instant` | Upper bound: `meta.lastUpdated < _until` (exclusive). | +| `profile` (repeatable) | `canonical` | Validate every resource against these profile URLs (in addition to its base schema), conjunctively (see [Profiles](#profiles)). | +| `number-of-chunks` (default `12`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task streams its slice, so heap stays bounded regardless. No fixed maximum: a sync run streams the tasks through a bounded thread pool (heap stays proportional to the executor count, not the task count), and an async run writes one scheduler row per task, so a very large value costs task rows and scans, not memory. | +| `disable-terminology-validation` | `boolean` | Skip coded-binding / terminology checks. | +| `disable-primitive-validation` | `boolean` | Skip primitive type & format checks. | +| `disable-slicing-validation` | `boolean` | Skip slice validation. | +| `disable-constraint-validation` | `boolean` | Blanket switch for FHIRPath invariants: `true` skips **all** of them; `false` checks **all** (see [Validator options](#validator-options)). | +| `disable-constraint` (repeatable) | `string` | Skip specific invariants by key (e.g. `us-core-8`). | +| `strict-profile-resolution` | `boolean` | Treat an unresolved `profile` / `meta.profile` canonical as an error instead of skipping it. | +| `strict-extension-resolution` | `boolean` | Treat an unresolved extension as an error. | + +{% hint style="warning" %} +The body must be a valid `Parameters` resource. Each parameter must use the **exact** `value[x]` type above (`profile` as `valueCanonical`, `_since`/`_until` as `valueInstant`, `number-of-chunks` as `valuePositiveInt`, the `disable-*`/`strict-*` flags as `valueBoolean`, `disable-constraint` as `valueString`). Aidbox rejects an unknown parameter, a wrong value type, or a missing `_since` with `422` and an `OperationOutcome` that names the offending parameter. +{% endhint %} + +## Validator options + +The `disable-*` and `strict-*` parameters tune what the validator checks, **for this run only**. Each is three-state: **omit it to keep the box's configured setting** (see [FHIR Schema Validator](fhir-schema-validator/README.md)), or pass it to override that setting (`true`/`false`). Use them to trade completeness for speed on a huge type (e.g. skip the terminology and slicing passes for a structural-only sweep), or to tighten a run beyond the box defaults (e.g. `strict-profile-resolution` to surface resources whose declared profiles don't resolve, which otherwise read as compliant). + +Constraints (FHIRPath invariants) have two controls that compose: + +* `disable-constraint-validation` is the blanket switch: `true` skips **every** invariant, `false` checks **every** invariant (including any the box normally mutes). +* `disable-constraint` names specific invariants to skip (repeat it per key). +* When both are given, the blanket wins: `true` skips everything (the list is moot); with `false`, only the listed keys are skipped and all others are checked. With the blanket omitted, the listed keys are skipped **on top of** the box's defaults. + +## Indexing the hash partition + +Each task validates its `mod(abs(hashtextextended(id, 0)), N)` slice of the type, where `N` is `number-of-chunks`. `N` stays fixed for a run, so the partition predicate is a constant expression that a PostgreSQL expression index can cover. Without such an index, every task scans the whole table to find its slice, so a run with `N` tasks costs `N` full scans (the scan cost the `number-of-chunks` note above describes). With a matching index, each task reads only its slice through the index. + +The gain grows as the slice shrinks, so an index matters most for a large type validated with a high `number-of-chunks`. Build it on the resource's storage table (named after the lowercased resource type) with the **same modulus** as the `number-of-chunks` the run passes: + +```sql +-- Observation, validated in 10000 chunks +CREATE INDEX CONCURRENTLY observation_batch_validate_10000 + ON observation (mod(abs(hashtextextended(id, 0)), 10000)); +``` + +Then run with the matching `number-of-chunks`: + +```yaml +POST /fhir/Observation/$batch-validate +resourceType: Parameters +parameter: + - {name: _since, valueInstant: '1970-01-01T00:00:00Z'} + - {name: number-of-chunks, valuePositiveInt: 10000} +``` + +{% hint style="warning" %} +The index modulus must equal `number-of-chunks`. A different value is a different expression, so PostgreSQL skips the index and the run falls back to per-task scans. Check the plan with `EXPLAIN` on one task's query before relying on the index. +{% endhint %} + +{% hint style="info" %} +`CREATE INDEX CONCURRENTLY` builds the index without blocking writes to the table. The index costs disk and slows writes, so drop it after a one-off sweep: `DROP INDEX observation_batch_validate_10000`. +{% endhint %} + +## Synchronous response + +A `Parameters` resource holds the `task-id`, the headline counts, a link to the offending resources, and one `issue` per distinct error (each with its own filtered drill-down link). + +```yaml +status: 200 + +resourceType: Parameters +parameter: + - {name: task-id, valueString: ''} + - {name: validated, valueUnsignedInt: 1804646} # resources validated + - {name: valid, valueUnsignedInt: 1317494} # resources with no issues + - {name: invalid, valueUnsignedInt: 487152} # distinct resources with ≥1 issue + - {name: bytes, valueDecimal: 5242880000} # total bytes of resource JSON processed + - {name: invalid-resources, valueUrl: '/fhir/$batch-validate//invalid-resources'} + - name: issue + part: + - {name: id, valueString: '5b4b07e4…'} # issue-id (for drill-down) + - {name: invalid-resources, valueUrl: '/fhir/$batch-validate//invalid-resources?_issue=5b4b07e4…'} + - {name: code, valueCode: invalid-slice-cardinality} + - {name: expression, valueString: category} # the element + - {name: profile, valueString: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'} + - {name: count, valueUnsignedInt: 486018} # distinct offending resources + - {name: constraint, valueString: us-core-8} # for invariants + - {name: diagnostics, valueString: '…human-readable message…'} +``` + +* **`bytes`** is the total size of the resource JSON the run processed, a `decimal` because the total overflows `unsignedInt` at scale. +* **`count`** is the number of **distinct offending resources** for the issue, derived from the offender index. +* For **invariant** issues, the `constraint` part carries the constraint key. The `diagnostics` part is a **generated** summary keyed off the issue `code` (for an invariant, `: constraint is not satisfied`). It is not the validator's original message text, which is not stored (see [How results are stored](#how-results-are-stored)). +* Other issue kinds add a type-specific part: `slice` (the slice name) for slice issues, `binding` (the value-set URL) for terminology-binding issues, and `unknown-profile` (the unresolved canonical) for an unresolved `profile` / `meta.profile`. A part with no value is omitted. +* Each `issue` carries its own `invalid-resources` link, pre-filtered to that issue. +* The summary lists at most **10,000** distinct issues, worst first. If a run produces more, it lists the worst 10,000 and adds `issues-total` (the true count) and `issues-truncated: true`, so a truncated list is not mistaken for the whole. +* On failure the response is an `OperationOutcome`. + +## Asynchronous response + +```http +status: 202 Accepted +Content-Location: /fhir/$batch-validate/ +``` + +The endpoints below are **system-level**, keyed by `task-id` alone (no resource type in the path). + +Poll the `Content-Location`: + +```http +GET /fhir/$batch-validate/ +``` + +* **In progress** → `202 Accepted` with an `X-Progress` header (percent of tasks completed, e.g. `45%`). +* **Complete** → `200` with the same `Parameters` summary as the synchronous response. +* **Cancelled** → `200`, an informational `OperationOutcome` (issue code `informational`). +* **Failed** → `200` with the partial `Parameters` summary from the tasks that completed, plus a `status: failed` parameter; if no task completed, a `200` `OperationOutcome`. +* **Unknown task** → `404`. + +## Drill into the offending resources + +The summary tells you which issues occur and how many resources hit each. To get the **offending resources**, each linked to the version that was validated and carrying its full `OperationOutcome`, call `invalid-resources`: + +```http +GET /fhir/$batch-validate//invalid-resources + ?_issue=&_issue=&_count=50&_page=1&_fullurl-only=false +``` + +| Query parameter | Meaning | +| --- | --- | +| `_issue` (repeatable) | Restrict to offenders of these issue(s). Omit for **all** offending resources. | +| `_count` / `_page` | Page size (default `50`, max `1000`) and 1-based page number. | +| `_fullurl-only` (default `false`) | When `true`, return only the `fullUrl` of each offender (omit the body and outcome). | + +The response is a **`Parameters` report** rather than a Bundle (see [Response format](#response-format)): a `total`, flat paging links, and one repeated `resource` parameter per offending resource. + +```yaml +resourceType: Parameters +parameter: + - {name: total, valueUnsignedInt: 1114} + # paging: flat links. first/previous appear past page 1; next/last before the last page + - {name: self, valueUrl: '/fhir/$batch-validate//invalid-resources?_count=50&_page=1'} + - {name: next, valueUrl: '/fhir/$batch-validate//invalid-resources?_count=50&_page=2'} + - {name: last, valueUrl: '/fhir/$batch-validate//invalid-resources?_count=50&_page=23'} + - name: resource + part: + - {name: fullUrl, valueUrl: '/Observation//_history/'} + - name: resource # omitted when _fullurl-only=true + resource: {resourceType: Observation, meta: {versionId: ''}, …} + - name: outcome # omitted when _fullurl-only=true + resource: + resourceType: OperationOutcome + issue: + - {severity: fatal, code: invalid, expression: [Observation.category], + diagnostics: '…', details: {coding: [{code: invalid-slice-cardinality}]}} +``` + +* Each `resource` parameter is **one distinct offending resource**: its versioned `fullUrl`, the `resource` body (read from history at the validated version), and its `outcome`, an `OperationOutcome` listing **every** issue that resource has (its full issue set, even when `_issue` narrows which resources come back). +* `_fullurl-only=true` drops the `resource` and `outcome` parts and keeps the `fullUrl`. +* Paging is flat: `self` always; `first` and `previous` once past page 1; `next` and `last` while before the last page. +* An unknown `_issue` on a known task returns an empty report (`total: 0`). An unknown task returns `404`. + +{% hint style="info" %} +The `fullUrl` is **version-specific** (`/_history/`), so a vread resolves to the resource version that was validated rather than the current one. (If that version was pruned, the body is absent; the `fullUrl` and `outcome` remain.) +{% endhint %} + +## Cancel + +```http +DELETE /fhir/$batch-validate/ +``` + +Responds `202 Accepted`. Cancellation removes the run's **pending** tasks and marks it **cancelled** (a later poll reports `cancelled`). Aidbox does not interrupt a task already running, and keeps partial results. + +## Profiles + +Pass `profile` (as `valueCanonical`) to validate every resource against those URLs (via `meta.profile`), say to check the impact of a new profile version on existing data. The profiles must be loaded (an installed IG package, or a `StructureDefinition` you created). + +{% hint style="info" %} +**Multiple profiles are conjunctive (AND).** A resource is compliant only if it conforms to **every** listed profile, and issues are the union of violations across them (same as FHIR `$validate`). + +For **OR** ("valid against US Core 6 **or** 7"), run one validation per profile and intersect the offender sets: a resource is OR-invalid only if it appears in **every** run's offenders. +{% endhint %} + +{% hint style="warning" %} +Aidbox skips an unresolvable profile URL when strict profile resolution is off, so a typo yields a "compliant" report that is wrong. Make sure the profile (and its package) is installed and the URL matches. +{% endhint %} + +## Filtering by date + +`_since` and `_until` are FHIR `instant`s that filter on **`meta.lastUpdated`**, following the bulk-export `_since`/`_until` semantics: + +```yaml +parameter: + - {name: _since, valueInstant: '2025-06-02T00:00:00Z'} # lastUpdated in [2025-06-02, 2025-06-09) + - {name: _until, valueInstant: '2025-06-09T00:00:00Z'} +``` + +* `_since` is an **inclusive** lower bound (`lastUpdated >= _since`); `_until` is an **exclusive** upper bound (`lastUpdated < _until`). +* `_since` is **required**: every run declares a window, so no call scans a whole (possibly huge) type by accident. To validate everything, pass an epoch `_since` (`1970-01-01T00:00:00Z`). +* A narrow window keeps a synchronous run small. Date filters compose with `profile`. + +## How results are stored + +Aidbox stores results in an **aggregated, compact** form, so validating 100 GB of non-conformant data does **not** add 100 GB to the database. The `aidbox_batch_validation` schema holds: + +| Table | Holds | +| --- | --- | +| `issue` | one row per distinct error (no per-resource rows) | +| `invalid_resource` | a tiny `(issue_id, resource_id, version_id)` row per offending resource: ids and versions only | +| `chunk_stat` | one row per task, written once when the task finishes: its `validated`/`invalid`/`bytes` tallies | + +The aggregation key is `profile`, `resource_type`, index-normalized `path` (`identifier[2].system` → `identifier.system`), `code`, `constraint_key`, and (where they apply) the slice name, binding value set, and unresolved profile canonical. All occurrences that share these collapse into one issue; the issue's **count is the number of offender rows** (distinct resources). + +Aidbox does **not** store the invalid resource bodies, their `OperationOutcome`s, or the validator's original message text. The drill-down re-reads the body from history at the validated version and reconstructs the `OperationOutcome` from the stored machine fields (`code`, `expression`, `constraint`, and the type-specific parts), synthesizing the `diagnostics` message from the `code`. + +Both synchronous and asynchronous runs persist these tables under the run's `task-id`, so you poll and drill into either until you cancel it. + +## Response format + +The `invalid-resources` response is a `Parameters` resource rather than a `Bundle`. A `searchset` or `collection` Bundle cannot carry a `total`, version-specific links, a per-offender `OperationOutcome`, and the invalid resource bodies together while remaining FHIR-valid: Bundle invariants prohibit a version-specific `fullUrl`, permit `total` and `entry.response` only on certain Bundle types, and require each embedded resource to be valid in its own right, which the invalid bodies are not. A `Parameters` resource is subject to none of these constraints: it preserves the version-specific drill-down links and embeds each `OperationOutcome` beside the resource it describes. The invalid resource bodies are the report's content: the data under review. + +## Terminology + +Coded-binding and slice validation may call the configured terminology server. Bindings that resolve **locally** (local code systems, or a hybrid engine with local content) validate offline and surface invalid codes as ordinary `terminology-binding-error` issues. But if validation needs the configured terminology server and it is **unreachable or errors**, the validator cannot complete and the **whole run fails**: a synchronous call returns `422` (`OperationOutcome`, "Batch validation failed…"), an asynchronous run reports `failed`. Point `fhir.terminology.service-base-url` at a reachable server (a local or hybrid engine works best) so coded validation is accurate and does not fail the run. diff --git a/docs/overview/faq.md b/docs/overview/faq.md index 32c384296..62e23e4a5 100644 --- a/docs/overview/faq.md +++ b/docs/overview/faq.md @@ -660,7 +660,7 @@ Learn more: For large datasets, enable async validation to process resources in the background without blocking API responses. -Learn more: [Asynchronous Resource Validation](../modules/profiling-and-validation/asynchronous-resource-validation.md) +Learn more: [Batch resource validation](../modules/profiling-and-validation/batch-resource-validation.md) ## Terminology & ValueSets diff --git a/docs/tutorials/artifact-registry-tutorials/custom-resources/migrate-to-fhirschema/README.md b/docs/tutorials/artifact-registry-tutorials/custom-resources/migrate-to-fhirschema/README.md index 659117eea..98d600dc3 100644 --- a/docs/tutorials/artifact-registry-tutorials/custom-resources/migrate-to-fhirschema/README.md +++ b/docs/tutorials/artifact-registry-tutorials/custom-resources/migrate-to-fhirschema/README.md @@ -194,7 +194,7 @@ If you have custom Aidbox SearchParameters defined via Zen or Entities, use this ### Step 6: Validate Resources and Resolve Issues -Since the FHIR Schema validation engine is more reliable and validates all the FHIR container cases, we need to validate existing data with FHIR Schema using [Aidbox Asynchronous Validation API](../../../../modules/profiling-and-validation/asynchronous-resource-validation.md). +Since the FHIR Schema validation engine is more reliable and validates all the FHIR container cases, we need to validate existing data with FHIR Schema using [Aidbox Batch Validation API](../../../../modules/profiling-and-validation/batch-resource-validation.md). ### Check Deprecated Capabilities diff --git a/redirects.yaml b/redirects.yaml index 0802a2df1..5a992cfaf 100644 --- a/redirects.yaml +++ b/redirects.yaml @@ -375,7 +375,8 @@ redirects: modules-1/observability/traces/how-to-use-tracing: modules/observability/traces/how-to-use-tracing.md modules-1/observability/traces/otel-traces-exporter-parameters: modules/observability/traces/otel-traces-exporter-parameters.md modules-1/other-modules/mcp: modules/other-modules/mcp.md - modules-1/profiling-and-validation/asynchronous-resource-validation: modules/profiling-and-validation/asynchronous-resource-validation.md + modules-1/profiling-and-validation/asynchronous-resource-validation: modules/profiling-and-validation/batch-resource-validation.md + modules/profiling-and-validation/asynchronous-resource-validation: modules/profiling-and-validation/batch-resource-validation.md modules-1/profiling-and-validation/fhir-schema-validator/setup-aidbox-with-fhir-schema-validation-engine: modules/profiling-and-validation/fhir-schema-validator/setup-aidbox-with-fhir-schema-validation-engine.md modules-1/profiling-and-validation/fhir-schema-validator/tutorials/how-to-create-fhir-npm-package: tutorials/artifact-registry-tutorials/how-to-create-fhir-npm-package.md modules-1/profiling-and-validation/skip-validation-of-references-in-resource-using-request-header: modules/profiling-and-validation/skip-validation-of-references-in-resource-using-request-header.md