From 328fdfd0c8fce19d32d9d242be331d9d92e9c86f Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Thu, 4 Jun 2026 18:03:42 +0300 Subject: [PATCH 01/14] update batch validation for async task engine --- docs/api/bulk-api/load-and-fhir-load.md | 2 +- .../asynchronous-resource-validation.md | 185 ++++++++---------- 2 files changed, 79 insertions(+), 108 deletions(-) diff --git a/docs/api/bulk-api/load-and-fhir-load.md b/docs/api/bulk-api/load-and-fhir-load.md index 73eb274b4..49d67518c 100644 --- a/docs/api/bulk-api/load-and-fhir-load.md +++ b/docs/api/bulk-api/load-and-fhir-load.md @@ -24,7 +24,7 @@ When loading resources with references, remember that '`/'` is {% endhint %} {% hint style="info" %} -Please consider using [Asynchronous validation API](../../modules/profiling-and-validation/asynchronous-resource-validation.md#asynchronous-batch-validation-draft) to validate data after $load +Please consider using [Asynchronous validation API](../../modules/profiling-and-validation/asynchronous-resource-validation.md#asynchronous-batch-validation) to validate data after $load {% endhint %} Load 100 synthea Patients to Aidbox (see [tutorial](../../tutorials/bulk-api-tutorials/synthea-by-bulk-api.md)): diff --git a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md b/docs/modules/profiling-and-validation/asynchronous-resource-validation.md index a518445df..7a09fed8f 100644 --- a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md +++ b/docs/modules/profiling-and-validation/asynchronous-resource-validation.md @@ -4,15 +4,25 @@ description: Use RPC to run a validation operation to check a resource conforman # Asynchronous resource validation -### Asynchronous Batch Validation +### Asynchronous Batch Validation -It may happen that you updated your profiles when data is already in your database or you want to do efficiently load a batch of data and validate it later. API consists of 4 procedures and a couple of resources: +It may happen that you updated your profiles when data is already in your database, or you want to efficiently load a batch of data and validate it later. Batch validation runs on the asynchronous task engine: each validation run is split into id-range chunks that are executed in parallel by the task scheduler, and the results are persisted, so they survive Aidbox restarts. -* [aidbox.validation/batch-validation](asynchronous-resource-validation.md#aidbox-validation-batch-validation) **-** run validation -* [aidbox.validation/resources-batch-validation-task](asynchronous-resource-validation.md#aidbox.validation-batch-validation-result) - run validation with Aidbox Workflow -* [aidbox.validation/batch-validation-result](asynchronous-resource-validation.md#aidbox-validation-batch-validation-result) - inspect results (useful for async mode) +The API consists of 4 procedures and a couple of resources: + +* [aidbox.validation/batch-validation](asynchronous-resource-validation.md#aidbox-validation-batch-validation) - run validation for one resource type +* [aidbox.validation/resources-batch-validation-task](asynchronous-resource-validation.md#aidbox-validation-resources-batch-validation-task) - run validation for many resource types +* [aidbox.validation/batch-validation-result](asynchronous-resource-validation.md#aidbox-validation-batch-validation-result) - inspect results * [aidbox.validation/clear-batch-validation](asynchronous-resource-validation.md#aidbox-validation-clear-batch-validation) - clear validation results +#### Parallelism + +A validation run is split into chunks of `chunkSize` resources (1000 by default). Chunks are independent tasks executed by the scheduler, so a single large table is validated in parallel. The number of executor threads is controlled by the `scheduler-executors` setting (`BOX_SCHEDULER_EXECUTORS`, default `4`) — increase it to speed up validation of large datasets. + +{% hint style="warning" %} +The previous implementation based on the Aidbox Workflow (AWF) engine is deprecated. It is still used when the setting `batch-validation-legacy-engine` (`BOX_BATCH_VALIDATION_LEGACY_ENGINE`) is set to `true`, or when the async task scheduler is not available. The legacy engine will be removed in a future release. Legacy-only parameters: `filter`, `limit`, `async`. +{% endhint %} + #### Prepare data To illustrate let's create some invalid data in Aidbox: @@ -28,15 +38,15 @@ birthDate: '1980-03-05' Break data from DB Console: ```sql -update patient +update patient set resource = resource || '{"ups": "extra"}' -where id = 'pt1' +where id = 'pt1' returning * ``` #### aidbox.validation/batch-validation -You can validate your existing data with our new rpc `aidbox.validation/batch-validation`: +Validate existing data of one resource type with the rpc `aidbox.validation/batch-validation`: ```yaml POST /rpc @@ -46,108 +56,62 @@ method: aidbox.validation/batch-validation params: # resourceType to validate resource: Patient - id: pt-validation-run-1 - # you can limit number of resources to validate - limit: 100 - # you can stop process on specific number of invalid resources - errorsThreshold: 10 - # where section of resources query - filter: "resource#>>'{birthDate}' is not null" - ## run validation asynchronously - # async: true - ## specify profiles to validate + ## specify profiles to validate against # profiles: ['profile-url-1', 'profile-url-2'] - - - + ## stop the run after this many invalid resources + # errorsThreshold: 10 + ## resources per chunk task (default 1000) + # chunkSize: 1000 + # response result: - id: pt-validation-run-2 - valid: 0 - invalid: 1 - duration: 15 - problems: - - resource: - id: pt1 - ups: extra - meta: - createdAt: '2021-08-05T16:36:37.723008+03:00' - versionId: '1224' - lastUpdated: '2021-08-05T16:36:37.723008+03:00' - birthDate: '1980-03-05' - resourceType: Patient - errors: - - path: - - ups - message: extra property + run-id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 + operation-id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 + status: in-progress + chunks: 1 ``` +The call returns immediately; validation runs in the background. Use `run-id` with [batch-validation-result](asynchronous-resource-validation.md#aidbox-validation-batch-validation-result) to poll for the outcome. + +If you pass `profiles`, every resource is validated against the given profile URLs in addition to its base schema — this is the way to check how a new profile version would affect existing data. + #### aidbox.validation/resources-batch-validation-task -You can run validation workflow with rpc method, which creates task for every resource provided in rpc's params fields `include` or `exclude`: +Validate several resource types at once. One chunked run is created across every table selected by `include`/`exclude`: -
POST /rpc
-accept: text/yaml
+```yaml
+POST /rpc
+accept: text/yaml
 content-type: text/yaml
 
-
 method: aidbox.validation/resources-batch-validation-task
 params:
   include: ['patient', 'observation']
-  error-threshold: 10000
-  
-  
+  # error-threshold: 10000
+  # profiles: ['profile-url-1']
+  # chunkSize: 1000
+
 # response
-params:
-  tables:
-    - patient
-    - observation
-status: in-progress
-definition: aidbox.validation/resource-types-batch-validation-workflow
-id: >-
-  7addda33-003e-4892-a1d9-0faffbedf86d
-resourceType: AidboxWorkflow
-
+result: + run-id: 7addda33-003e-4892-a1d9-0faffbedf86d + operation-id: 7addda33-003e-4892-a1d9-0faffbedf86d + status: in-progress + chunks: 12 +``` {% hint style="info" %} If you specify `include` param, only types you passed will be validated. If you specify `exclude` param, all types will be validated except the ones you passed. -`include` and `exclude` params cannot be used together. +`include` and `exclude` params cannot be used together. With neither, all resource types are validated. {% endhint %} -You can check a progress of workflow in Aidbox UI or by rpc method: - -```yaml -POST /rpc -accept: text/yaml -content-type: text/yaml - - -method: awf.workflow/status -params: - id: 7addda33-003e-4892-a1d9-0faffbedf86d - -#response -result: - resource: - params: - tables: - - patient - - observation - result: Finished - status: done - outcome: succeeded - definition: aidbox.validation/resource-types-batch-validation-workflow - id: >- - 7addda33-003e-4892-a1d9-0faffbedf86d - resourceType: AidboxWorkflow -``` +When `error-threshold` (or `errorsThreshold`) is reached, the whole operation is cancelled — remaining chunks are stopped and the run status becomes `cancelled`. #### aidbox.validation/batch-validation-result -If you run validation in async mode or aidbox.validation/resources-batch-validation-task, it will respond instantly and run validation in the background. You can get validation results with RPC `aidbox.validation/batch-validation-result` +Both run methods respond instantly and validate in the background. Get the current state and validation problems with `aidbox.validation/batch-validation-result`: ```yaml POST /rpc?_format=yaml @@ -155,23 +119,31 @@ content-type: text/yaml method: aidbox.validation/batch-validation-result params: - id: pt-validation-run-1 - + id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 + # problems are paged; defaults: page 0, pageSize 100 + # page: 0 + # pageSize: 100 + # response status: 200 result: - valid: 1543 - invalid: 2 - duration: 3293 + run: + id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 + resource: patient + status: in-progress + invalid: 2 + resourceType: BatchValidationRun + status: completed # in-progress | completed | cancelled | failed problems: - - resource: {....} - errors: [{...}, {...}] + - resource: + id: pt1 + resourceType: Patient + errors: + - path: ups + type: unknown-key ``` -{% hint style="info" %} -`aidbox.validation/batch-validation-result` method requires `resourceType` param, which has a default value `BatchValidationRun`. -So, if you want to get the result from aidbox.validation/resources-batch-validation-task you need pass "AidboxWorkflow" to `resourceType` param. -{% endhint %} +`status` reflects the live state of the scheduled chunk tasks. `run.invalid` is the total number of invalid resources recorded so far. `problems` is paged with `page`/`pageSize`, so large result sets can be inspected fully. #### aidbox.validation/clear-batch-validation @@ -183,30 +155,29 @@ content-type: text/yaml method: aidbox.validation/clear-batch-validation params: - id: pt-validation-run-1 + id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 ``` +This deletes the `BatchValidationRun`, its `BatchValidationError` resources and the scheduler bookkeeping for the operation. + #### BatchValidationRun & BatchValidationError Resources -When you run validation operation aidbox internally creates resource BatchValidationRun and put errors of validation in BatchValidationError. You can access these resources through standard CRUD/Search API +When you run a validation operation Aidbox internally creates a BatchValidationRun resource and puts errors of validation in BatchValidationError. You can access these resources through standard CRUD/Search API — for example, to aggregate errors by type or build a report: ```yaml -GET /BatchValidationError?.run.id=pt-validation-run-2&_format=yaml&_result=array +GET /BatchValidationError?.run.id=c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99&_format=yaml&_result=array # response - run: - id: pt-validation-run-2 + id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 resourceType: BatchValidationRun errors: - - path: - - ups - message: extra property + - path: ups + type: unknown-key resource: id: pt1 resourceType: Patient - id: pt-validation-run-2-Patient-pt1 + id: 6c8c5045-71b8-43d4-9e44-b3a0bfbe6e54 ``` -{% hint style="info" %} -If you restart Aidbox you have to start validation over -{% endhint %} +Both resources are persisted in the database, so validation results survive Aidbox restarts. From 7a2c300a7352c33ab86f3b37794ae8d77091d458 Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Thu, 4 Jun 2026 19:27:57 +0300 Subject: [PATCH 02/14] batch validation run lifecycle and id param --- .../asynchronous-resource-validation.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md b/docs/modules/profiling-and-validation/asynchronous-resource-validation.md index 7a09fed8f..9608dc1a9 100644 --- a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md +++ b/docs/modules/profiling-and-validation/asynchronous-resource-validation.md @@ -20,7 +20,7 @@ The API consists of 4 procedures and a couple of resources: A validation run is split into chunks of `chunkSize` resources (1000 by default). Chunks are independent tasks executed by the scheduler, so a single large table is validated in parallel. The number of executor threads is controlled by the `scheduler-executors` setting (`BOX_SCHEDULER_EXECUTORS`, default `4`) — increase it to speed up validation of large datasets. {% hint style="warning" %} -The previous implementation based on the Aidbox Workflow (AWF) engine is deprecated. It is still used when the setting `batch-validation-legacy-engine` (`BOX_BATCH_VALIDATION_LEGACY_ENGINE`) is set to `true`, or when the async task scheduler is not available. The legacy engine will be removed in a future release. Legacy-only parameters: `filter`, `limit`, `async`. +The previous implementation based on the Aidbox Workflow (AWF) engine is deprecated. It is still used when the setting `batch-validation-legacy-engine` (`BOX_BATCH_VALIDATION_LEGACY_ENGINE`) is set to `true`, or when the async task scheduler is not available. The legacy engine will be removed in a future release. The `filter`, `limit` and `async` parameters are supported only by the legacy engine — the default engine rejects them with an error. {% endhint %} #### Prepare data @@ -56,6 +56,8 @@ method: aidbox.validation/batch-validation params: # resourceType to validate resource: Patient + ## optional run id; rejected if a run with this id already exists + # id: my-validation-run ## specify profiles to validate against # profiles: ['profile-url-1', 'profile-url-2'] ## stop the run after this many invalid resources @@ -130,7 +132,7 @@ result: run: id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 resource: patient - status: in-progress + status: complete # in-progress | complete | cancelled | failed invalid: 2 resourceType: BatchValidationRun status: completed # in-progress | completed | cancelled | failed From a6d84f105a7eba8fa83163652d7985cf6a9bfc44 Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Thu, 25 Jun 2026 11:56:52 +0300 Subject: [PATCH 03/14] batch validation fixes --- .../asynchronous-resource-validation.md | 367 ++++++++++++------ 1 file changed, 248 insertions(+), 119 deletions(-) diff --git a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md b/docs/modules/profiling-and-validation/asynchronous-resource-validation.md index 9608dc1a9..cc8b63889 100644 --- a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md +++ b/docs/modules/profiling-and-validation/asynchronous-resource-validation.md @@ -1,185 +1,314 @@ --- -description: Use RPC to run a validation operation to check a resource conformance +description: >- + Validate resources already stored in the database against their schemas and + FHIR profiles, asynchronously and at scale, and analyze the results. --- # Asynchronous resource validation -### Asynchronous Batch Validation +{% hint style="danger" %} +**Breaking change since version 2607.** Batch validation was reworked onto the asynchronous engine and a new API. If you used it before 2607: -It may happen that you updated your profiles when data is already in your database, or you want to efficiently load a batch of data and validate it later. Batch validation runs on the asynchronous task engine: each validation run is split into id-range chunks that are executed in parallel by the task scheduler, and the results are persisted, so they survive Aidbox restarts. +* The RPC procedures (`aidbox.validation/batch-validation`, `resources-batch-validation-task`, `batch-validation-result`, `clear-batch-validation`) are **removed**. Use the FHIR operations on `BatchValidationRun` described below. +* `BatchValidationError` resources are **no longer produced**. Results are stored in an aggregated form and read via `$result` / `$offenders` (see [How results are stored](#how-results-are-stored)). +* The deprecated AWF/legacy engine and the `batch-validation-legacy-engine` (`BOX_BATCH_VALIDATION_LEGACY_ENGINE`) setting are **removed**; an asynchronous task scheduler is now required. +* The `filter`, `limit`, and `async` parameters are **no longer supported**. +{% endhint %} -The API consists of 4 procedures and a couple of resources: +## Overview -* [aidbox.validation/batch-validation](asynchronous-resource-validation.md#aidbox-validation-batch-validation) - run validation for one resource type -* [aidbox.validation/resources-batch-validation-task](asynchronous-resource-validation.md#aidbox-validation-resources-batch-validation-task) - run validation for many resource types -* [aidbox.validation/batch-validation-result](asynchronous-resource-validation.md#aidbox-validation-batch-validation-result) - inspect results -* [aidbox.validation/clear-batch-validation](asynchronous-resource-validation.md#aidbox-validation-clear-batch-validation) - clear validation results +Batch validation checks resources that are **already in the database** against the active FHIR schemas and, optionally, against a set of profiles. Use it when you loaded data with validation off, or when you publish a new version of a profile and want to know **how many existing resources are non-compliant and why**. -#### Parallelism +A run validates one or more resource types. Each table is split into id-range **chunks** that are executed in parallel by the asynchronous task scheduler, so a single large table fans out across workers. Results are persisted in a compact, aggregated form (see [How results are stored](#how-results-are-stored)) and survive Aidbox restarts. -A validation run is split into chunks of `chunkSize` resources (1000 by default). Chunks are independent tasks executed by the scheduler, so a single large table is validated in parallel. The number of executor threads is controlled by the `scheduler-executors` setting (`BOX_SCHEDULER_EXECUTORS`, default `4`) — increase it to speed up validation of large datasets. +The API is a set of FHIR operations on the `BatchValidationRun` resource: -{% hint style="warning" %} -The previous implementation based on the Aidbox Workflow (AWF) engine is deprecated. It is still used when the setting `batch-validation-legacy-engine` (`BOX_BATCH_VALIDATION_LEGACY_ENGINE`) is set to `true`, or when the async task scheduler is not available. The legacy engine will be removed in a future release. The `filter`, `limit` and `async` parameters are supported only by the legacy engine — the default engine rejects them with an error. +| Operation | Purpose | +| --- | --- | +| `POST /fhir/BatchValidationRun/$run` | Start a run | +| `GET /fhir/BatchValidationRun/{id}/$result` | Status, compliance summary, findings, per‑resource problems | +| `GET /fhir/BatchValidationRun/{id}/$offenders` | All resource ids hitting one finding pattern | +| `POST /fhir/BatchValidationRun/{id}/$clear` | Delete a run and its results | + +{% hint style="info" %} +An asynchronous task scheduler is required. The number of executor threads is controlled by the `scheduler-executors` setting (`BOX_SCHEDULER_EXECUTORS`, default `4`) — raise it to validate large datasets faster. Chunks are also distributed across nodes in a clustered deployment. {% endhint %} -#### Prepare data +## Start a run + +`POST /fhir/BatchValidationRun/$run` starts a run and returns immediately with a run id; validation proceeds in the background. -To illustrate let's create some invalid data in Aidbox: +The body can be either a FHIR **`Parameters`** resource (the canonical operation-input form) or a **plain JSON** object with the same fields (an accepted shorthand). The examples below use the plain JSON shorthand; the equivalent `Parameters` form is shown under [Body shapes](#body-shapes). ```yaml -POST /Patient +POST /fhir/BatchValidationRun/$run content-type: text/yaml -id: 'pt1' -birthDate: '1980-03-05' -``` +# What to validate — choose one of: +resource: Patient # a single resource type +# include: [Patient, Observation] # several types +# exclude: [Provenance] # all types except these +# (omit all three to validate every persistable resource type) -Break data from DB Console: +# Optional: +# id: my-run # client-supplied run id (rejected if it already exists) +# profiles: ['http://hl7.org/fhir/us/core/StructureDefinition/us-core-patient'] +# errorsThreshold: 10000 # cancel the run once this many violations are recorded +# chunkSize: 1000 # max rows per chunk task (byte-bounded, see note) +# incremental: true # only validate resources changed since the last run of this stream +# configId: nightly-uscore # incremental stream id -```sql -update patient -set resource = resource || '{"ups": "extra"}' -where id = 'pt1' -returning * +# response +status: 202 +Content-Location: /fhir/BatchValidationRun/my-run/$result +result: + run-id: my-run + operation-id: my-run + status: in-progress ``` -#### aidbox.validation/batch-validation +{% hint style="info" %} +`include` and `exclude` are mutually exclusive. With none of `resource`/`include`/`exclude`, every persistable resource type is validated. +{% endhint %} -Validate existing data of one resource type with the rpc `aidbox.validation/batch-validation`: +`errorsThreshold` (alias `error-threshold`) cancels the whole operation once that many violations are recorded — remaining chunks are stopped and the run status becomes `cancelled`. -```yaml -POST /rpc -content-type: text/yaml +`chunkSize` is an **upper bound**, not a fixed size. The planner caps each chunk by stored bytes (~8 MB), so heavy resource types (e.g. Provenance) get fewer rows per chunk than requested while light types use the full `chunkSize`. This keeps a single chunk's in-memory footprint bounded regardless of resource size. + +### Body shapes + +`$run` accepts the spec in two shapes, both producing the same run: + +* **FHIR `Parameters`** (canonical). Multi-valued fields (`include`, `exclude`, `profiles`) and date ranges repeat the parameter; this is the form to use for the prefixed date filters (see [Filtering by date range](#filtering-by-date-range)). + + ```yaml + POST /fhir/BatchValidationRun/$run + content-type: application/json + + resourceType: Parameters + parameter: + - {name: include, valueString: Patient} + - {name: include, valueString: Observation} + - {name: profiles, valueString: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-patient'} + - {name: id, valueString: my-run} + ``` + +* **Plain JSON** (shorthand). Terser for the common case: + + ```yaml + POST /fhir/BatchValidationRun/$run + content-type: text/yaml + + include: [Patient, Observation] + profiles: ['http://hl7.org/fhir/us/core/StructureDefinition/us-core-patient'] + id: my-run + ``` + +## Get results + +`GET /fhir/BatchValidationRun/{id}/$result` returns the run state and three views of the results. Poll it until `status` is terminal (`complete`, `failed`, or `cancelled`). -method: aidbox.validation/batch-validation -params: - # resourceType to validate - resource: Patient - ## optional run id; rejected if a run with this id already exists - # id: my-validation-run - ## specify profiles to validate against - # profiles: ['profile-url-1', 'profile-url-2'] - ## stop the run after this many invalid resources - # errorsThreshold: 10 - ## resources per chunk task (default 1000) - # chunkSize: 1000 +```yaml +GET /fhir/BatchValidationRun/my-run/$result?pageSize=50 +# optional query params: page (0-based), pageSize, severity (e.g. "error") # response +status: 200 result: - run-id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 - operation-id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 - status: in-progress - chunks: 1 + run: + id: my-run + resourceType: BatchValidationRun + status: complete # in-progress | complete | cancelled | failed + invalid: 14 # total violation occurrences (see note below) + status: complete + compliance: + scanned: 1127 # resources validated + compliant: 1115 # resources with no violations + non-compliant: 12 # distinct resources with at least one violation (14 occurrences across them) + percent: 98.94 + findings: # aggregate, worst (highest count) first + - path: bogusKey + code: unknown-key + resource_type: Patient + profile_url: '' + constraint_key: '' + severity: error + count: 3 + pattern_hash: 5b4b07e46a65a5f0 + message: 'Patient.bogusKey: element is not allowed by the profile' + problems: # per-resource view (paged by page/pageSize) + - resource: + id: p-1 + resourceType: Patient + gender: 5 + errors: + - path: gender + type: invalid-type + severity: error + diagnostics: 'Patient.gender: value has the wrong type' ``` -The call returns immediately; validation runs in the background. Use `run-id` with [batch-validation-result](asynchronous-resource-validation.md#aidbox-validation-batch-validation-result) to poll for the outcome. +The result has three parts: + +* **`compliance`** — the headline rollup: how many resources were `scanned`, how many are `compliant` vs `non-compliant` (distinct resources), and the `percent` compliant. +* **`findings`** — the aggregate "which paths fail and how often" view, worst-first. Each finding is one error **pattern** with a `count`, a reconstructed human `message`, and a `pattern_hash` for drill-down (the resource ids hitting the pattern are fetched separately via `$offenders`). Pass `severity=error` to see only errors. +* **`problems`** — the per-resource view (the offending resource plus its reconstructed errors), paged by `page`/`pageSize`. The `resource` body is **re-read live from the primary table** at read time (it is not stored by the run), so it reflects the resource's **current** state — if it was edited after the run it may no longer match the errors, and if it was deleted the body collapses to an `{id, resourceType}` stub. -If you pass `profiles`, every resource is validated against the given profile URLs in addition to its base schema — this is the way to check how a new profile version would affect existing data. +{% hint style="warning" %} +`run.invalid` and the sum of finding `count`s are **violation occurrences** — one resource that breaks three rules contributes three. `compliance.non-compliant` is the number of **distinct resources** with at least one violation. They are different numbers; use `compliance` for "how many of my resources are bad". + +If chunks fail or vanish, the run also records synthetic `chunk-validation-failed` / `chunk-incomplete` findings (so a partially-covered run can't read as clean — see [Run status](#run-status-as-a-resource)). These count toward `run.invalid` and appear in `findings`, but are **excluded** from `compliance.non-compliant`. So on a healthy run `run.invalid` is pure violation occurrences; on a degraded run it is inflated by the failure markers — another reason to read `compliance` for the real picture. +{% endhint %} -#### aidbox.validation/resources-batch-validation-task +## Drill into a finding -Validate several resource types at once. One chunked run is created across every table selected by `include`/`exclude`: +A finding tells you *which* pattern fails and *how many* resources hit it, but not *which* resources. To get the resource ids, take the finding's `pattern_hash` and call `$offenders`: ```yaml -POST /rpc -accept: text/yaml -content-type: text/yaml +GET /fhir/BatchValidationRun/my-run/$offenders?pattern=5b4b07e46a65a5f0&pageSize=1000 +# optional: page (0-based), pageSize + +# response +result: + run-id: my-run + pattern-hash: 5b4b07e46a65a5f0 + resource-ids: [p-4, p-8, p-12] +``` -method: aidbox.validation/resources-batch-validation-task -params: - include: ['patient', 'observation'] - # error-threshold: 10000 - # profiles: ['profile-url-1'] - # chunkSize: 1000 +## Clear a run + +```yaml +POST /fhir/BatchValidationRun/my-run/$clear # response result: - run-id: 7addda33-003e-4892-a1d9-0faffbedf86d - operation-id: 7addda33-003e-4892-a1d9-0faffbedf86d - status: in-progress - chunks: 12 + message: "Batch validation results for run.id='my-run' are cleared" ``` -{% hint style="info" %} -If you specify `include` param, only types you passed will be validated. +This deletes the `BatchValidationRun`, all of its stored findings, offenders, and scanned counter, and the scheduler bookkeeping for the operation. (The incremental `watermark` is keyed by `configId`, not by run id, so it is left intact.) -If you specify `exclude` param, all types will be validated except the ones you passed. +## Run status as a resource -`include` and `exclude` params cannot be used together. With neither, all resource types are validated. -{% endhint %} +`BatchValidationRun` is a regular resource; you can read or search it through the standard API to list runs and check status: -When `error-threshold` (or `errorsThreshold`) is reached, the whole operation is cancelled — remaining chunks are stopped and the run status becomes `cancelled`. +```yaml +GET /fhir/BatchValidationRun/my-run +``` -#### aidbox.validation/batch-validation-result +## Validating against profiles -Both run methods respond instantly and validate in the background. Get the current state and validation problems with `aidbox.validation/batch-validation-result`: +Pass `profiles` to validate every resource against those profile URLs in addition to its base schema — this is how you check the impact of a new profile version on existing data. The profiles must be loaded (e.g. an installed IG package, or a `StructureDefinition` you created). ```yaml -POST /rpc?_format=yaml +POST /fhir/BatchValidationRun/$run content-type: text/yaml -method: aidbox.validation/batch-validation-result -params: - id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 - # problems are paged; defaults: page 0, pageSize 100 - # page: 0 - # pageSize: 100 +resource: Patient +profiles: + - http://hl7.org/fhir/us/core/StructureDefinition/us-core-patient +``` -# response -status: 200 -result: - run: - id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 - resource: patient - status: complete # in-progress | complete | cancelled | failed - invalid: 2 - resourceType: BatchValidationRun - status: completed # in-progress | completed | cancelled | failed - problems: - - resource: - id: pt1 - resourceType: Patient - errors: - - path: ups - type: unknown-key +{% hint style="info" %} +**Multiple profiles are conjunctive (AND).** All listed profiles are applied together (via `meta.profile`), exactly like the FHIR `$validate` operation: a resource is compliant only if it conforms to **every** profile, and the findings are the union of violations across them. + +To express **OR** ("valid against US Core 6 **or** 7"), run one validation per profile and intersect the offender sets: a resource is OR‑invalid only if it appears in **every** run's offenders (i.e. it failed all of them). +{% endhint %} + +{% hint style="warning" %} +If a profile URL cannot be resolved and strict profile resolution is disabled, it is silently skipped — so a typo yields a falsely "compliant" report. Make sure the profile (and its package) is installed and the URL matches. +{% endhint %} + +## Filtering by date range + +To validate only resources loaded (or modified) in a time window, use the FHIR date search parameters with a prefix (`ge`, `gt`, `le`, `lt`) — `createdAt` filters by load time (`cts`), `_lastUpdated` by modification time (`ts`). A closed range is the same parameter twice. These are most naturally supplied in a `Parameters` body: + +```yaml +POST /fhir/BatchValidationRun/$run +content-type: application/json + +resourceType: Parameters +parameter: + - name: resource + valueString: Patient + # loaded in the week before last: createdAt in [now-14d, now-7d) + - name: createdAt + valueString: 'ge2025-06-02' + - name: createdAt + valueString: 'lt2025-06-09' ``` -`status` reflects the live state of the scheduled chunk tasks. `run.invalid` is the total number of invalid resources recorded so far. `problems` is paged with `page`/`pageSize`, so large result sets can be inspected fully. +- *Loaded over the last week*: a single `createdAt` = `ge`. +- *Modified in a range*: use `_lastUpdated` with the same prefix grammar. + +{% hint style="info" %} +`createdAt` (load time) is usually what you want for "newly loaded data"; `_lastUpdated` is the FHIR-standard parameter and reflects the last modification. Date filters compose with `profiles`, `include`/`exclude`, and `incremental` — they are all additional conditions on the same scan. +{% endhint %} -#### aidbox.validation/clear-batch-validation +## Incremental validation -When you do not need results of this validation you can clean up resources with: +For recurring runs, set `incremental: true` and a stable `configId`. The first run validates everything; each subsequent run validates only resources written since the previous run of that `configId` (tracked by a transaction-id watermark). This makes nightly "re-check what changed" runs cheap. The watermark only advances when a run completes with full coverage. ```yaml -POST /rpc?_format=yaml +POST /fhir/BatchValidationRun/$run content-type: text/yaml -method: aidbox.validation/clear-batch-validation -params: - id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 +include: [Patient, Observation] +profiles: ['http://hl7.org/fhir/us/core/StructureDefinition/us-core-patient'] +incremental: true +configId: nightly-uscore ``` -This deletes the `BatchValidationRun`, its `BatchValidationError` resources and the scheduler bookkeeping for the operation. +{% hint style="warning" %} +**Incremental tracks writes, so it can miss newly-invalid references.** A resource is re-validated only when **its own** transaction id changes (a create or update). But a reference becomes dangling when its **target** is deleted — with no write to the referencing resource, so its txid does not move and an incremental run will **not** re-check it. Incremental mode is for "validate freshly loaded or modified data"; to re-verify referential integrity across data that didn't change, run a full (non-incremental) validation. +{% endhint %} -#### BatchValidationRun & BatchValidationError Resources +{% hint style="warning" %} +**One `configId` ⇄ one fixed scope and profile set.** The watermark is keyed by `configId` alone — not by the `include`/`exclude`/`profiles` of the run. If you reuse a `configId` for a different scope (e.g. Patients one night, Observations the next), the second run starts from the first run's watermark and **silently skips** everything written before it that the new scope never validated. Use a distinct `configId` per recurring stream. +{% endhint %} -When you run a validation operation Aidbox internally creates a BatchValidationRun resource and puts errors of validation in BatchValidationError. You can access these resources through standard CRUD/Search API — for example, to aggregate errors by type or build a report: +## How results are stored -```yaml -GET /BatchValidationError?.run.id=c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99&_format=yaml&_result=array +Results are stored in an **aggregated, compact** form rather than as one record per violation. The goal is that validating 100 GB of non-conformant data does **not** add 100 GB to the database. The `aidbox_batch_validation` schema holds four tables: -# response -- run: - id: c2b2bcb9-30f3-4632-9a8a-1d04b3a14b99 - resourceType: BatchValidationRun - errors: - - path: ups - type: unknown-key - resource: - id: pt1 - resourceType: Patient - id: 6c8c5045-71b8-43d4-9e44-b3a0bfbe6e54 -``` +| Table | Keyed by | Holds | +| --- | --- | --- | +| `finding` | run id | one row per error **pattern** + its `count` | +| `offender` | run id | full `(pattern, resource_id)` index — ids only | +| `run_stat` | run id | the `scanned` counter for the run | +| `watermark` | `configId` | incremental cursor: last validated transaction id per stream | + +### Findings — aggregate by error pattern + +A **finding** is one row per distinct **error pattern**, with a count. Patterns are bounded by profile structure (a few thousand at most), not by data volume, so the findings table stays small regardless of how many resources are invalid. + +The aggregation key — what defines a "pattern" — is: + +| Field | Meaning | Example | +| --- | --- | --- | +| `profile_url` | which schema/profile raised the error | `us-core-patient` | +| `resource_type` | the resource type | `Patient` | +| `norm_path` | element path with **array indices stripped** | `identifier[2].system` → `identifier.system` | +| `code` | the kind of error | `unknown-key`, `cardinality`, `required`, `invalid-type`, `binding`, `reference` | +| `constraint_key` | the constraint / slice / binding id, when applicable | `us-core-8` | + +All occurrences sharing these fields collapse into one finding. Index normalization is key: every array element of the same element (`identifier[0]`, `identifier[1]`, …) folds into one `identifier.system` pattern, giving the clean "this path is the problem on N resources" view. Each finding keeps only a running `count` — no resource ids are stored on the finding itself. + +### Offenders — the full pattern → resource index + +The **offender** index is where resource ids live: one tiny row per `(pattern, resource_id)` pair — ids only, no resource bodies. This is what `$offenders` reads to return every resource hitting a pattern. Storage is ids-only (tens of bytes per row), so the full drill-down list is available without storing copies of the invalid resources. + +### Scanned counter and incremental watermark + +`run_stat` keeps the per-run `scanned` total (resources examined), incremented additively as chunks complete; it feeds `compliance.scanned`. `watermark` is unrelated to results — it is the incremental cursor: one row per `configId` recording the last transaction id validated by that stream, advanced only when a run completes with full chunk coverage (see [Incremental validation](#incremental-validation)). + +### What is not stored + +The invalid resources themselves, their OperationOutcomes, and the human-readable messages are **not** stored. The finding `message` and the per-resource `errors` are **reconstructed at read time** from the stored machine fields (`code`, `path`, `constraint_key`, …). As a result the reconstructed errors are approximate: the path is index-normalized and the offending value is not named in the diagnostics. + +The `problems` view's `resource` body is the one exception — it is not stored either, but it is **re-read live from the primary resource table** when you call `$result` (so the actual offending value *is* visible there, just sourced from the current resource, not the validated snapshot; deleted resources fall back to an `{id, resourceType}` stub). + +Writes are batched per chunk and merge with `INSERT … ON CONFLICT DO UPDATE SET count = count + …`, so chunks running concurrently across workers and nodes accumulate into the same findings without coordination. + +## Terminology -Both resources are persisted in the database, so validation results survive Aidbox restarts. +Coded-binding and slice validation may call the configured terminology server. If the server is unreachable or returns an error, the affected resource is recorded as invalid with a `terminology-unavailable` finding (naming the server), and the run still completes — it is not aborted. Point `fhir.terminology.service-base-url` at a reachable server (ideally with a local/hybrid engine) for accurate coded validation. From 7cc926745621a366acb913f269afe5ef7af58b63 Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Mon, 29 Jun 2026 18:54:09 +0300 Subject: [PATCH 04/14] rework batch validation api --- SUMMARY.md | 2 +- .../asynchronous-resource-validation.md | 378 +++++++----------- 2 files changed, 149 insertions(+), 231 deletions(-) diff --git a/SUMMARY.md b/SUMMARY.md index 2fb58cd47..bd82052af 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -286,7 +286,7 @@ * [FHIR Schema Validator](modules/profiling-and-validation/fhir-schema-validator/README.md) * [Setup Aidbox with FHIR Schema validation engine](modules/profiling-and-validation/fhir-schema-validator/setup-aidbox-with-fhir-schema-validation-engine.md) * [Skip validation of references in resource using request header](modules/profiling-and-validation/skip-validation-of-references-in-resource-using-request-header.md) - * [Asynchronous resource validation](modules/profiling-and-validation/asynchronous-resource-validation.md) + * [Batch resource validation](modules/profiling-and-validation/asynchronous-resource-validation.md) * [Observability](modules/observability/README.md) * [Getting started](modules/observability/getting-started/README.md) * [Run Aidbox with OpenTelemetry locally](modules/observability/getting-started/run-aidbox-with-opentelemetry-locally.md) diff --git a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md b/docs/modules/profiling-and-validation/asynchronous-resource-validation.md index cc8b63889..d4a34ef1f 100644 --- a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md +++ b/docs/modules/profiling-and-validation/asynchronous-resource-validation.md @@ -1,314 +1,232 @@ --- description: >- Validate resources already stored in the database against their schemas and - FHIR profiles, asynchronously and at scale, and analyze the results. + FHIR profiles, at scale, via a resource-level $batch-validate operation, + synchronously or asynchronously, then drill into the offending resources. --- -# Asynchronous resource validation +# Batch resource validation + +{% hint style="info" %} +Available in Aidbox starting from version **2606**. +{% endhint %} {% hint style="danger" %} -**Breaking change since version 2607.** Batch validation was reworked onto the asynchronous engine and a new API. If you used it before 2607: +**Breaking change.** Batch validation is a **resource-type-level FHIR operation**, `POST /fhir//$batch-validate`, following the FHIR Async pattern (like `$purge`). If you used the earlier `BatchValidationRun` API: -* The RPC procedures (`aidbox.validation/batch-validation`, `resources-batch-validation-task`, `batch-validation-result`, `clear-batch-validation`) are **removed**. Use the FHIR operations on `BatchValidationRun` described below. -* `BatchValidationError` resources are **no longer produced**. Results are stored in an aggregated form and read via `$result` / `$offenders` (see [How results are stored](#how-results-are-stored)). -* The deprecated AWF/legacy engine and the `batch-validation-legacy-engine` (`BOX_BATCH_VALIDATION_LEGACY_ENGINE`) setting are **removed**; an asynchronous task scheduler is now required. -* The `filter`, `limit`, and `async` parameters are **no longer supported**. +* `BatchValidationRun` and its operations (`$run`, `$result`, `$offenders`, `$clear`) are **removed**. +* There is **no validate-all-types** operation. You validate one resource type per call (scope further with `_since`/`_until`). +* **Incremental validation** (`incremental`/`configId`) and **`errorsThreshold`** are removed. +* The `batch-validation-max-batch-size` / `batch-validation-max-refs-in-flight` **settings** are removed. They are now per-request parameters (`max-batch-size` / `max-ref-size`). {% endhint %} ## Overview -Batch validation checks resources that are **already in the database** against the active FHIR schemas and, optionally, against a set of profiles. Use it when you loaded data with validation off, or when you publish a new version of a profile and want to know **how many existing resources are non-compliant and why**. - -A run validates one or more resource types. Each table is split into id-range **chunks** that are executed in parallel by the asynchronous task scheduler, so a single large table fans out across workers. Results are persisted in a compact, aggregated form (see [How results are stored](#how-results-are-stored)) and survive Aidbox restarts. - -The API is a set of FHIR operations on the `BatchValidationRun` resource: - -| Operation | Purpose | -| --- | --- | -| `POST /fhir/BatchValidationRun/$run` | Start a run | -| `GET /fhir/BatchValidationRun/{id}/$result` | Status, compliance summary, findings, per‑resource problems | -| `GET /fhir/BatchValidationRun/{id}/$offenders` | All resource ids hitting one finding pattern | -| `POST /fhir/BatchValidationRun/{id}/$clear` | Delete a run and its results | - -{% hint style="info" %} -An asynchronous task scheduler is required. The number of executor threads is controlled by the `scheduler-executors` setting (`BOX_SCHEDULER_EXECUTORS`, default `4`) — raise it to validate large datasets faster. Chunks are also distributed across nodes in a clustered deployment. -{% endhint %} +Batch validation checks resources **already in the database** against the active FHIR schemas and an optional set of profiles. Use it when you loaded data with validation off, or when you publish a new profile version and want to know **how many existing resources are non-compliant and why**. -## Start a run +`$batch-validate` runs against **one resource type**. Aidbox splits the table into id-range **chunks**, validates them in parallel, and aggregates the results into a compact, offender-indexed form (see [How results are stored](#how-results-are-stored)). -`POST /fhir/BatchValidationRun/$run` starts a run and returns immediately with a run id; validation proceeds in the background. +It works two ways, chosen by the `Prefer` header: -The body can be either a FHIR **`Parameters`** resource (the canonical operation-input form) or a **plain JSON** object with the same fields (an accepted shorthand). The examples below use the plain JSON shorthand; the equivalent `Parameters` form is shown under [Body shapes](#body-shapes). +| | Trigger | Response | +| --- | --- | --- | +| **Synchronous** (default) | `POST …/$batch-validate` | blocks, returns a `Parameters` summary | +| **Asynchronous** | same + `Prefer: respond-async` | `202` + `Content-Location`; poll for the result | -```yaml -POST /fhir/BatchValidationRun/$run -content-type: text/yaml - -# What to validate — choose one of: -resource: Patient # a single resource type -# include: [Patient, Observation] # several types -# exclude: [Provenance] # all types except these -# (omit all three to validate every persistable resource type) - -# Optional: -# id: my-run # client-supplied run id (rejected if it already exists) -# profiles: ['http://hl7.org/fhir/us/core/StructureDefinition/us-core-patient'] -# errorsThreshold: 10000 # cancel the run once this many violations are recorded -# chunkSize: 1000 # max rows per chunk task (byte-bounded, see note) -# incremental: true # only validate resources changed since the last run of this stream -# configId: nightly-uscore # incremental stream id - -# response -status: 202 -Content-Location: /fhir/BatchValidationRun/my-run/$result -result: - run-id: my-run - operation-id: my-run - status: in-progress -``` +Both paths produce the same result under a **`task-id`** and persist it the same way, so you drill into a synchronous run the same as an asynchronous one. {% hint style="info" %} -`include` and `exclude` are mutually exclusive. With none of `resource`/`include`/`exclude`, every persistable resource type is validated. +Synchronous validation blocks the request until it finishes. That suits a type scoped by a narrow `_since`/`_until` window; for a large type, use `Prefer: respond-async`. Both paths run chunks in parallel on a local pool sized by `scheduler-executors` (`BOX_SCHEDULER_EXECUTORS`, default `4`); the async path also distributes chunks across nodes via the task scheduler. {% endhint %} -`errorsThreshold` (alias `error-threshold`) cancels the whole operation once that many violations are recorded — remaining chunks are stopped and the run status becomes `cancelled`. - -`chunkSize` is an **upper bound**, not a fixed size. The planner caps each chunk by stored bytes (~8 MB), so heavy resource types (e.g. Provenance) get fewer rows per chunk than requested while light types use the full `chunkSize`. This keeps a single chunk's in-memory footprint bounded regardless of resource size. - -### Body shapes - -`$run` accepts the spec in two shapes, both producing the same run: - -* **FHIR `Parameters`** (canonical). Multi-valued fields (`include`, `exclude`, `profiles`) and date ranges repeat the parameter; this is the form to use for the prefixed date filters (see [Filtering by date range](#filtering-by-date-range)). - - ```yaml - POST /fhir/BatchValidationRun/$run - content-type: application/json +## Start a validation - resourceType: Parameters - parameter: - - {name: include, valueString: Patient} - - {name: include, valueString: Observation} - - {name: profiles, valueString: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-patient'} - - {name: id, valueString: my-run} - ``` - -* **Plain JSON** (shorthand). Terser for the common case: - - ```yaml - POST /fhir/BatchValidationRun/$run - content-type: text/yaml - - include: [Patient, Observation] - profiles: ['http://hl7.org/fhir/us/core/StructureDefinition/us-core-patient'] - id: my-run - ``` - -## Get results - -`GET /fhir/BatchValidationRun/{id}/$result` returns the run state and three views of the results. Poll it until `status` is terminal (`complete`, `failed`, or `cancelled`). +`POST /fhir//$batch-validate` with a FHIR `Parameters` body: ```yaml -GET /fhir/BatchValidationRun/my-run/$result?pageSize=50 -# optional query params: page (0-based), pageSize, severity (e.g. "error") +POST /fhir/Observation/$batch-validate +content-type: application/json -# response -status: 200 -result: - run: - id: my-run - resourceType: BatchValidationRun - status: complete # in-progress | complete | cancelled | failed - invalid: 14 # total violation occurrences (see note below) - status: complete - compliance: - scanned: 1127 # resources validated - compliant: 1115 # resources with no violations - non-compliant: 12 # distinct resources with at least one violation (14 occurrences across them) - percent: 98.94 - findings: # aggregate, worst (highest count) first - - path: bogusKey - code: unknown-key - resource_type: Patient - profile_url: '' - constraint_key: '' - severity: error - count: 3 - pattern_hash: 5b4b07e46a65a5f0 - message: 'Patient.bogusKey: element is not allowed by the profile' - problems: # per-resource view (paged by page/pageSize) - - resource: - id: p-1 - resourceType: Patient - gender: 5 - errors: - - path: gender - type: invalid-type - severity: error - diagnostics: 'Patient.gender: value has the wrong type' +resourceType: Parameters +parameter: + # required: only resources whose meta.lastUpdated >= _since + - {name: _since, valueInstant: '2025-06-02T00:00:00Z'} + # optional upper bound (exclusive) + - {name: _until, valueInstant: '2025-06-09T00:00:00Z'} + # validate against these profiles (conjunctive, see Profiles) + - {name: profile, valueCanonical: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'} + # memory tuning (optional) + - {name: max-batch-size, valuePositiveInt: 256} + - {name: max-ref-size, valuePositiveInt: 2000} ``` -The result has three parts: - -* **`compliance`** — the headline rollup: how many resources were `scanned`, how many are `compliant` vs `non-compliant` (distinct resources), and the `percent` compliant. -* **`findings`** — the aggregate "which paths fail and how often" view, worst-first. Each finding is one error **pattern** with a `count`, a reconstructed human `message`, and a `pattern_hash` for drill-down (the resource ids hitting the pattern are fetched separately via `$offenders`). Pass `severity=error` to see only errors. -* **`problems`** — the per-resource view (the offending resource plus its reconstructed errors), paged by `page`/`pageSize`. The `resource` body is **re-read live from the primary table** at read time (it is not stored by the run), so it reflects the resource's **current** state — if it was edited after the run it may no longer match the errors, and if it was deleted the body collapses to an `{id, resourceType}` stub. +| Parameter | Type | Meaning | +| --- | --- | --- | +| `_since` **(required)** | `instant` | Only resources whose `meta.lastUpdated >= _since` (inclusive). Required so that a run declares a window instead of scanning a whole type (see [Filtering by date](#filtering-by-date)). | +| `_until` | `instant` | Upper bound: `meta.lastUpdated < _until` (exclusive). | +| `profile` (repeatable) | `canonical` | Validate every resource against these profile URLs (in addition to its base schema), conjunctively (see [Profiles](#profiles)). | +| `max-batch-size` (default `256`) | `positiveInt` | Resources read and validated in memory at once. Bounds heap per executor. | +| `max-ref-size` (default `20000`) | `positiveInt` | References pooled before a batched existence check. The binding heap constraint for reference-dense types. | {% hint style="warning" %} -`run.invalid` and the sum of finding `count`s are **violation occurrences** — one resource that breaks three rules contributes three. `compliance.non-compliant` is the number of **distinct resources** with at least one violation. They are different numbers; use `compliance` for "how many of my resources are bad". - -If chunks fail or vanish, the run also records synthetic `chunk-validation-failed` / `chunk-incomplete` findings (so a partially-covered run can't read as clean — see [Run status](#run-status-as-a-resource)). These count toward `run.invalid` and appear in `findings`, but are **excluded** from `compliance.non-compliant`. So on a healthy run `run.invalid` is pure violation occurrences; on a degraded run it is inflated by the failure markers — another reason to read `compliance` for the real picture. +The body must be a valid `Parameters` resource. Each parameter must use the **exact** `value[x]` type above (`profile` as `valueCanonical`, `_since` as `valueInstant`, the sizes as `valuePositiveInt`). Aidbox rejects an unknown parameter, a wrong value type, or a missing `_since` with `422` and an `OperationOutcome` that names the offending parameter. {% endhint %} -## Drill into a finding +## Synchronous response -A finding tells you *which* pattern fails and *how many* resources hit it, but not *which* resources. To get the resource ids, take the finding's `pattern_hash` and call `$offenders`: +A `Parameters` resource holds the `task-id`, the headline counts, a link to the offending resources, and one `issue` per distinct error pattern (each with its own filtered drill-down link). ```yaml -GET /fhir/BatchValidationRun/my-run/$offenders?pattern=5b4b07e46a65a5f0&pageSize=1000 -# optional: page (0-based), pageSize - -# response -result: - run-id: my-run - pattern-hash: 5b4b07e46a65a5f0 - resource-ids: [p-4, p-8, p-12] +status: 200 + +resourceType: Parameters +parameter: + - {name: task-id, valueString: ''} + - {name: validated, valueUnsignedInt: 1804646} # resources validated + - {name: valid, valueUnsignedInt: 1317494} # resources with no issues + - {name: invalid, valueUnsignedInt: 487152} # distinct resources with ≥1 issue + - {name: invalid-resources, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources'} + - name: issue + part: + - {name: id, valueString: '5b4b07e4…'} # issue-id (for drill-down) + - {name: invalid-resources, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources?_issue=5b4b07e4…'} + - {name: severity, valueCode: error} + - {name: code, valueCode: invalid-slice-cardinality} + - {name: expression, valueString: category} # the element + - {name: profile, valueString: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'} + - {name: count, valueUnsignedInt: 486018} # distinct offending resources + - {name: constraint, valueString: us-core-8} # for invariants + - {name: diagnostics, valueString: '…human-readable message…'} ``` -## Clear a run +* **`count`** is the number of **distinct offending resources** for the pattern, derived from the offender index. +* For **invariant** issues, the `constraint` part carries the constraint key and `diagnostics` carries the validator's human-readable description. +* Each `issue` carries its own `invalid-resources` link, pre-filtered to that issue. +* On failure the response is an `OperationOutcome`. -```yaml -POST /fhir/BatchValidationRun/my-run/$clear +## Asynchronous response -# response -result: - message: "Batch validation results for run.id='my-run' are cleared" +``` +status: 202 Accepted +Content-Location: http://localhost:8765/fhir/$batch-validate/ ``` -This deletes the `BatchValidationRun`, all of its stored findings, offenders, and scanned counter, and the scheduler bookkeeping for the operation. (The incremental `watermark` is keyed by `configId`, not by run id, so it is left intact.) - -## Run status as a resource +The endpoints below are **system-level**, keyed by `task-id` alone (no resource type in the path). -`BatchValidationRun` is a regular resource; you can read or search it through the standard API to list runs and check status: +Poll the `Content-Location`: ```yaml -GET /fhir/BatchValidationRun/my-run +GET /fhir/$batch-validate/ ``` -## Validating against profiles +* **In progress** → `202 Accepted` with an `X-Progress` header (resources scanned so far). +* **Complete** → `200` with the same `Parameters` summary as the synchronous response. +* **Cancelled** → `200` `OperationOutcome` (`cancelled`); **failed** → `200` `OperationOutcome`. +* **Unknown task** → `404`. -Pass `profiles` to validate every resource against those profile URLs in addition to its base schema — this is how you check the impact of a new profile version on existing data. The profiles must be loaded (e.g. an installed IG package, or a `StructureDefinition` you created). +## Drill into the offending resources -```yaml -POST /fhir/BatchValidationRun/$run -content-type: text/yaml +The summary tells you which patterns fail and how many resources hit each. To get the **offending resources**, each linked to the version that was validated and carrying its full `OperationOutcome`, call `invalid-resources`: -resource: Patient -profiles: - - http://hl7.org/fhir/us/core/StructureDefinition/us-core-patient +``` +GET /fhir/$batch-validate//invalid-resources + ?_issue=&_issue=&_count=50&_page=1&_fullurl-only=false ``` -{% hint style="info" %} -**Multiple profiles are conjunctive (AND).** All listed profiles are applied together (via `meta.profile`), exactly like the FHIR `$validate` operation: a resource is compliant only if it conforms to **every** profile, and the findings are the union of violations across them. - -To express **OR** ("valid against US Core 6 **or** 7"), run one validation per profile and intersect the offender sets: a resource is OR‑invalid only if it appears in **every** run's offenders (i.e. it failed all of them). -{% endhint %} - -{% hint style="warning" %} -If a profile URL cannot be resolved and strict profile resolution is disabled, it is silently skipped — so a typo yields a falsely "compliant" report. Make sure the profile (and its package) is installed and the URL matches. -{% endhint %} - -## Filtering by date range +| Query parameter | Meaning | +| --- | --- | +| `_issue` (repeatable) | Restrict to offenders of these issue(s). Omit for **all** offending resources. | +| `_count` / `_page` | Page size (default `50`) and 1-based page number. | +| `_fullurl-only` (default `false`) | When `true`, return only the `fullUrl` of each offender (omit the body and outcome). | -To validate only resources loaded (or modified) in a time window, use the FHIR date search parameters with a prefix (`ge`, `gt`, `le`, `lt`) — `createdAt` filters by load time (`cts`), `_lastUpdated` by modification time (`ts`). A closed range is the same parameter twice. These are most naturally supplied in a `Parameters` body: +The response is a **`Parameters` report** rather than a Bundle (see [why](#why-a-parameters-report)): a `total`, flat paging links, and one repeated `resource` parameter per offending resource. ```yaml -POST /fhir/BatchValidationRun/$run -content-type: application/json - resourceType: Parameters parameter: + - {name: total, valueUnsignedInt: 1114} + # paging: flat links. first/previous appear past page 1; next/last before the last page + - {name: self, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources?_count=50&_page=1'} + - {name: next, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources?_count=50&_page=2'} + - {name: last, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources?_count=50&_page=23'} - name: resource - valueString: Patient - # loaded in the week before last: createdAt in [now-14d, now-7d) - - name: createdAt - valueString: 'ge2025-06-02' - - name: createdAt - valueString: 'lt2025-06-09' + part: + - {name: fullUrl, valueUrl: 'http://localhost:8765/Observation//_history/'} + - name: resource # omitted when _fullurl-only=true + resource: {resourceType: Observation, meta: {versionId: ''}, …} + - name: outcome # omitted when _fullurl-only=true + resource: + resourceType: OperationOutcome + issue: + - {severity: error, code: structure, expression: [Observation.category], + diagnostics: '…', details: {coding: [{code: invalid-slice-cardinality}]}} ``` -- *Loaded over the last week*: a single `createdAt` = `ge`. -- *Modified in a range*: use `_lastUpdated` with the same prefix grammar. +* Each `resource` parameter is **one distinct offending resource**: its versioned `fullUrl`, the `resource` body (read from history at the validated version), and its `outcome`, an `OperationOutcome` listing **every** issue that resource has (its full issue set, even when `_issue` narrows which resources come back). +* `_fullurl-only=true` drops the `resource` and `outcome` parts and keeps the `fullUrl`. +* Paging is flat: `self` always; `first` and `previous` once past page 1; `next` and `last` while before the last page. +* An unknown `_issue` on a known task returns an empty report (`total: 0`). An unknown task returns `404`. {% hint style="info" %} -`createdAt` (load time) is usually what you want for "newly loaded data"; `_lastUpdated` is the FHIR-standard parameter and reflects the last modification. Date filters compose with `profiles`, `include`/`exclude`, and `incremental` — they are all additional conditions on the same scan. +The `fullUrl` is **version-specific** (`/_history/`), so a vread resolves to the resource version that was validated rather than the current one. (If that version was pruned, the body is absent; the `fullUrl` and `outcome` remain.) {% endhint %} -## Incremental validation - -For recurring runs, set `incremental: true` and a stable `configId`. The first run validates everything; each subsequent run validates only resources written since the previous run of that `configId` (tracked by a transaction-id watermark). This makes nightly "re-check what changed" runs cheap. The watermark only advances when a run completes with full coverage. - -```yaml -POST /fhir/BatchValidationRun/$run -content-type: text/yaml +## Cancel -include: [Patient, Observation] -profiles: ['http://hl7.org/fhir/us/core/StructureDefinition/us-core-patient'] -incremental: true -configId: nightly-uscore +``` +DELETE /fhir/$batch-validate/ ``` -{% hint style="warning" %} -**Incremental tracks writes, so it can miss newly-invalid references.** A resource is re-validated only when **its own** transaction id changes (a create or update). But a reference becomes dangling when its **target** is deleted — with no write to the referencing resource, so its txid does not move and an incremental run will **not** re-check it. Incremental mode is for "validate freshly loaded or modified data"; to re-verify referential integrity across data that didn't change, run a full (non-incremental) validation. -{% endhint %} +Responds `202 Accepted`. Cancellation removes the run's **pending** chunks and marks it **cancelled** (a later poll reports `cancelled`). Aidbox does not interrupt a chunk already running, and keeps partial results. -{% hint style="warning" %} -**One `configId` ⇄ one fixed scope and profile set.** The watermark is keyed by `configId` alone — not by the `include`/`exclude`/`profiles` of the run. If you reuse a `configId` for a different scope (e.g. Patients one night, Observations the next), the second run starts from the first run's watermark and **silently skips** everything written before it that the new scope never validated. Use a distinct `configId` per recurring stream. -{% endhint %} +## Profiles -## How results are stored +Pass `profile` (as `valueCanonical`) to validate every resource against those URLs (via `meta.profile`), say to check the impact of a new profile version on existing data. The profiles must be loaded (an installed IG package, or a `StructureDefinition` you created). -Results are stored in an **aggregated, compact** form rather than as one record per violation. The goal is that validating 100 GB of non-conformant data does **not** add 100 GB to the database. The `aidbox_batch_validation` schema holds four tables: +{% hint style="info" %} +**Multiple profiles are conjunctive (AND).** A resource is compliant only if it conforms to **every** listed profile, and issues are the union of violations across them (same as FHIR `$validate`). -| Table | Keyed by | Holds | -| --- | --- | --- | -| `finding` | run id | one row per error **pattern** + its `count` | -| `offender` | run id | full `(pattern, resource_id)` index — ids only | -| `run_stat` | run id | the `scanned` counter for the run | -| `watermark` | `configId` | incremental cursor: last validated transaction id per stream | +For **OR** ("valid against US Core 6 **or** 7"), run one validation per profile and intersect the offender sets: a resource is OR-invalid only if it appears in **every** run's offenders. +{% endhint %} -### Findings — aggregate by error pattern +{% hint style="warning" %} +Aidbox skips an unresolvable profile URL when strict profile resolution is off, so a typo yields a "compliant" report that is wrong. Make sure the profile (and its package) is installed and the URL matches. +{% endhint %} -A **finding** is one row per distinct **error pattern**, with a count. Patterns are bounded by profile structure (a few thousand at most), not by data volume, so the findings table stays small regardless of how many resources are invalid. +## Filtering by date -The aggregation key — what defines a "pattern" — is: +`_since` and `_until` are FHIR `instant`s that filter on **`meta.lastUpdated`**, following the bulk-export `_since`/`_until` semantics: -| Field | Meaning | Example | -| --- | --- | --- | -| `profile_url` | which schema/profile raised the error | `us-core-patient` | -| `resource_type` | the resource type | `Patient` | -| `norm_path` | element path with **array indices stripped** | `identifier[2].system` → `identifier.system` | -| `code` | the kind of error | `unknown-key`, `cardinality`, `required`, `invalid-type`, `binding`, `reference` | -| `constraint_key` | the constraint / slice / binding id, when applicable | `us-core-8` | +```yaml +parameter: + - {name: _since, valueInstant: '2025-06-02T00:00:00Z'} # lastUpdated in [2025-06-02, 2025-06-09) + - {name: _until, valueInstant: '2025-06-09T00:00:00Z'} +``` -All occurrences sharing these fields collapse into one finding. Index normalization is key: every array element of the same element (`identifier[0]`, `identifier[1]`, …) folds into one `identifier.system` pattern, giving the clean "this path is the problem on N resources" view. Each finding keeps only a running `count` — no resource ids are stored on the finding itself. +* `_since` is an **inclusive** lower bound (`lastUpdated >= _since`); `_until` is an **exclusive** upper bound (`lastUpdated < _until`). +* `_since` is **required**: every run declares a window, so no call scans a whole (possibly huge) type by accident. To validate everything, pass an epoch `_since` (`1970-01-01T00:00:00Z`). +* A narrow window keeps a synchronous run small. Date filters compose with `profile`. -### Offenders — the full pattern → resource index +## How results are stored -The **offender** index is where resource ids live: one tiny row per `(pattern, resource_id)` pair — ids only, no resource bodies. This is what `$offenders` reads to return every resource hitting a pattern. Storage is ids-only (tens of bytes per row), so the full drill-down list is available without storing copies of the invalid resources. +Aidbox stores results in an **aggregated, compact** form, so validating 100 GB of non-conformant data does **not** add 100 GB to the database. The `aidbox_batch_validation` schema holds: -### Scanned counter and incremental watermark +| Table | Holds | +| --- | --- | +| `issue` | one row per distinct error **pattern** (no per-resource rows) | +| `offender` | a tiny `(issue_id, resource_id, version_id)` row per offending resource: ids and versions only | +| `run_stat` | the `scanned` counter | -`run_stat` keeps the per-run `scanned` total (resources examined), incremented additively as chunks complete; it feeds `compliance.scanned`. `watermark` is unrelated to results — it is the incremental cursor: one row per `configId` recording the last transaction id validated by that stream, advanced only when a run completes with full chunk coverage (see [Incremental validation](#incremental-validation)). +A **pattern** is the aggregation key: `profile`, `resource_type`, index-normalized `path` (`identifier[2].system` → `identifier.system`), `code`, and `constraint_key`. All occurrences that share these collapse into one issue; the issue's **count is the number of offender rows** (distinct resources). Invariant issues also keep the validator's `human` description. -### What is not stored +Aidbox does **not** store the invalid resource bodies or their `OperationOutcome`s. The drill-down re-reads the body from history at the validated version and reconstructs the `OperationOutcome` from the stored machine fields (`code`, `expression`, `constraint`, `human`). -The invalid resources themselves, their OperationOutcomes, and the human-readable messages are **not** stored. The finding `message` and the per-resource `errors` are **reconstructed at read time** from the stored machine fields (`code`, `path`, `constraint_key`, …). As a result the reconstructed errors are approximate: the path is index-normalized and the offending value is not named in the diagnostics. +Both synchronous and asynchronous runs persist these tables under the run's `task-id`, so you poll and drill into either until you cancel it. -The `problems` view's `resource` body is the one exception — it is not stored either, but it is **re-read live from the primary resource table** when you call `$result` (so the actual offending value *is* visible there, just sourced from the current resource, not the validated snapshot; deleted resources fall back to an `{id, resourceType}` stub). +## Why a Parameters report -Writes are batched per chunk and merge with `INSERT … ON CONFLICT DO UPDATE SET count = count + …`, so chunks running concurrently across workers and nodes accumulate into the same findings without coordination. +The `invalid-resources` response is a `Parameters` resource rather than a `Bundle`. A `searchset`/`collection` Bundle cannot carry a `total`, **version-specific** links, a per-offender `OperationOutcome`, and the invalid resource bodies while staying FHIR-valid: Bundle invariants forbid a version-specific `fullUrl`, restrict `total` and `entry.response` by Bundle type, and the FHIR validator validates embedded resources in full. A `Parameters` report avoids those constraints, keeps the versioned drill-down links, and embeds each `OperationOutcome` without trouble. The embedded resource bodies are the report's payload: the invalid data under review. ## Terminology -Coded-binding and slice validation may call the configured terminology server. If the server is unreachable or returns an error, the affected resource is recorded as invalid with a `terminology-unavailable` finding (naming the server), and the run still completes — it is not aborted. Point `fhir.terminology.service-base-url` at a reachable server (ideally with a local/hybrid engine) for accurate coded validation. +Coded-binding and slice validation may call the configured terminology server. If it is unreachable or errors, Aidbox records the affected resource with a `terminology-unavailable` finding (which names the server) and the run still completes. Point `fhir.terminology.service-base-url` at a reachable server (a local or hybrid engine works best) for accurate, fast coded validation. From 5f142a4d2b2f2ff64233d179e36ead2c72411774 Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Fri, 3 Jul 2026 13:10:23 +0300 Subject: [PATCH 05/14] update docs to match current impl --- SUMMARY.md | 2 +- docs/api/bulk-api/import-and-fhir-import.md | 2 +- docs/api/bulk-api/load-and-fhir-load.md | 2 +- ...dation.md => batch-resource-validation.md} | 40 ++++++++----------- docs/overview/faq.md | 2 +- .../migrate-to-fhirschema/README.md | 2 +- redirects.yaml | 3 +- 7 files changed, 23 insertions(+), 30 deletions(-) rename docs/modules/profiling-and-validation/{asynchronous-resource-validation.md => batch-resource-validation.md} (79%) diff --git a/SUMMARY.md b/SUMMARY.md index bd82052af..90ec27008 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -286,7 +286,7 @@ * [FHIR Schema Validator](modules/profiling-and-validation/fhir-schema-validator/README.md) * [Setup Aidbox with FHIR Schema validation engine](modules/profiling-and-validation/fhir-schema-validator/setup-aidbox-with-fhir-schema-validation-engine.md) * [Skip validation of references in resource using request header](modules/profiling-and-validation/skip-validation-of-references-in-resource-using-request-header.md) - * [Batch resource validation](modules/profiling-and-validation/asynchronous-resource-validation.md) + * [Batch resource validation](modules/profiling-and-validation/batch-resource-validation.md) * [Observability](modules/observability/README.md) * [Getting started](modules/observability/getting-started/README.md) * [Run Aidbox with OpenTelemetry locally](modules/observability/getting-started/run-aidbox-with-opentelemetry-locally.md) diff --git a/docs/api/bulk-api/import-and-fhir-import.md b/docs/api/bulk-api/import-and-fhir-import.md index b4ca2d636..cb084e256 100644 --- a/docs/api/bulk-api/import-and-fhir-import.md +++ b/docs/api/bulk-api/import-and-fhir-import.md @@ -18,7 +18,7 @@ Keep in mind that $import **does not validate** inserted resources for the sake {% endhint %} {% hint style="info" %} -Please consider using [Asynchronous validation API](../../modules/profiling-and-validation/asynchronous-resource-validation.md) to validate data after $import +Please consider using [Batch validation API](../../modules/profiling-and-validation/batch-resource-validation.md) to validate data after $import {% endhint %} ## Example diff --git a/docs/api/bulk-api/load-and-fhir-load.md b/docs/api/bulk-api/load-and-fhir-load.md index 49d67518c..edc4783d3 100644 --- a/docs/api/bulk-api/load-and-fhir-load.md +++ b/docs/api/bulk-api/load-and-fhir-load.md @@ -24,7 +24,7 @@ When loading resources with references, remember that '`/'` is {% endhint %} {% hint style="info" %} -Please consider using [Asynchronous validation API](../../modules/profiling-and-validation/asynchronous-resource-validation.md#asynchronous-batch-validation) to validate data after $load +Please consider using [Batch validation API](../../modules/profiling-and-validation/batch-resource-validation.md) to validate data after $load {% endhint %} Load 100 synthea Patients to Aidbox (see [tutorial](../../tutorials/bulk-api-tutorials/synthea-by-bulk-api.md)): diff --git a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md similarity index 79% rename from docs/modules/profiling-and-validation/asynchronous-resource-validation.md rename to docs/modules/profiling-and-validation/batch-resource-validation.md index d4a34ef1f..e07ca5441 100644 --- a/docs/modules/profiling-and-validation/asynchronous-resource-validation.md +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -8,23 +8,14 @@ description: >- # Batch resource validation {% hint style="info" %} -Available in Aidbox starting from version **2606**. -{% endhint %} - -{% hint style="danger" %} -**Breaking change.** Batch validation is a **resource-type-level FHIR operation**, `POST /fhir//$batch-validate`, following the FHIR Async pattern (like `$purge`). If you used the earlier `BatchValidationRun` API: - -* `BatchValidationRun` and its operations (`$run`, `$result`, `$offenders`, `$clear`) are **removed**. -* There is **no validate-all-types** operation. You validate one resource type per call (scope further with `_since`/`_until`). -* **Incremental validation** (`incremental`/`configId`) and **`errorsThreshold`** are removed. -* The `batch-validation-max-batch-size` / `batch-validation-max-refs-in-flight` **settings** are removed. They are now per-request parameters (`max-batch-size` / `max-ref-size`). +Available in Aidbox starting from version **2607**. {% endhint %} ## Overview Batch validation checks resources **already in the database** against the active FHIR schemas and an optional set of profiles. Use it when you loaded data with validation off, or when you publish a new profile version and want to know **how many existing resources are non-compliant and why**. -`$batch-validate` runs against **one resource type**. Aidbox splits the table into id-range **chunks**, validates them in parallel, and aggregates the results into a compact, offender-indexed form (see [How results are stored](#how-results-are-stored)). +`$batch-validate` runs against **one resource type**. Aidbox hash-partitions it into a **fixed number of tasks** (`number-of-chunks`, default `12`), each task validating its `mod(hash(id), N)` slice, and aggregates the results into a compact, offender-indexed form (see [How results are stored](#how-results-are-stored)). It works two ways, chosen by the `Prefer` header: @@ -36,7 +27,9 @@ It works two ways, chosen by the `Prefer` header: Both paths produce the same result under a **`task-id`** and persist it the same way, so you drill into a synchronous run the same as an asynchronous one. {% hint style="info" %} -Synchronous validation blocks the request until it finishes. That suits a type scoped by a narrow `_since`/`_until` window; for a large type, use `Prefer: respond-async`. Both paths run chunks in parallel on a local pool sized by `scheduler-executors` (`BOX_SCHEDULER_EXECUTORS`, default `4`); the async path also distributes chunks across nodes via the task scheduler. +Synchronous validation blocks the request until it finishes. That suits a type scoped by a narrow `_since`/`_until` window; for a large type, use `Prefer: respond-async`. The synchronous path runs the N tasks on a local pool sized by `scheduler-executors` (`BOX_SCHEDULER_EXECUTORS`, default `4`); the async path schedules them on the task scheduler, which spreads them across nodes. + +Each task scans the window once for its hash slice, so total scan work grows with the task count. More chunks means more parallelism (the async path spreads them across nodes) but more scans; fewer chunks means fewer scans but less parallelism. The default of `12` balances the two — raise `number-of-chunks` to parallelize a large type further. {% endhint %} ## Start a validation @@ -55,9 +48,8 @@ parameter: - {name: _until, valueInstant: '2025-06-09T00:00:00Z'} # validate against these profiles (conjunctive, see Profiles) - {name: profile, valueCanonical: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'} - # memory tuning (optional) - - {name: max-batch-size, valuePositiveInt: 256} - - {name: max-ref-size, valuePositiveInt: 2000} + # tuning (optional): number of parallel tasks (default 12, max 256) + - {name: number-of-chunks, valuePositiveInt: 24} ``` | Parameter | Type | Meaning | @@ -65,11 +57,10 @@ parameter: | `_since` **(required)** | `instant` | Only resources whose `meta.lastUpdated >= _since` (inclusive). Required so that a run declares a window instead of scanning a whole type (see [Filtering by date](#filtering-by-date)). | | `_until` | `instant` | Upper bound: `meta.lastUpdated < _until` (exclusive). | | `profile` (repeatable) | `canonical` | Validate every resource against these profile URLs (in addition to its base schema), conjunctively (see [Profiles](#profiles)). | -| `max-batch-size` (default `256`) | `positiveInt` | Resources read and validated in memory at once. Bounds heap per executor. | -| `max-ref-size` (default `20000`) | `positiveInt` | References pooled before a batched existence check. The binding heap constraint for reference-dense types. | +| `number-of-chunks` (default `12`, max `256`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task streams its slice, so heap stays bounded regardless. A value above `256` is rejected with `422`. | {% hint style="warning" %} -The body must be a valid `Parameters` resource. Each parameter must use the **exact** `value[x]` type above (`profile` as `valueCanonical`, `_since` as `valueInstant`, the sizes as `valuePositiveInt`). Aidbox rejects an unknown parameter, a wrong value type, or a missing `_since` with `422` and an `OperationOutcome` that names the offending parameter. +The body must be a valid `Parameters` resource. Each parameter must use the **exact** `value[x]` type above (`profile` as `valueCanonical`, `_since`/`_until` as `valueInstant`, `number-of-chunks` as `valuePositiveInt`). Aidbox rejects an unknown parameter, a wrong value type, or a missing `_since` with `422` and an `OperationOutcome` that names the offending parameter. {% endhint %} ## Synchronous response @@ -119,9 +110,10 @@ Poll the `Content-Location`: GET /fhir/$batch-validate/ ``` -* **In progress** → `202 Accepted` with an `X-Progress` header (resources scanned so far). +* **In progress** → `202 Accepted` with an `X-Progress` header (percent of tasks completed, e.g. `45%`). * **Complete** → `200` with the same `Parameters` summary as the synchronous response. -* **Cancelled** → `200` `OperationOutcome` (`cancelled`); **failed** → `200` `OperationOutcome`. +* **Cancelled** → `200` `OperationOutcome` (`cancelled`). +* **Failed** → `200` with the partial `Parameters` summary from the tasks that completed, plus a `status: failed` parameter; if no task completed, a `200` `OperationOutcome`. * **Unknown task** → `404`. ## Drill into the offending resources @@ -177,7 +169,7 @@ The `fullUrl` is **version-specific** (`/_history/`), so a vread resolv DELETE /fhir/$batch-validate/ ``` -Responds `202 Accepted`. Cancellation removes the run's **pending** chunks and marks it **cancelled** (a later poll reports `cancelled`). Aidbox does not interrupt a chunk already running, and keeps partial results. +Responds `202 Accepted`. Cancellation removes the run's **pending** tasks and marks it **cancelled** (a later poll reports `cancelled`). Aidbox does not interrupt a task already running, and keeps partial results. ## Profiles @@ -214,8 +206,8 @@ Aidbox stores results in an **aggregated, compact** form, so validating 100 GB o | Table | Holds | | --- | --- | | `issue` | one row per distinct error **pattern** (no per-resource rows) | -| `offender` | a tiny `(issue_id, resource_id, version_id)` row per offending resource: ids and versions only | -| `run_stat` | the `scanned` counter | +| `invalid_resource` | a tiny `(issue_id, resource_id, version_id)` row per offending resource: ids and versions only | +| `chunk_stat` | per-task progress: `validated`/`invalid` counts and completion, one row per task | A **pattern** is the aggregation key: `profile`, `resource_type`, index-normalized `path` (`identifier[2].system` → `identifier.system`), `code`, and `constraint_key`. All occurrences that share these collapse into one issue; the issue's **count is the number of offender rows** (distinct resources). Invariant issues also keep the validator's `human` description. @@ -229,4 +221,4 @@ The `invalid-resources` response is a `Parameters` resource rather than a `Bundl ## Terminology -Coded-binding and slice validation may call the configured terminology server. If it is unreachable or errors, Aidbox records the affected resource with a `terminology-unavailable` finding (which names the server) and the run still completes. Point `fhir.terminology.service-base-url` at a reachable server (a local or hybrid engine works best) for accurate, fast coded validation. +Coded-binding and slice validation may call the configured terminology server. Bindings that resolve **locally** (local code systems, or a hybrid engine with local content) validate offline and surface invalid codes as ordinary `terminology-binding-error` issues. But if validation needs the configured terminology server and it is **unreachable or errors**, the validator cannot complete and the **whole run fails**: a synchronous call returns `422` (`OperationOutcome`, "Batch validation failed…"), an asynchronous run reports `failed`. Point `fhir.terminology.service-base-url` at a reachable server (a local or hybrid engine works best) so coded validation is accurate and does not fail the run. diff --git a/docs/overview/faq.md b/docs/overview/faq.md index 32c384296..62e23e4a5 100644 --- a/docs/overview/faq.md +++ b/docs/overview/faq.md @@ -660,7 +660,7 @@ Learn more: For large datasets, enable async validation to process resources in the background without blocking API responses. -Learn more: [Asynchronous Resource Validation](../modules/profiling-and-validation/asynchronous-resource-validation.md) +Learn more: [Batch resource validation](../modules/profiling-and-validation/batch-resource-validation.md) ## Terminology & ValueSets diff --git a/docs/tutorials/artifact-registry-tutorials/custom-resources/migrate-to-fhirschema/README.md b/docs/tutorials/artifact-registry-tutorials/custom-resources/migrate-to-fhirschema/README.md index 659117eea..98d600dc3 100644 --- a/docs/tutorials/artifact-registry-tutorials/custom-resources/migrate-to-fhirschema/README.md +++ b/docs/tutorials/artifact-registry-tutorials/custom-resources/migrate-to-fhirschema/README.md @@ -194,7 +194,7 @@ If you have custom Aidbox SearchParameters defined via Zen or Entities, use this ### Step 6: Validate Resources and Resolve Issues -Since the FHIR Schema validation engine is more reliable and validates all the FHIR container cases, we need to validate existing data with FHIR Schema using [Aidbox Asynchronous Validation API](../../../../modules/profiling-and-validation/asynchronous-resource-validation.md). +Since the FHIR Schema validation engine is more reliable and validates all the FHIR container cases, we need to validate existing data with FHIR Schema using [Aidbox Batch Validation API](../../../../modules/profiling-and-validation/batch-resource-validation.md). ### Check Deprecated Capabilities diff --git a/redirects.yaml b/redirects.yaml index 0802a2df1..5a992cfaf 100644 --- a/redirects.yaml +++ b/redirects.yaml @@ -375,7 +375,8 @@ redirects: modules-1/observability/traces/how-to-use-tracing: modules/observability/traces/how-to-use-tracing.md modules-1/observability/traces/otel-traces-exporter-parameters: modules/observability/traces/otel-traces-exporter-parameters.md modules-1/other-modules/mcp: modules/other-modules/mcp.md - modules-1/profiling-and-validation/asynchronous-resource-validation: modules/profiling-and-validation/asynchronous-resource-validation.md + modules-1/profiling-and-validation/asynchronous-resource-validation: modules/profiling-and-validation/batch-resource-validation.md + modules/profiling-and-validation/asynchronous-resource-validation: modules/profiling-and-validation/batch-resource-validation.md modules-1/profiling-and-validation/fhir-schema-validator/setup-aidbox-with-fhir-schema-validation-engine: modules/profiling-and-validation/fhir-schema-validator/setup-aidbox-with-fhir-schema-validation-engine.md modules-1/profiling-and-validation/fhir-schema-validator/tutorials/how-to-create-fhir-npm-package: tutorials/artifact-registry-tutorials/how-to-create-fhir-npm-package.md modules-1/profiling-and-validation/skip-validation-of-references-in-resource-using-request-header: modules/profiling-and-validation/skip-validation-of-references-in-resource-using-request-header.md From c3fd7f685986e2e9f2eaf7cb4f895e85e5b59632 Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Mon, 6 Jul 2026 16:04:45 +0300 Subject: [PATCH 06/14] update --- .../batch-resource-validation.md | 22 +++++++++++-------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/modules/profiling-and-validation/batch-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md index e07ca5441..0482b7035 100644 --- a/docs/modules/profiling-and-validation/batch-resource-validation.md +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -11,6 +11,10 @@ description: >- Available in Aidbox starting from version **2607**. {% endhint %} +{% hint style="warning" %} +**Breaking change.** `$batch-validate` **replaces** the previous batch-validation API, which is **removed**. The `aidbox.validation/batch-validation`, `aidbox.validation/batch-validation-result`, `aidbox.validation/clear-batch-validation`, and `aidbox.validation/resources-batch-validation-task` RPCs no longer exist, and a run no longer produces `BatchValidationRun` / `BatchValidationError` resources — results now live in the aggregated `aidbox_batch_validation` schema (see [How results are stored](#how-results-are-stored)). Migrate to the `$batch-validate` operation described below. +{% endhint %} + ## Overview Batch validation checks resources **already in the database** against the active FHIR schemas and an optional set of profiles. Use it when you loaded data with validation off, or when you publish a new profile version and want to know **how many existing resources are non-compliant and why**. @@ -57,7 +61,7 @@ parameter: | `_since` **(required)** | `instant` | Only resources whose `meta.lastUpdated >= _since` (inclusive). Required so that a run declares a window instead of scanning a whole type (see [Filtering by date](#filtering-by-date)). | | `_until` | `instant` | Upper bound: `meta.lastUpdated < _until` (exclusive). | | `profile` (repeatable) | `canonical` | Validate every resource against these profile URLs (in addition to its base schema), conjunctively (see [Profiles](#profiles)). | -| `number-of-chunks` (default `12`, max `256`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task streams its slice, so heap stays bounded regardless. A value above `256` is rejected with `422`. | +| `number-of-chunks` (default `12`, max `256`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task pages through its slice, so heap stays bounded regardless. A value above `256` is rejected with `422`. | {% hint style="warning" %} The body must be a valid `Parameters` resource. Each parameter must use the **exact** `value[x]` type above (`profile` as `valueCanonical`, `_since`/`_until` as `valueInstant`, `number-of-chunks` as `valuePositiveInt`). Aidbox rejects an unknown parameter, a wrong value type, or a missing `_since` with `422` and an `OperationOutcome` that names the offending parameter. @@ -65,7 +69,7 @@ The body must be a valid `Parameters` resource. Each parameter must use the **ex ## Synchronous response -A `Parameters` resource holds the `task-id`, the headline counts, a link to the offending resources, and one `issue` per distinct error pattern (each with its own filtered drill-down link). +A `Parameters` resource holds the `task-id`, the headline counts, a link to the offending resources, and one `issue` per distinct error (each with its own filtered drill-down link). ```yaml status: 200 @@ -81,7 +85,6 @@ parameter: part: - {name: id, valueString: '5b4b07e4…'} # issue-id (for drill-down) - {name: invalid-resources, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources?_issue=5b4b07e4…'} - - {name: severity, valueCode: error} - {name: code, valueCode: invalid-slice-cardinality} - {name: expression, valueString: category} # the element - {name: profile, valueString: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'} @@ -90,9 +93,10 @@ parameter: - {name: diagnostics, valueString: '…human-readable message…'} ``` -* **`count`** is the number of **distinct offending resources** for the pattern, derived from the offender index. +* **`count`** is the number of **distinct offending resources** for the issue, derived from the offender index. * For **invariant** issues, the `constraint` part carries the constraint key and `diagnostics` carries the validator's human-readable description. * Each `issue` carries its own `invalid-resources` link, pre-filtered to that issue. +* The summary lists at most **10,000** distinct issues, worst first. If a run produces more, it lists the worst 10,000 and adds `issues-total` (the true count) and `issues-truncated: true`, so a truncated list is not mistaken for the whole. * On failure the response is an `OperationOutcome`. ## Asynchronous response @@ -118,7 +122,7 @@ GET /fhir/$batch-validate/ ## Drill into the offending resources -The summary tells you which patterns fail and how many resources hit each. To get the **offending resources**, each linked to the version that was validated and carrying its full `OperationOutcome`, call `invalid-resources`: +The summary tells you which issues occur and how many resources hit each. To get the **offending resources**, each linked to the version that was validated and carrying its full `OperationOutcome`, call `invalid-resources`: ``` GET /fhir/$batch-validate//invalid-resources @@ -128,7 +132,7 @@ GET /fhir/$batch-validate//invalid-resources | Query parameter | Meaning | | --- | --- | | `_issue` (repeatable) | Restrict to offenders of these issue(s). Omit for **all** offending resources. | -| `_count` / `_page` | Page size (default `50`) and 1-based page number. | +| `_count` / `_page` | Page size (default `50`, max `1000`) and 1-based page number. | | `_fullurl-only` (default `false`) | When `true`, return only the `fullUrl` of each offender (omit the body and outcome). | The response is a **`Parameters` report** rather than a Bundle (see [why](#why-a-parameters-report)): a `total`, flat paging links, and one repeated `resource` parameter per offending resource. @@ -150,7 +154,7 @@ parameter: resource: resourceType: OperationOutcome issue: - - {severity: error, code: structure, expression: [Observation.category], + - {severity: fatal, code: invalid, expression: [Observation.category], diagnostics: '…', details: {coding: [{code: invalid-slice-cardinality}]}} ``` @@ -205,11 +209,11 @@ Aidbox stores results in an **aggregated, compact** form, so validating 100 GB o | Table | Holds | | --- | --- | -| `issue` | one row per distinct error **pattern** (no per-resource rows) | +| `issue` | one row per distinct error (no per-resource rows) | | `invalid_resource` | a tiny `(issue_id, resource_id, version_id)` row per offending resource: ids and versions only | | `chunk_stat` | per-task progress: `validated`/`invalid` counts and completion, one row per task | -A **pattern** is the aggregation key: `profile`, `resource_type`, index-normalized `path` (`identifier[2].system` → `identifier.system`), `code`, and `constraint_key`. All occurrences that share these collapse into one issue; the issue's **count is the number of offender rows** (distinct resources). Invariant issues also keep the validator's `human` description. +The aggregation key is `profile`, `resource_type`, index-normalized `path` (`identifier[2].system` → `identifier.system`), `code`, and `constraint_key`. All occurrences that share these collapse into one issue; the issue's **count is the number of offender rows** (distinct resources). Invariant issues also keep the validator's `human` description. Aidbox does **not** store the invalid resource bodies or their `OperationOutcome`s. The drill-down re-reads the body from history at the validated version and reconstructs the `OperationOutcome` from the stored machine fields (`code`, `expression`, `constraint`, `human`). From 48dd596cc22bdee87ff9e09014c51d5194835e58 Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Mon, 6 Jul 2026 16:10:11 +0300 Subject: [PATCH 07/14] more fixes --- .../batch-resource-validation.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/modules/profiling-and-validation/batch-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md index 0482b7035..d994a2d12 100644 --- a/docs/modules/profiling-and-validation/batch-resource-validation.md +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -80,11 +80,11 @@ parameter: - {name: validated, valueUnsignedInt: 1804646} # resources validated - {name: valid, valueUnsignedInt: 1317494} # resources with no issues - {name: invalid, valueUnsignedInt: 487152} # distinct resources with ≥1 issue - - {name: invalid-resources, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources'} + - {name: invalid-resources, valueUrl: '/fhir/$batch-validate//invalid-resources'} - name: issue part: - {name: id, valueString: '5b4b07e4…'} # issue-id (for drill-down) - - {name: invalid-resources, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources?_issue=5b4b07e4…'} + - {name: invalid-resources, valueUrl: '/fhir/$batch-validate//invalid-resources?_issue=5b4b07e4…'} - {name: code, valueCode: invalid-slice-cardinality} - {name: expression, valueString: category} # the element - {name: profile, valueString: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'} @@ -103,7 +103,7 @@ parameter: ``` status: 202 Accepted -Content-Location: http://localhost:8765/fhir/$batch-validate/ +Content-Location: /fhir/$batch-validate/ ``` The endpoints below are **system-level**, keyed by `task-id` alone (no resource type in the path). @@ -135,19 +135,19 @@ GET /fhir/$batch-validate//invalid-resources | `_count` / `_page` | Page size (default `50`, max `1000`) and 1-based page number. | | `_fullurl-only` (default `false`) | When `true`, return only the `fullUrl` of each offender (omit the body and outcome). | -The response is a **`Parameters` report** rather than a Bundle (see [why](#why-a-parameters-report)): a `total`, flat paging links, and one repeated `resource` parameter per offending resource. +The response is a **`Parameters` report** rather than a Bundle (see [Response format](#response-format)): a `total`, flat paging links, and one repeated `resource` parameter per offending resource. ```yaml resourceType: Parameters parameter: - {name: total, valueUnsignedInt: 1114} # paging: flat links. first/previous appear past page 1; next/last before the last page - - {name: self, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources?_count=50&_page=1'} - - {name: next, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources?_count=50&_page=2'} - - {name: last, valueUrl: 'http://localhost:8765/fhir/$batch-validate//invalid-resources?_count=50&_page=23'} + - {name: self, valueUrl: '/fhir/$batch-validate//invalid-resources?_count=50&_page=1'} + - {name: next, valueUrl: '/fhir/$batch-validate//invalid-resources?_count=50&_page=2'} + - {name: last, valueUrl: '/fhir/$batch-validate//invalid-resources?_count=50&_page=23'} - name: resource part: - - {name: fullUrl, valueUrl: 'http://localhost:8765/Observation//_history/'} + - {name: fullUrl, valueUrl: '/Observation//_history/'} - name: resource # omitted when _fullurl-only=true resource: {resourceType: Observation, meta: {versionId: ''}, …} - name: outcome # omitted when _fullurl-only=true @@ -219,9 +219,9 @@ Aidbox does **not** store the invalid resource bodies or their `OperationOutcome Both synchronous and asynchronous runs persist these tables under the run's `task-id`, so you poll and drill into either until you cancel it. -## Why a Parameters report +## Response format -The `invalid-resources` response is a `Parameters` resource rather than a `Bundle`. A `searchset`/`collection` Bundle cannot carry a `total`, **version-specific** links, a per-offender `OperationOutcome`, and the invalid resource bodies while staying FHIR-valid: Bundle invariants forbid a version-specific `fullUrl`, restrict `total` and `entry.response` by Bundle type, and the FHIR validator validates embedded resources in full. A `Parameters` report avoids those constraints, keeps the versioned drill-down links, and embeds each `OperationOutcome` without trouble. The embedded resource bodies are the report's payload: the invalid data under review. +The `invalid-resources` response is a `Parameters` resource rather than a `Bundle`. A `searchset` or `collection` Bundle cannot carry a `total`, version-specific links, a per-offender `OperationOutcome`, and the invalid resource bodies together while remaining FHIR-valid: Bundle invariants prohibit a version-specific `fullUrl`, permit `total` and `entry.response` only on certain Bundle types, and require each embedded resource to be valid in its own right — which the intentionally invalid bodies are not. A `Parameters` resource is subject to none of these constraints: it preserves the version-specific drill-down links and embeds each `OperationOutcome` beside the resource it describes. The invalid resource bodies are the report's content — the data under review. ## Terminology From afb264e4a41e52fb0149af3c4f57c9dfdb14ae67 Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Tue, 7 Jul 2026 16:31:24 +0300 Subject: [PATCH 08/14] fix --- .../profiling-and-validation/batch-resource-validation.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/modules/profiling-and-validation/batch-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md index d994a2d12..12598ce04 100644 --- a/docs/modules/profiling-and-validation/batch-resource-validation.md +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -36,6 +36,10 @@ Synchronous validation blocks the request until it finishes. That suits a type s Each task scans the window once for its hash slice, so total scan work grows with the task count. More chunks means more parallelism (the async path spreads them across nodes) but more scans; fewer chunks means fewer scans but less parallelism. The default of `12` balances the two — raise `number-of-chunks` to parallelize a large type further. {% endhint %} +{% hint style="warning" %} +**PostgreSQL connections.** Running tasks each use their own PostgreSQL connections, so a run with many parallel tasks (a high `number-of-chunks` together with a high `scheduler-executors`) raises connection use. Make sure PostgreSQL `max_connections` (and any external pooler) has the headroom, or tasks will fail to acquire a connection. +{% endhint %} + ## Start a validation `POST /fhir//$batch-validate` with a FHIR `Parameters` body: @@ -61,7 +65,7 @@ parameter: | `_since` **(required)** | `instant` | Only resources whose `meta.lastUpdated >= _since` (inclusive). Required so that a run declares a window instead of scanning a whole type (see [Filtering by date](#filtering-by-date)). | | `_until` | `instant` | Upper bound: `meta.lastUpdated < _until` (exclusive). | | `profile` (repeatable) | `canonical` | Validate every resource against these profile URLs (in addition to its base schema), conjunctively (see [Profiles](#profiles)). | -| `number-of-chunks` (default `12`, max `256`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task pages through its slice, so heap stays bounded regardless. A value above `256` is rejected with `422`. | +| `number-of-chunks` (default `12`, max `256`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task streams its slice, so heap stays bounded regardless. A value above `256` is rejected with `422`. | {% hint style="warning" %} The body must be a valid `Parameters` resource. Each parameter must use the **exact** `value[x]` type above (`profile` as `valueCanonical`, `_since`/`_until` as `valueInstant`, `number-of-chunks` as `valuePositiveInt`). Aidbox rejects an unknown parameter, a wrong value type, or a missing `_since` with `422` and an `OperationOutcome` that names the offending parameter. From 3655a1ccc7361898cf06df5667665a7ea2e7d7c3 Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Fri, 10 Jul 2026 11:10:12 +0300 Subject: [PATCH 09/14] add validator options --- .../batch-resource-validation.md | 25 ++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/docs/modules/profiling-and-validation/batch-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md index 12598ce04..e2c939255 100644 --- a/docs/modules/profiling-and-validation/batch-resource-validation.md +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -56,8 +56,10 @@ parameter: - {name: _until, valueInstant: '2025-06-09T00:00:00Z'} # validate against these profiles (conjunctive, see Profiles) - {name: profile, valueCanonical: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'} - # tuning (optional): number of parallel tasks (default 12, max 256) + # tuning (optional): number of parallel tasks (default 12, max 10000) - {name: number-of-chunks, valuePositiveInt: 24} + # validator options (optional): override the box's validation settings for this run + - {name: disable-terminology-validation, valueBoolean: true} ``` | Parameter | Type | Meaning | @@ -65,12 +67,29 @@ parameter: | `_since` **(required)** | `instant` | Only resources whose `meta.lastUpdated >= _since` (inclusive). Required so that a run declares a window instead of scanning a whole type (see [Filtering by date](#filtering-by-date)). | | `_until` | `instant` | Upper bound: `meta.lastUpdated < _until` (exclusive). | | `profile` (repeatable) | `canonical` | Validate every resource against these profile URLs (in addition to its base schema), conjunctively (see [Profiles](#profiles)). | -| `number-of-chunks` (default `12`, max `256`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task streams its slice, so heap stays bounded regardless. A value above `256` is rejected with `422`. | +| `number-of-chunks` (default `12`, max `10000`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task streams its slice, so heap stays bounded regardless. A value above `10000` is rejected with `422`. | +| `disable-terminology-validation` | `boolean` | Skip coded-binding / terminology checks. | +| `disable-primitive-validation` | `boolean` | Skip primitive type & format checks. | +| `disable-slicing-validation` | `boolean` | Skip slice validation. | +| `disable-constraint-validation` | `boolean` | Blanket switch for FHIRPath invariants: `true` skips **all** of them; `false` checks **all** (see [Validator options](#validator-options)). | +| `disable-constraint` (repeatable) | `string` | Skip specific invariants by key (e.g. `us-core-8`). | +| `strict-profile-resolution` | `boolean` | Treat an unresolved `profile` / `meta.profile` canonical as an error instead of silently skipping it. | +| `strict-extension-resolution` | `boolean` | Treat an unresolved extension as an error. | {% hint style="warning" %} -The body must be a valid `Parameters` resource. Each parameter must use the **exact** `value[x]` type above (`profile` as `valueCanonical`, `_since`/`_until` as `valueInstant`, `number-of-chunks` as `valuePositiveInt`). Aidbox rejects an unknown parameter, a wrong value type, or a missing `_since` with `422` and an `OperationOutcome` that names the offending parameter. +The body must be a valid `Parameters` resource. Each parameter must use the **exact** `value[x]` type above (`profile` as `valueCanonical`, `_since`/`_until` as `valueInstant`, `number-of-chunks` as `valuePositiveInt`, the `disable-*`/`strict-*` flags as `valueBoolean`, `disable-constraint` as `valueString`). Aidbox rejects an unknown parameter, a wrong value type, or a missing `_since` with `422` and an `OperationOutcome` that names the offending parameter. {% endhint %} +## Validator options + +The `disable-*` and `strict-*` parameters tune what the validator checks, **for this run only**. Each is three-state: **omit it to keep the box's configured setting**, or pass it to override that setting (`true`/`false`). Use them to trade completeness for speed on a huge type (e.g. skip the terminology and slicing passes for a structural-only sweep), or to tighten a run beyond the box defaults (e.g. `strict-profile-resolution` to surface resources whose declared profiles don't resolve — which otherwise read as compliant). + +Constraints (FHIRPath invariants) have two controls that compose: + +* `disable-constraint-validation` is the blanket switch — `true` skips **every** invariant, `false` checks **every** invariant (including any the box normally mutes). +* `disable-constraint` names specific invariants to skip (repeat it per key). +* When both are given, the blanket wins: `true` skips everything (the list is moot); with `false`, only the listed keys are skipped and all others are checked. With the blanket omitted, the listed keys are skipped **on top of** the box's defaults. + ## Synchronous response A `Parameters` resource holds the `task-id`, the headline counts, a link to the offending resources, and one `issue` per distinct error (each with its own filtered drill-down link). From edcd9f8d897f528e08b8f502af9bbe871053611b Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Fri, 10 Jul 2026 14:42:25 +0300 Subject: [PATCH 10/14] no bound to max tasks --- .../profiling-and-validation/batch-resource-validation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/modules/profiling-and-validation/batch-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md index e2c939255..2deea562c 100644 --- a/docs/modules/profiling-and-validation/batch-resource-validation.md +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -56,7 +56,7 @@ parameter: - {name: _until, valueInstant: '2025-06-09T00:00:00Z'} # validate against these profiles (conjunctive, see Profiles) - {name: profile, valueCanonical: 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-lab'} - # tuning (optional): number of parallel tasks (default 12, max 10000) + # tuning (optional): number of parallel tasks (default 12) - {name: number-of-chunks, valuePositiveInt: 24} # validator options (optional): override the box's validation settings for this run - {name: disable-terminology-validation, valueBoolean: true} @@ -67,7 +67,7 @@ parameter: | `_since` **(required)** | `instant` | Only resources whose `meta.lastUpdated >= _since` (inclusive). Required so that a run declares a window instead of scanning a whole type (see [Filtering by date](#filtering-by-date)). | | `_until` | `instant` | Upper bound: `meta.lastUpdated < _until` (exclusive). | | `profile` (repeatable) | `canonical` | Validate every resource against these profile URLs (in addition to its base schema), conjunctively (see [Profiles](#profiles)). | -| `number-of-chunks` (default `12`, max `10000`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task streams its slice, so heap stays bounded regardless. A value above `10000` is rejected with `422`. | +| `number-of-chunks` (default `12`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task streams its slice, so heap stays bounded regardless. No fixed maximum: a sync run streams the tasks through a bounded thread pool (heap stays proportional to the executor count, not the task count), and an async run writes one scheduler row per task — so a very large value costs task rows and scans, not memory. | | `disable-terminology-validation` | `boolean` | Skip coded-binding / terminology checks. | | `disable-primitive-validation` | `boolean` | Skip primitive type & format checks. | | `disable-slicing-validation` | `boolean` | Skip slice validation. | From 5b91638537530608c9cf3a507999f7d696b8dee3 Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Fri, 10 Jul 2026 14:43:33 +0300 Subject: [PATCH 11/14] fix --- .../profiling-and-validation/batch-resource-validation.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/modules/profiling-and-validation/batch-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md index 2deea562c..5f1d58661 100644 --- a/docs/modules/profiling-and-validation/batch-resource-validation.md +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -103,6 +103,7 @@ parameter: - {name: validated, valueUnsignedInt: 1804646} # resources validated - {name: valid, valueUnsignedInt: 1317494} # resources with no issues - {name: invalid, valueUnsignedInt: 487152} # distinct resources with ≥1 issue + - {name: bytes, valueDecimal: 5242880000} # total bytes of resource JSON processed - {name: invalid-resources, valueUrl: '/fhir/$batch-validate//invalid-resources'} - name: issue part: @@ -116,6 +117,7 @@ parameter: - {name: diagnostics, valueString: '…human-readable message…'} ``` +* **`bytes`** is the total size of the resource JSON the run processed — a `decimal` because the total overflows `unsignedInt` at scale. * **`count`** is the number of **distinct offending resources** for the issue, derived from the offender index. * For **invariant** issues, the `constraint` part carries the constraint key and `diagnostics` carries the validator's human-readable description. * Each `issue` carries its own `invalid-resources` link, pre-filtered to that issue. @@ -234,7 +236,7 @@ Aidbox stores results in an **aggregated, compact** form, so validating 100 GB o | --- | --- | | `issue` | one row per distinct error (no per-resource rows) | | `invalid_resource` | a tiny `(issue_id, resource_id, version_id)` row per offending resource: ids and versions only | -| `chunk_stat` | per-task progress: `validated`/`invalid` counts and completion, one row per task | +| `chunk_stat` | one row per task, written once when the task finishes: its `validated`/`invalid`/`bytes` tallies | The aggregation key is `profile`, `resource_type`, index-normalized `path` (`identifier[2].system` → `identifier.system`), `code`, and `constraint_key`. All occurrences that share these collapse into one issue; the issue's **count is the number of offender rows** (distinct resources). Invariant issues also keep the validator's `human` description. From 98f5d42c56f081591283fae7bbcaf8981bf68ac6 Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Mon, 13 Jul 2026 12:34:46 +0300 Subject: [PATCH 12/14] fix --- .../batch-resource-validation.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/modules/profiling-and-validation/batch-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md index 5f1d58661..98a541f15 100644 --- a/docs/modules/profiling-and-validation/batch-resource-validation.md +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -119,7 +119,8 @@ parameter: * **`bytes`** is the total size of the resource JSON the run processed — a `decimal` because the total overflows `unsignedInt` at scale. * **`count`** is the number of **distinct offending resources** for the issue, derived from the offender index. -* For **invariant** issues, the `constraint` part carries the constraint key and `diagnostics` carries the validator's human-readable description. +* For **invariant** issues, the `constraint` part carries the constraint key. The `diagnostics` part is a **generated** summary keyed off the issue `code` (for an invariant, `: constraint is not satisfied`) — it is not the validator's original message text, which is not stored (see [How results are stored](#how-results-are-stored)). +* Other issue kinds add a type-specific part: `slice` (the slice name) for slice issues, `binding` (the value-set URL) for terminology-binding issues, and `unknown-profile` (the unresolved canonical) for an unresolved `profile` / `meta.profile`. A part with no value is omitted. * Each `issue` carries its own `invalid-resources` link, pre-filtered to that issue. * The summary lists at most **10,000** distinct issues, worst first. If a run produces more, it lists the worst 10,000 and adds `issues-total` (the true count) and `issues-truncated: true`, so a truncated list is not mistaken for the whole. * On failure the response is an `OperationOutcome`. @@ -141,7 +142,7 @@ GET /fhir/$batch-validate/ * **In progress** → `202 Accepted` with an `X-Progress` header (percent of tasks completed, e.g. `45%`). * **Complete** → `200` with the same `Parameters` summary as the synchronous response. -* **Cancelled** → `200` `OperationOutcome` (`cancelled`). +* **Cancelled** → `200`, an informational `OperationOutcome` (issue code `informational`). * **Failed** → `200` with the partial `Parameters` summary from the tasks that completed, plus a `status: failed` parameter; if no task completed, a `200` `OperationOutcome`. * **Unknown task** → `404`. @@ -238,9 +239,9 @@ Aidbox stores results in an **aggregated, compact** form, so validating 100 GB o | `invalid_resource` | a tiny `(issue_id, resource_id, version_id)` row per offending resource: ids and versions only | | `chunk_stat` | one row per task, written once when the task finishes: its `validated`/`invalid`/`bytes` tallies | -The aggregation key is `profile`, `resource_type`, index-normalized `path` (`identifier[2].system` → `identifier.system`), `code`, and `constraint_key`. All occurrences that share these collapse into one issue; the issue's **count is the number of offender rows** (distinct resources). Invariant issues also keep the validator's `human` description. +The aggregation key is `profile`, `resource_type`, index-normalized `path` (`identifier[2].system` → `identifier.system`), `code`, `constraint_key`, and — where they apply — the slice name, binding value set, and unresolved profile canonical. All occurrences that share these collapse into one issue; the issue's **count is the number of offender rows** (distinct resources). -Aidbox does **not** store the invalid resource bodies or their `OperationOutcome`s. The drill-down re-reads the body from history at the validated version and reconstructs the `OperationOutcome` from the stored machine fields (`code`, `expression`, `constraint`, `human`). +Aidbox does **not** store the invalid resource bodies, their `OperationOutcome`s, or the validator's original message text. The drill-down re-reads the body from history at the validated version and reconstructs the `OperationOutcome` from the stored machine fields (`code`, `expression`, `constraint`, and the type-specific parts), synthesizing the `diagnostics` message from the `code`. Both synchronous and asynchronous runs persist these tables under the run's `task-id`, so you poll and drill into either until you cancel it. From da53a634fd1df0bdc1a4682db6ff58c413c80bef Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Mon, 13 Jul 2026 14:52:50 +0300 Subject: [PATCH 13/14] apply fixes from claude.md --- .../batch-resource-validation.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/modules/profiling-and-validation/batch-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md index 98a541f15..b34f5836b 100644 --- a/docs/modules/profiling-and-validation/batch-resource-validation.md +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -12,7 +12,7 @@ Available in Aidbox starting from version **2607**. {% endhint %} {% hint style="warning" %} -**Breaking change.** `$batch-validate` **replaces** the previous batch-validation API, which is **removed**. The `aidbox.validation/batch-validation`, `aidbox.validation/batch-validation-result`, `aidbox.validation/clear-batch-validation`, and `aidbox.validation/resources-batch-validation-task` RPCs no longer exist, and a run no longer produces `BatchValidationRun` / `BatchValidationError` resources — results now live in the aggregated `aidbox_batch_validation` schema (see [How results are stored](#how-results-are-stored)). Migrate to the `$batch-validate` operation described below. +**Breaking change.** `$batch-validate` **replaces** the previous batch-validation API, which is **removed**. The `aidbox.validation/batch-validation`, `aidbox.validation/batch-validation-result`, `aidbox.validation/clear-batch-validation`, and `aidbox.validation/resources-batch-validation-task` RPCs no longer exist, and a run no longer produces `BatchValidationRun` / `BatchValidationError` resources. Results now live in the aggregated `aidbox_batch_validation` schema (see [How results are stored](#how-results-are-stored)). Migrate to the `$batch-validate` operation described below. {% endhint %} ## Overview @@ -33,7 +33,7 @@ Both paths produce the same result under a **`task-id`** and persist it the same {% hint style="info" %} Synchronous validation blocks the request until it finishes. That suits a type scoped by a narrow `_since`/`_until` window; for a large type, use `Prefer: respond-async`. The synchronous path runs the N tasks on a local pool sized by `scheduler-executors` (`BOX_SCHEDULER_EXECUTORS`, default `4`); the async path schedules them on the task scheduler, which spreads them across nodes. -Each task scans the window once for its hash slice, so total scan work grows with the task count. More chunks means more parallelism (the async path spreads them across nodes) but more scans; fewer chunks means fewer scans but less parallelism. The default of `12` balances the two — raise `number-of-chunks` to parallelize a large type further. +Each task scans the window once for its hash slice, so total scan work grows with the task count. More chunks means more parallelism (the async path spreads them across nodes) but more scans; fewer chunks means fewer scans but less parallelism. The default of `12` balances the two. Raise `number-of-chunks` to parallelize a large type further. {% endhint %} {% hint style="warning" %} @@ -67,13 +67,13 @@ parameter: | `_since` **(required)** | `instant` | Only resources whose `meta.lastUpdated >= _since` (inclusive). Required so that a run declares a window instead of scanning a whole type (see [Filtering by date](#filtering-by-date)). | | `_until` | `instant` | Upper bound: `meta.lastUpdated < _until` (exclusive). | | `profile` (repeatable) | `canonical` | Validate every resource against these profile URLs (in addition to its base schema), conjunctively (see [Profiles](#profiles)). | -| `number-of-chunks` (default `12`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task streams its slice, so heap stays bounded regardless. No fixed maximum: a sync run streams the tasks through a bounded thread pool (heap stays proportional to the executor count, not the task count), and an async run writes one scheduler row per task — so a very large value costs task rows and scans, not memory. | +| `number-of-chunks` (default `12`) | `positiveInt` | Number of hash-partitioned tasks the run is split into. More parallelizes a large type (across nodes when async) at the cost of more scans; each task streams its slice, so heap stays bounded regardless. No fixed maximum: a sync run streams the tasks through a bounded thread pool (heap stays proportional to the executor count, not the task count), and an async run writes one scheduler row per task, so a very large value costs task rows and scans, not memory. | | `disable-terminology-validation` | `boolean` | Skip coded-binding / terminology checks. | | `disable-primitive-validation` | `boolean` | Skip primitive type & format checks. | | `disable-slicing-validation` | `boolean` | Skip slice validation. | | `disable-constraint-validation` | `boolean` | Blanket switch for FHIRPath invariants: `true` skips **all** of them; `false` checks **all** (see [Validator options](#validator-options)). | | `disable-constraint` (repeatable) | `string` | Skip specific invariants by key (e.g. `us-core-8`). | -| `strict-profile-resolution` | `boolean` | Treat an unresolved `profile` / `meta.profile` canonical as an error instead of silently skipping it. | +| `strict-profile-resolution` | `boolean` | Treat an unresolved `profile` / `meta.profile` canonical as an error instead of skipping it. | | `strict-extension-resolution` | `boolean` | Treat an unresolved extension as an error. | {% hint style="warning" %} @@ -82,11 +82,11 @@ The body must be a valid `Parameters` resource. Each parameter must use the **ex ## Validator options -The `disable-*` and `strict-*` parameters tune what the validator checks, **for this run only**. Each is three-state: **omit it to keep the box's configured setting**, or pass it to override that setting (`true`/`false`). Use them to trade completeness for speed on a huge type (e.g. skip the terminology and slicing passes for a structural-only sweep), or to tighten a run beyond the box defaults (e.g. `strict-profile-resolution` to surface resources whose declared profiles don't resolve — which otherwise read as compliant). +The `disable-*` and `strict-*` parameters tune what the validator checks, **for this run only**. Each is three-state: **omit it to keep the box's configured setting** (see [FHIR Schema Validator](fhir-schema-validator/README.md)), or pass it to override that setting (`true`/`false`). Use them to trade completeness for speed on a huge type (e.g. skip the terminology and slicing passes for a structural-only sweep), or to tighten a run beyond the box defaults (e.g. `strict-profile-resolution` to surface resources whose declared profiles don't resolve, which otherwise read as compliant). Constraints (FHIRPath invariants) have two controls that compose: -* `disable-constraint-validation` is the blanket switch — `true` skips **every** invariant, `false` checks **every** invariant (including any the box normally mutes). +* `disable-constraint-validation` is the blanket switch: `true` skips **every** invariant, `false` checks **every** invariant (including any the box normally mutes). * `disable-constraint` names specific invariants to skip (repeat it per key). * When both are given, the blanket wins: `true` skips everything (the list is moot); with `false`, only the listed keys are skipped and all others are checked. With the blanket omitted, the listed keys are skipped **on top of** the box's defaults. @@ -117,9 +117,9 @@ parameter: - {name: diagnostics, valueString: '…human-readable message…'} ``` -* **`bytes`** is the total size of the resource JSON the run processed — a `decimal` because the total overflows `unsignedInt` at scale. +* **`bytes`** is the total size of the resource JSON the run processed, a `decimal` because the total overflows `unsignedInt` at scale. * **`count`** is the number of **distinct offending resources** for the issue, derived from the offender index. -* For **invariant** issues, the `constraint` part carries the constraint key. The `diagnostics` part is a **generated** summary keyed off the issue `code` (for an invariant, `: constraint is not satisfied`) — it is not the validator's original message text, which is not stored (see [How results are stored](#how-results-are-stored)). +* For **invariant** issues, the `constraint` part carries the constraint key. The `diagnostics` part is a **generated** summary keyed off the issue `code` (for an invariant, `: constraint is not satisfied`). It is not the validator's original message text, which is not stored (see [How results are stored](#how-results-are-stored)). * Other issue kinds add a type-specific part: `slice` (the slice name) for slice issues, `binding` (the value-set URL) for terminology-binding issues, and `unknown-profile` (the unresolved canonical) for an unresolved `profile` / `meta.profile`. A part with no value is omitted. * Each `issue` carries its own `invalid-resources` link, pre-filtered to that issue. * The summary lists at most **10,000** distinct issues, worst first. If a run produces more, it lists the worst 10,000 and adds `issues-total` (the true count) and `issues-truncated: true`, so a truncated list is not mistaken for the whole. @@ -127,7 +127,7 @@ parameter: ## Asynchronous response -``` +```http status: 202 Accepted Content-Location: /fhir/$batch-validate/ ``` @@ -136,7 +136,7 @@ The endpoints below are **system-level**, keyed by `task-id` alone (no resource Poll the `Content-Location`: -```yaml +```http GET /fhir/$batch-validate/ ``` @@ -150,7 +150,7 @@ GET /fhir/$batch-validate/ The summary tells you which issues occur and how many resources hit each. To get the **offending resources**, each linked to the version that was validated and carrying its full `OperationOutcome`, call `invalid-resources`: -``` +```http GET /fhir/$batch-validate//invalid-resources ?_issue=&_issue=&_count=50&_page=1&_fullurl-only=false ``` @@ -195,7 +195,7 @@ The `fullUrl` is **version-specific** (`/_history/`), so a vread resolv ## Cancel -``` +```http DELETE /fhir/$batch-validate/ ``` @@ -239,7 +239,7 @@ Aidbox stores results in an **aggregated, compact** form, so validating 100 GB o | `invalid_resource` | a tiny `(issue_id, resource_id, version_id)` row per offending resource: ids and versions only | | `chunk_stat` | one row per task, written once when the task finishes: its `validated`/`invalid`/`bytes` tallies | -The aggregation key is `profile`, `resource_type`, index-normalized `path` (`identifier[2].system` → `identifier.system`), `code`, `constraint_key`, and — where they apply — the slice name, binding value set, and unresolved profile canonical. All occurrences that share these collapse into one issue; the issue's **count is the number of offender rows** (distinct resources). +The aggregation key is `profile`, `resource_type`, index-normalized `path` (`identifier[2].system` → `identifier.system`), `code`, `constraint_key`, and (where they apply) the slice name, binding value set, and unresolved profile canonical. All occurrences that share these collapse into one issue; the issue's **count is the number of offender rows** (distinct resources). Aidbox does **not** store the invalid resource bodies, their `OperationOutcome`s, or the validator's original message text. The drill-down re-reads the body from history at the validated version and reconstructs the `OperationOutcome` from the stored machine fields (`code`, `expression`, `constraint`, and the type-specific parts), synthesizing the `diagnostics` message from the `code`. @@ -247,7 +247,7 @@ Both synchronous and asynchronous runs persist these tables under the run's `tas ## Response format -The `invalid-resources` response is a `Parameters` resource rather than a `Bundle`. A `searchset` or `collection` Bundle cannot carry a `total`, version-specific links, a per-offender `OperationOutcome`, and the invalid resource bodies together while remaining FHIR-valid: Bundle invariants prohibit a version-specific `fullUrl`, permit `total` and `entry.response` only on certain Bundle types, and require each embedded resource to be valid in its own right — which the intentionally invalid bodies are not. A `Parameters` resource is subject to none of these constraints: it preserves the version-specific drill-down links and embeds each `OperationOutcome` beside the resource it describes. The invalid resource bodies are the report's content — the data under review. +The `invalid-resources` response is a `Parameters` resource rather than a `Bundle`. A `searchset` or `collection` Bundle cannot carry a `total`, version-specific links, a per-offender `OperationOutcome`, and the invalid resource bodies together while remaining FHIR-valid: Bundle invariants prohibit a version-specific `fullUrl`, permit `total` and `entry.response` only on certain Bundle types, and require each embedded resource to be valid in its own right, which the invalid bodies are not. A `Parameters` resource is subject to none of these constraints: it preserves the version-specific drill-down links and embeds each `OperationOutcome` beside the resource it describes. The invalid resource bodies are the report's content: the data under review. ## Terminology From 9e690bb93eac25217417dee1fa02c910b6cabe3f Mon Sep 17 00:00:00 2001 From: Andrey Listopadov Date: Mon, 13 Jul 2026 15:19:58 +0300 Subject: [PATCH 14/14] add note on index --- .../batch-resource-validation.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/modules/profiling-and-validation/batch-resource-validation.md b/docs/modules/profiling-and-validation/batch-resource-validation.md index b34f5836b..803a0dad9 100644 --- a/docs/modules/profiling-and-validation/batch-resource-validation.md +++ b/docs/modules/profiling-and-validation/batch-resource-validation.md @@ -90,6 +90,36 @@ Constraints (FHIRPath invariants) have two controls that compose: * `disable-constraint` names specific invariants to skip (repeat it per key). * When both are given, the blanket wins: `true` skips everything (the list is moot); with `false`, only the listed keys are skipped and all others are checked. With the blanket omitted, the listed keys are skipped **on top of** the box's defaults. +## Indexing the hash partition + +Each task validates its `mod(abs(hashtextextended(id, 0)), N)` slice of the type, where `N` is `number-of-chunks`. `N` stays fixed for a run, so the partition predicate is a constant expression that a PostgreSQL expression index can cover. Without such an index, every task scans the whole table to find its slice, so a run with `N` tasks costs `N` full scans (the scan cost the `number-of-chunks` note above describes). With a matching index, each task reads only its slice through the index. + +The gain grows as the slice shrinks, so an index matters most for a large type validated with a high `number-of-chunks`. Build it on the resource's storage table (named after the lowercased resource type) with the **same modulus** as the `number-of-chunks` the run passes: + +```sql +-- Observation, validated in 10000 chunks +CREATE INDEX CONCURRENTLY observation_batch_validate_10000 + ON observation (mod(abs(hashtextextended(id, 0)), 10000)); +``` + +Then run with the matching `number-of-chunks`: + +```yaml +POST /fhir/Observation/$batch-validate +resourceType: Parameters +parameter: + - {name: _since, valueInstant: '1970-01-01T00:00:00Z'} + - {name: number-of-chunks, valuePositiveInt: 10000} +``` + +{% hint style="warning" %} +The index modulus must equal `number-of-chunks`. A different value is a different expression, so PostgreSQL skips the index and the run falls back to per-task scans. Check the plan with `EXPLAIN` on one task's query before relying on the index. +{% endhint %} + +{% hint style="info" %} +`CREATE INDEX CONCURRENTLY` builds the index without blocking writes to the table. The index costs disk and slows writes, so drop it after a one-off sweep: `DROP INDEX observation_batch_validate_10000`. +{% endhint %} + ## Synchronous response A `Parameters` resource holds the `task-id`, the headline counts, a link to the offending resources, and one `issue` per distinct error (each with its own filtered drill-down link).