From 041618511f3fd15cf90339827e1fbac6b3e0a0c0 Mon Sep 17 00:00:00 2001 From: Tom Brooke Date: Thu, 24 Sep 2026 11:45:24 -0400 Subject: [PATCH 1/3] Read contract state as of the request's own time; refuse late state changes trigger! handed every clause today's state and wrote the new state valid from the request's time with no end. A report arriving after later ones therefore computed from the future and overwrote every later state in valid time: with visits on Feb 2 and Feb 20 recorded, a late Jan 15 report made Jan 20 and Feb 10 both read back as three visits paid. - current-state and store/get-instance take :at; trigger! reads the state as of its own now. - A trigger whose new state would land before a state change already recorded is refused as :out-of-order (HTTP 409), checked before the write and again by an ASSERT inside the transaction. - Answer-only triggers (attestations) may still be back-dated. Replaying later executions to accept a late report is left as a design decision, since a replay can change answers already acted on. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 36 ++++--- docs/walkthrough.md | 1 + src/com/trustblocks/app/trigger.clj | 1 + src/com/trustblocks/xtdb/execute.clj | 81 +++++++++++++-- src/com/trustblocks/xtdb/store.clj | 12 ++- test/com/trustblocks/app/trigger_test.clj | 78 ++++++++++++++ test/com/trustblocks/xtdb/execute_test.clj | 114 +++++++++++++++++++++ 7 files changed, 296 insertions(+), 27 deletions(-) create mode 100644 test/com/trustblocks/app/trigger_test.clj diff --git a/AGENTS.md b/AGENTS.md index c4d29d4..2c1b54b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -264,9 +264,10 @@ not a starting point to improve on.** rely on manual verification alone. - **Coverage is uneven, intentionally not backfilled wholesale**: DID/ identity/credential/certification/certify (the newest code) have real unit - tests. `template/render.clj`, `model/graph.clj`, `app/trigger.clj`, - `app/contracts.clj` do not — they're exercised by the `dev/*_demo.clj` - scripts and manual curl instead. Don't assume test presence maps to code + tests. `app/trigger.clj` has a handler test for its 409 refusals only. + `template/render.clj`, `model/graph.clj` and `app/contracts.clj` have + none — they're exercised by the `dev/*_demo.clj` scripts and manual curl + instead. Don't assume test presence maps to code age or importance. - **One PR per coherent increment**, each independently mergeable, each with its own "what/why/verification" description — this repo's entire history @@ -354,13 +355,16 @@ not a starting point to improve on.** Display code (`attestations-for-contract`) doesn't walk the chain at all, so this is inert today, but a future feature that *does* walk evidence chains should treat this as unverified territory. -- **`current-state`'s temporal semantics** (`execute.clj`) were not fully - traced this session — it's unclear from a quick read alone whether it - reads state as of real wall-clock "now" or some other point when a clause - consults `state`. Every real scenario tested so far only needed "state - after a permanent, never-revoked approval," which is insensitive to this - distinction; don't assume it's been verified for a case where state - changes and un-changes over time. +- **Late-arriving state changes are refused, not replayed.** `trigger!` + hands a clause the state as of the request's own `now` (valid time), and + refuses with `:out-of-order` (HTTP 409) any trigger whose *new state* would + land before a state change already recorded — writing it would overwrite + later state in valid time (it had been doing exactly that before the fix). + Answer-only triggers (attestations) may still be back-dated. Accepting a + late report properly means replaying every later execution in valid-time + order — possible because clauses are pure — but a replay can change answers + already acted on (payouts), so it is a design decision still open, not a + missing guard. See `execute/assert-no-later-state`. - **`docs/walkthrough.md` promises every output came from a real run.** Part 5 (identity, DID/VC, attestation) was checked by executing its Clojure blocks verbatim. If you change what `certify!`, `grant-office!`, @@ -419,7 +423,7 @@ identity for this email". ```sh bin/clj -M:run test ``` -Currently 45 tests / 151 assertions, 0 failures. Focus a namespace or var +Currently 50 tests / 175 assertions, 0 failures. Focus a namespace or var with `--focus `. **Lint + format + test together** (what to run before considering any change @@ -462,10 +466,12 @@ bin/template-conformance REPL path now exists in `docs/operator-runbook.md`; onboarding no longer requires reverse-engineering the seed script, but it is still manual and deliberately has no browser write surface. -3. **Trace `current-state`'s exact temporal read semantics** (see §6) and - add a test that actually exercises a contract whose state changes more - than once, so the "as of now vs. as of the clause's own `now`" question - has a checked answer instead of an assumed one. +3. **Design contract state beyond one snapshot** — replay of late-arriving + executions (see §6), and a data-driven statechart for multi-step contracts + (e.g. construction: inspections certified by an engineer or planning + director, progress payments, retainage). Align with Accord's proposed + `runtime@1.0.0.AgreementState` and `obligation@1.0.0.ObligationTransition` + (accordproject/models PR #200, open) rather than inventing new shapes. 4. **If a second contract template is ever added, use it to pressure-test `MunicipalAgreement` reuse and the attestation-clause pattern** — both are currently proven against exactly one real contract type each. A diff --git a/docs/walkthrough.md b/docs/walkthrough.md index e6ea6c6..bf77f34 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -436,6 +436,7 @@ Failures are a code to branch on and a message to read: | `400 invalid-request` | the request is not a valid instance of its `$class` | | `404 no-such-contract` | | | `409 duplicate-request` | that `requestId` has already run on this contract | +| `409 out-of-order` | the request would change state before a state change already recorded (`at` is too early) | | `422 clause-refused` | the contract declined — *Already approved by the Board* is an answer, not a failure | | `422 no-clause` | no logic is stored for this contract's type | | `501 runtime-unavailable` | the clause's runtime is not implemented here (today: `:typescript`) | diff --git a/src/com/trustblocks/app/trigger.clj b/src/com/trustblocks/app/trigger.clj index a36df30..8946a5b 100644 --- a/src/com/trustblocks/app/trigger.clj +++ b/src/com/trustblocks/app/trigger.clj @@ -100,6 +100,7 @@ :invalid-request 400 :no-such-contract 404 :duplicate-request 409 + :out-of-order 409 :no-clause 422 :clause-refused 422 :runtime-unavailable 501 diff --git a/src/com/trustblocks/xtdb/execute.clj b/src/com/trustblocks/xtdb/execute.clj index 9b86cfb..4759afb 100644 --- a/src/com/trustblocks/xtdb/execute.clj +++ b/src/com/trustblocks/xtdb/execute.clj @@ -135,9 +135,14 @@ ;; ---------------------------------------------------------------- state (defn current-state - "The contract's State instance as Concerto JSON, or nil before any execution." - [node contract-id & {:keys [reg]}] - (store/get-instance node contract-id :table state-table :reg reg)) + "The contract's State instance as Concerto JSON, or nil before any execution. + + As of wall-clock now by default. `:at` (an Instant) reads it as it stood at + that point in valid time instead -- which is what trigger! passes, so a + clause sees the state its own request took effect in, not whatever later + executions have made of it since." + [node contract-id & {:keys [reg at]}] + (store/get-instance node contract-id :table state-table :reg reg :at at)) (defn state-as-of "The State as it stood at `instant` in valid time -- the question a lawyer @@ -201,6 +206,45 @@ "WHERE contract = ? AND request_id = ?)") [contract-id request-id]]) +(defn- later-state-change + "The valid-from of this contract's earliest state change strictly after + `at`, or nil if there is none." + [node contract-id at] + (->> (xt/q node (list 'from state-table + {:for-valid-time :all-time + :bind [{:xt/id contract-id} 'xt/valid-from]})) + (map #(.toInstant ^java.time.ZonedDateTime (:xt/valid-from %))) + (filter #(.isAfter ^java.time.Instant % at)) + sort + first)) + +(defn- out-of-order! [contract-id at later] + (throw (ex-info (str "This request takes effect before a state change already " + "recorded for this contract") + {:trustblocks/error :out-of-order + :contract contract-id + :at (str at) + :later-state-from (str later)}))) + +(defn- assert-no-later-state + "A transaction op that aborts the whole transaction if this contract's state + already changed after `at` -- the in-transaction half of later-state-change, + closing the gap between reading and writing the way assert-unused does for + request ids. + + Why refuse at all: a new state is written valid from `at` onward with no + end, so writing it behind a later change would overwrite that change in + valid time, and the later execution's own answer was computed from a state + that no longer includes this one. Accepting a late report properly means + re-running every later execution in valid-time order -- possible, since + clauses are pure, but a replay can change answers already acted on (a + payout already made), which is a decision for a person, not a silent + side effect of a trigger." + [contract-id at] + [:sql (str "ASSERT NOT EXISTS (SELECT 1 FROM contract_state FOR ALL VALID_TIME " + "WHERE _id = ? AND _valid_from > ?)") + [contract-id at]]) + (defn- assert-failed? [e] (= :xtdb/assert-failed (:xtdb.error/code (ex-data e)))) @@ -256,6 +300,11 @@ landed. Re-running is not harmless -- a second snow-removal visit report would quietly pay the contractor twice. + The clause sees the state as it stood at `now`, not as it stands today, and + a trigger whose new state would land behind a state change already recorded + is refused as :out-of-order rather than overwriting it -- see + assert-no-later-state. + `:by` records who submitted it. Everything else here answers what happened and when; for a record that may be read in a dispute, who is the third question, and the one nothing else in the row answers. @@ -292,7 +341,8 @@ :errors (restate-errors reg (get request "$class") errs)}))) - (let [state (current-state node contract-id :reg reg) + (let [at (->instant now) + state (current-state node contract-id :reg reg :at at) result (run-clause runtime ctx src {:data contract :request request :state state :now now}) response (:response result)] @@ -311,12 +361,21 @@ :errors (restate-errors reg (get instance "$class") errs)})))) + ;; A clause that only answers -- an attestation, say -- may be dated + ;; before later state changes: it read the state as of its own time and + ;; writes none. Only a new state has to land after every earlier one. + (when (:state result) + (when-let [later (later-state-change node contract-id at)] + (out-of-order! contract-id at later))) + (try (xt/execute-tx node (cond-> [] request-id (conj (assert-unused contract-id request-id)) + (:state result) (conj (assert-no-later-state contract-id at)) + :always (conj [:put-docs execution-table (cond-> {:xt/id (random-uuid) @@ -330,17 +389,21 @@ ;; The new state is valid from the moment the request took effect, ;; not from when the database learned of it. That separation is the - ;; whole point: a back-dated correction lands in valid time while - ;; system time still records when we believed it. + ;; whole point: a back-dated execution lands in valid time while + ;; system time still records when we believed it -- provided it is + ;; not behind a later state change, see assert-no-later-state. (:state result) - (conj [:put-docs {:into state-table :valid-from (->instant now)} + (conj [:put-docs {:into state-table :valid-from at} (k/instance->doc (:state result) :identity-field (mm/identity-field reg (get (:state result) "$class")))]))) (catch Exception e (if (assert-failed? e) - (duplicate! contract-id request-id - (execution-for-request node contract-id request-id)) + ;; Two assertions can fail; whichever condition now holds says + ;; which one lost the race. + (if-let [prior (execution-for-request node contract-id request-id)] + (duplicate! contract-id request-id prior) + (out-of-order! contract-id at (later-state-change node contract-id at))) (throw e)))) {:response response diff --git a/src/com/trustblocks/xtdb/store.clj b/src/com/trustblocks/xtdb/store.clj index 121d4ce..508ebb7 100644 --- a/src/com/trustblocks/xtdb/store.clj +++ b/src/com/trustblocks/xtdb/store.clj @@ -135,12 +135,18 @@ (defn get-instance "Read an instance back out as Concerto JSON, restoring $class and the original - camelCase property names." - [node id & {:keys [table reg] :or {table contract-table}}] + camelCase property names. + + `:at`, an Instant, reads the instance as it stood at that point in valid + time rather than as it stands now." + [node id & {:keys [table reg at] :or {table contract-table}}] ;; The id must be matched as a literal in the binding pattern. Writing ;; {:xt/id $id} with an :args param does NOT filter -- XTDB treats $id as ;; another column to project and returns the entire table. - (let [q (list 'from table [{:xt/id id} (symbol "*")])] + (let [bind [{:xt/id id} (symbol "*")] + q (list 'from table (if at + {:for-valid-time (list 'at at) :bind bind} + bind))] (when-let [record (first (xt/q node q))] ;; Falls back to the stored schema rather than to ->camelCase: inferring ;; the name is lossy for anything not plain camelCase (vechileID, diff --git a/test/com/trustblocks/app/trigger_test.clj b/test/com/trustblocks/app/trigger_test.clj new file mode 100644 index 0000000..49edf97 --- /dev/null +++ b/test/com/trustblocks/app/trigger_test.clj @@ -0,0 +1,78 @@ +(ns com.trustblocks.app.trigger-test + "Exercises the trigger handler directly, as com.trustblocks.app.certify-test + does -- a plain ctx map and a function call, no HTTP server. Covers the + status each refusal reaches a caller with, starting with the one most + likely to surprise one: a report dated before a state change already + recorded." + (:require [cheshire.core :as json] + [clojure.test :refer [deftest is testing]] + [com.trustblocks.app.auth :as auth] + [com.trustblocks.app.trigger :as trigger] + [com.trustblocks.xtdb.execute :as execute] + [com.trustblocks.xtdb.store :as store] + [xtdb.node :as xt.node])) + +(def ^:private fqn "com.trustblocks.snow@1.0.0.SnowRemoval") +(def ^:private contract-id "snow-trigger") + +(def ^:private clause + "(let [paid (if state (get state \"visitsPaid\") 0)] + {:response {\"$class\" \"com.trustblocks.snow@1.0.0.VisitOutcome\" + \"outcomeId\" (get request \"reportId\") + \"$identifier\" (get request \"reportId\") + \"$timestamp\" now + \"outcome\" \"PAYABLE\" + \"amountDue\" {\"$class\" \"com.trustblocks.snow@1.0.0.Money\" + \"amount\" 480.0 \"currency\" \"USD\"}} + :state {\"$class\" \"com.trustblocks.snow@1.0.0.PaymentState\" + \"contractId\" (get data \"contractId\") + \"$identifier\" (get data \"contractId\") + \"visitsPaid\" (+ paid 1) + \"totalPaid\" {\"$class\" \"com.trustblocks.snow@1.0.0.Money\" + \"amount\" (* 480.0 (+ paid 1)) \"currency\" \"USD\"}}})") + +(defn- populate! [node] + (store/put-model! node "docs/examples/snow-removal.cto") + (execute/put-clause! node fqn clause) + (store/put-instance! node nil {"$class" fqn + "contractId" contract-id + "$identifier" contract-id + "contractor" "Glenwood Plowing LLC" + "route" "Ward 3 arterials" + "inchesTrigger" 2.0 + "ratePerVisit" {"$class" "com.trustblocks.snow@1.0.0.Money" + "amount" 480.0 + "currency" "USD"}})) + +(defn- post! + "The handler's reply to a visit report taking effect at `at`, with its + JSON body parsed. body-params are keyword-keyed, as muuntaja leaves them." + [node uid request-id at] + (let [{:keys [status body]} + (trigger/trigger {:biff.xtdb/node node + :uri (str "/contracts/" contract-id "/trigger") + :session {:uid uid} + :body-params {:requestId request-id + :at at + :request {:$class "com.trustblocks.snow@1.0.0.VisitReport" + :reportId request-id + :$identifier request-id + :$timestamp at + :visitedAt at + :inchesFallen 3.5}}})] + {:status status :body (json/parse-string body)})) + +(deftest a-late-report-is-refused-with-409 + (with-open [node (xt.node/start-node)] + (populate! node) + (let [uid (auth/create-user! {:biff.xtdb/node node} {:email "ops@example.gov"})] + (testing "an on-time report runs" + (is (= 200 (:status (post! node uid "v-2" "2026-02-20T06:00:00Z"))))) + (testing "one dated before it is refused, naming the later state change" + (let [{:keys [status body]} (post! node uid "v-1" "2026-01-15T06:00:00Z")] + (is (= 409 status)) + (is (= "out-of-order" (get body "error"))))) + (testing "a repeat of the on-time report is still a duplicate, not out of order" + (let [{:keys [status body]} (post! node uid "v-2" "2026-02-20T06:00:00Z")] + (is (= 409 status)) + (is (= "duplicate-request" (get body "error")))))))) diff --git a/test/com/trustblocks/xtdb/execute_test.clj b/test/com/trustblocks/xtdb/execute_test.clj index e1ea2ec..68362cf 100644 --- a/test/com/trustblocks/xtdb/execute_test.clj +++ b/test/com/trustblocks/xtdb/execute_test.clj @@ -12,6 +12,7 @@ (:require [clojure.test :refer [deftest is testing]] [com.trustblocks.xtdb.execute :as execute] [com.trustblocks.xtdb.store :as store] + [xtdb.api :as xt] [xtdb.node :as xt.node])) (def ^:private fqn "com.trustblocks.snow@1.0.0.SnowRemoval") @@ -130,3 +131,116 @@ (let [[execution] (execute/history node "snow-d")] (is (= "clerk@example.gov" (:by execution))) (is (= "r-1" (:request-id execution)))))) + +;; ------------------------------------------------------------ valid time + +(defn- visit-at + "A visit report taking effect at `at`, with `inches` of snow." + [report-id at inches] + {"$class" "com.trustblocks.snow@1.0.0.VisitReport" + "reportId" report-id + "$identifier" report-id + "$timestamp" at + "visitedAt" at + "inchesFallen" inches}) + +(defn- visits-paid-at [node id at] + (get (execute/current-state node id :at (java.time.Instant/parse at)) "visitsPaid")) + +(deftest state-changes-more-than-once-and-reads-back-at-each-date + (with-open [node (xt.node/start-node)] + (populate! node ["snow-e"]) + (doseq [[id at] [["v-1" "2026-01-15T06:00:00Z"] + ["v-2" "2026-02-02T06:00:00Z"] + ["v-3" "2026-02-20T06:00:00Z"]]] + (execute/trigger! node "snow-e" (visit-at id at 3.5) at)) + (is (nil? (visits-paid-at node "snow-e" "2026-01-01T00:00:00Z"))) + (is (= 1 (visits-paid-at node "snow-e" "2026-01-20T00:00:00Z"))) + (is (= 2 (visits-paid-at node "snow-e" "2026-02-10T00:00:00Z"))) + (is (= 3 (visits-paid-at node "snow-e" "2026-03-01T00:00:00Z"))) + (is (= 3 (visits-paid node "snow-e"))))) + +(deftest a-late-state-change-is-refused-rather-than-overwriting-later-ones + ;; Before this guard, a Jan 15 report arriving after Feb 2 and Feb 20 read + ;; today's state (two visits), and wrote "three visits" valid from Jan 15 + ;; with no end -- so Jan 20 and Feb 10 both read back as three. + (with-open [node (xt.node/start-node)] + (populate! node ["snow-f"]) + (execute/trigger! node "snow-f" (visit-at "v-2" "2026-02-02T06:00:00Z" 3.5) + "2026-02-02T06:00:00Z") + (execute/trigger! node "snow-f" (visit-at "v-3" "2026-02-20T06:00:00Z" 3.5) + "2026-02-20T06:00:00Z") + (is (= :out-of-order + (error-tag #(execute/trigger! node "snow-f" + (visit-at "v-1" "2026-01-15T06:00:00Z" 3.5) + "2026-01-15T06:00:00Z" :request-id "late-1")))) + (testing "nothing was written: the history, and every date, are as before" + (is (= 2 (count (execute/history node "snow-f")))) + (is (nil? (visits-paid-at node "snow-f" "2026-01-20T00:00:00Z"))) + (is (= 1 (visits-paid-at node "snow-f" "2026-02-10T00:00:00Z"))) + (is (= 2 (visits-paid node "snow-f")))) + (testing "a refused request id was not consumed" + (is (nil? (execute/execution-for-request node "snow-f" "late-1")))) + (testing "a state change at the same instant as the latest one still lands" + (execute/trigger! node "snow-f" (visit-at "v-4" "2026-02-20T06:00:00Z" 3.5) + "2026-02-20T06:00:00Z") + (is (= 3 (visits-paid node "snow-f")))))) + +(def ^:private answer-below-trigger-clause + "Like `clause`, but a visit below the trigger depth writes no state -- it + only answers, reporting in amountDue how many visits had been paid as of + its own time, so a test can see which state the clause was handed." + "(let [paid (if state (get state \"visitsPaid\") 0) + rate (get-in data [\"ratePerVisit\" \"amount\"])] + (if (< (get request \"inchesFallen\") (get data \"inchesTrigger\")) + {:response {\"$class\" \"com.trustblocks.snow@1.0.0.VisitOutcome\" + \"outcomeId\" (get request \"reportId\") + \"$identifier\" (get request \"reportId\") + \"$timestamp\" now + \"outcome\" \"BELOW_TRIGGER\" + \"amountDue\" {\"$class\" \"com.trustblocks.snow@1.0.0.Money\" + \"amount\" (* 1.0 paid) \"currency\" \"USD\"}}} + {:response {\"$class\" \"com.trustblocks.snow@1.0.0.VisitOutcome\" + \"outcomeId\" (get request \"reportId\") + \"$identifier\" (get request \"reportId\") + \"$timestamp\" now + \"outcome\" \"PAYABLE\" + \"amountDue\" {\"$class\" \"com.trustblocks.snow@1.0.0.Money\" + \"amount\" rate \"currency\" \"USD\"}} + :state {\"$class\" \"com.trustblocks.snow@1.0.0.PaymentState\" + \"contractId\" (get data \"contractId\") + \"$identifier\" (get data \"contractId\") + \"visitsPaid\" (+ paid 1) + \"totalPaid\" {\"$class\" \"com.trustblocks.snow@1.0.0.Money\" + \"amount\" (* rate (+ paid 1)) \"currency\" \"USD\"}}}))") + +(deftest a-back-dated-trigger-that-writes-no-state-runs-against-its-own-time + ;; The attestation case: certifying a months-old event after the contract + ;; has moved on is legitimate, because it only answers. + (with-open [node (xt.node/start-node)] + (populate! node ["snow-g"]) + (execute/put-clause! node fqn answer-below-trigger-clause) + (execute/trigger! node "snow-g" (visit-at "v-2" "2026-02-20T06:00:00Z" 4.0) + "2026-02-20T06:00:00Z") + (let [{:keys [response state]} + (execute/trigger! node "snow-g" (visit-at "v-1" "2026-01-15T06:00:00Z" 1.0) + "2026-01-15T06:00:00Z")] + (is (= "BELOW_TRIGGER" (get response "outcome"))) + (is (nil? state)) + (is (= 0.0 (get-in response ["amountDue" "amount"])) + "the clause saw the state as of Jan 15 (nothing paid), not today's") + (is (= 1 (visits-paid node "snow-g")))))) + +(deftest the-in-transaction-guard-catches-a-late-state-change-on-its-own + ;; trigger! checks for a later state change before writing, so the ASSERT + ;; is only reached when two triggers race. Exercise it directly, so a + ;; mistake in its SQL cannot hide behind the query that usually runs first. + (with-open [node (xt.node/start-node)] + (populate! node ["snow-h"]) + (execute/trigger! node "snow-h" (visit-at "v-2" "2026-02-20T06:00:00Z" 3.5) + "2026-02-20T06:00:00Z") + (let [guard #'execute/assert-no-later-state] + (is (thrown? Exception + (xt/execute-tx node [(guard "snow-h" (java.time.Instant/parse "2026-01-15T06:00:00Z"))]))) + (is (some? (xt/execute-tx node [(guard "snow-h" (java.time.Instant/parse "2026-02-20T06:00:00Z"))]))) + (is (some? (xt/execute-tx node [(guard "snow-other" (java.time.Instant/parse "2026-01-15T06:00:00Z"))])))))) From 7390da223e1a34a5e5d014d1e55d2778c3d8920f Mon Sep 17 00:00:00 2001 From: Tom Brooke Date: Thu, 24 Sep 2026 11:52:53 -0400 Subject: [PATCH 2/3] Guard concurrent triggers optimistically; accept a Date as now Review of #19 found the in-transaction check only rejected state changes dated after the request, so two triggers dated at or before one another could both read the same state and one would silently drop the other's change. The transaction now asserts that no state has been written since the version the clause read, and trigger! re-runs the pure clause against the new state when it has (up to five attempts, then :state-conflict, 409). The order check stays a refusal, re-evaluated on each attempt. trigger! also normalises a non-string now (an Instant or java.util.Date from a REPL) to its ISO string, so a clause never holds a platform date and the order check no longer throws ClassCastException on a Date. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 7 +- src/com/trustblocks/app/trigger.clj | 1 + src/com/trustblocks/xtdb/execute.clj | 232 +++++++++++++-------- test/com/trustblocks/xtdb/execute_test.clj | 59 +++++- 4 files changed, 198 insertions(+), 101 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2c1b54b..3e3fac9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -364,7 +364,10 @@ not a starting point to improve on.** late report properly means replaying every later execution in valid-time order — possible because clauses are pure — but a replay can change answers already acted on (payouts), so it is a design decision still open, not a - missing guard. See `execute/assert-no-later-state`. + missing guard. Concurrent triggers on one contract are optimistic: the + transaction asserts the state the clause read is still the latest, and + `trigger!` re-runs the (pure) clause if it is not. See + `execute/assert-state-unchanged`. - **`docs/walkthrough.md` promises every output came from a real run.** Part 5 (identity, DID/VC, attestation) was checked by executing its Clojure blocks verbatim. If you change what `certify!`, `grant-office!`, @@ -423,7 +426,7 @@ identity for this email". ```sh bin/clj -M:run test ``` -Currently 50 tests / 175 assertions, 0 failures. Focus a namespace or var +Currently 52 tests / 181 assertions, 0 failures. Focus a namespace or var with `--focus `. **Lint + format + test together** (what to run before considering any change diff --git a/src/com/trustblocks/app/trigger.clj b/src/com/trustblocks/app/trigger.clj index 8946a5b..344aa23 100644 --- a/src/com/trustblocks/app/trigger.clj +++ b/src/com/trustblocks/app/trigger.clj @@ -101,6 +101,7 @@ :no-such-contract 404 :duplicate-request 409 :out-of-order 409 + :state-conflict 409 :no-clause 422 :clause-refused 422 :runtime-unavailable 501 diff --git a/src/com/trustblocks/xtdb/execute.clj b/src/com/trustblocks/xtdb/execute.clj index 4759afb..7956603 100644 --- a/src/com/trustblocks/xtdb/execute.clj +++ b/src/com/trustblocks/xtdb/execute.clj @@ -40,7 +40,8 @@ other way." [dt] (cond - (inst? dt) dt + (instance? java.time.Instant dt) dt + (inst? dt) (.toInstant ^java.util.Date dt) (string? dt) (-> (try (java.time.OffsetDateTime/parse dt) (catch Exception _ (java.time.OffsetDateTime/parse (str dt "Z")))) @@ -226,24 +227,49 @@ :at (str at) :later-state-from (str later)}))) -(defn- assert-no-later-state - "A transaction op that aborts the whole transaction if this contract's state - already changed after `at` -- the in-transaction half of later-state-change, - closing the gap between reading and writing the way assert-unused does for - request ids. - - Why refuse at all: a new state is written valid from `at` onward with no - end, so writing it behind a later change would overwrite that change in - valid time, and the later execution's own answer was computed from a state - that no longer includes this one. Accepting a late report properly means +(defn- state-version + "The newest system-from among this contract's state rows, or nil before any + execution has written state. Any state write after it -- at whatever valid + time -- produces a row newer than this, so it identifies the state a clause + was handed." + [node contract-id] + (->> (xt/q node (list 'from state-table + {:for-valid-time :all-time + :bind [{:xt/id contract-id} 'xt/system-from]})) + (map #(.toInstant ^java.time.ZonedDateTime (:xt/system-from %))) + sort + last)) + +(defn- assert-state-unchanged + "A transaction op that aborts the whole transaction if any state has been + written for this contract since `version` (see state-version) -- the clause + ran against a state that is no longer the one on record, whether the new + write is dated before, at, or after this request. Optimistic concurrency: + trigger! runs the clause again against the new state, which is safe because + nothing was written and the clause is pure. + + The order check (later-state-change, before the write) is separate and is + a refusal rather than a retry. A new state is written valid from `at` + onward with no end, so writing it behind a later change would overwrite + that change in valid time. Accepting a late report properly means re-running every later execution in valid-time order -- possible, since clauses are pure, but a replay can change answers already acted on (a - payout already made), which is a decision for a person, not a silent - side effect of a trigger." - [contract-id at] - [:sql (str "ASSERT NOT EXISTS (SELECT 1 FROM contract_state FOR ALL VALID_TIME " - "WHERE _id = ? AND _valid_from > ?)") - [contract-id at]]) + payout already made), which is a decision for a person, not a silent side + effect of a trigger." + [contract-id version] + (if version + [:sql (str "ASSERT NOT EXISTS (SELECT 1 FROM contract_state FOR ALL VALID_TIME " + "WHERE _id = ? AND _system_from > ?)") + [contract-id version]] + [:sql (str "ASSERT NOT EXISTS (SELECT 1 FROM contract_state FOR ALL VALID_TIME " + "WHERE _id = ?)") + [contract-id]])) + +(def ^:private max-attempts + "How many times trigger! re-runs a clause whose state changed underneath it + before giving up. Each retry means another writer committed in between, so + exhausting this takes sustained contention on one contract." + 5) (defn- assert-failed? [e] (= :xtdb/assert-failed (:xtdb.error/code (ex-data e)))) @@ -280,6 +306,85 @@ {:trustblocks/error :runtime-unavailable :runtime runtime})))) +(defn- run-and-record! + "One attempt at trigger!'s clause run and write. Returns trigger!'s result, + or ::stale if another execution wrote this contract's state in between -- + see assert-state-unchanged." + [node reg ctx contract-id contract src runtime request now at request-id by] + (let [version (state-version node contract-id) + state (current-state node contract-id :reg reg :at at) + result (run-clause runtime ctx src {:data contract :request request + :state state :now now}) + response (:response result)] + + (when-not response + (throw (ex-info "Clause returned no :response" + {:trustblocks/error :invalid-clause-output + :contract contract-id + :returned (keys result)}))) + (doseq [[what instance] [[:response response] [:state (:state result)]] + :when instance] + (when-let [errs (store/validate reg instance)] + (throw (ex-info (str "Clause produced an invalid " (name what)) + {:trustblocks/error :invalid-clause-output + :$class (get instance "$class") + :errors (restate-errors + reg (get instance "$class") errs)})))) + + ;; A clause that only answers -- an attestation, say -- may be dated + ;; before later state changes: it read the state as of its own time and + ;; writes none. Only a new state has to land after every earlier one. + (when (:state result) + (when-let [later (later-state-change node contract-id at)] + (out-of-order! contract-id at later))) + + (let [written + (try + (xt/execute-tx + node + (cond-> [] + request-id (conj (assert-unused contract-id request-id)) + + :always (conj (assert-state-unchanged contract-id version)) + + :always + (conj [:put-docs execution-table + (cond-> {:xt/id (random-uuid) + :contract contract-id + :at now + :request (k/->xtdb request) + :response (k/->xtdb response) + :clause src} + request-id (assoc :request-id request-id) + by (assoc :by by))]) + + ;; The new state is valid from the moment the request took effect, + ;; not from when the database learned of it. That separation is the + ;; whole point: a back-dated execution lands in valid time while + ;; system time still records when we believed it -- provided it is + ;; not behind a later state change, see assert-state-unchanged. + (:state result) + (conj [:put-docs {:into state-table :valid-from at} + (k/instance->doc (:state result) + :identity-field (mm/identity-field + reg (get (:state result) "$class")))]))) + (catch Exception e + (if (assert-failed? e) + ;; Two assertions can fail. A used request id is final; otherwise + ;; the state moved underneath the clause, and trigger! runs it + ;; again -- which also re-checks the order against the new state. + (if-let [prior (execution-for-request node contract-id request-id)] + (duplicate! contract-id request-id prior) + ::stale) + (throw e))))] + + (if (= ::stale written) + ::stale + {:response response + :state (:state result) + :at now + :request-id request-id})))) + (defn trigger! "Run a contract's clause against `request` and record what happened. @@ -302,8 +407,9 @@ The clause sees the state as it stood at `now`, not as it stands today, and a trigger whose new state would land behind a state change already recorded - is refused as :out-of-order rather than overwriting it -- see - assert-no-later-state. + is refused as :out-of-order rather than overwriting it. If another execution + writes this contract's state while the clause runs, the clause is run again + against the new state -- see assert-state-unchanged. `:by` records who submitted it. Everything else here answers what happened and when; for a record that may be read in a dispute, who is the third @@ -313,7 +419,12 @@ a bad request from a refusal from a broken clause without reading messages. See com.trustblocks.app.trigger, which turns those tags into status codes." [node contract-id request now & {:keys [reg ctx request-id by]}] - (let [reg (or reg (store/registry-from-xtdb node)) + (let [;; A clause receives DateTimes as wire strings, never a platform date + ;; (see com.trustblocks.clause), so an Instant or Date from a REPL is + ;; normalised here. A string passes through untouched, keeping the + ;; caller's own spelling of it. + now (if (string? now) now (str (->instant now))) + reg (or reg (store/registry-from-xtdb node)) ctx (or ctx (clause/context)) contract (or (store/get-instance node contract-id :reg reg) (throw (ex-info "No such contract" @@ -341,72 +452,17 @@ :errors (restate-errors reg (get request "$class") errs)}))) - (let [at (->instant now) - state (current-state node contract-id :reg reg :at at) - result (run-clause runtime ctx src {:data contract :request request - :state state :now now}) - response (:response result)] + (let [at (->instant now)] + (loop [attempt 1] + (let [outcome (run-and-record! node reg ctx contract-id contract src runtime + request now at request-id by)] + (cond + (not= ::stale outcome) outcome - (when-not response - (throw (ex-info "Clause returned no :response" - {:trustblocks/error :invalid-clause-output - :contract contract-id - :returned (keys result)}))) - (doseq [[what instance] [[:response response] [:state (:state result)]] - :when instance] - (when-let [errs (store/validate reg instance)] - (throw (ex-info (str "Clause produced an invalid " (name what)) - {:trustblocks/error :invalid-clause-output - :$class (get instance "$class") - :errors (restate-errors - reg (get instance "$class") errs)})))) - - ;; A clause that only answers -- an attestation, say -- may be dated - ;; before later state changes: it read the state as of its own time and - ;; writes none. Only a new state has to land after every earlier one. - (when (:state result) - (when-let [later (later-state-change node contract-id at)] - (out-of-order! contract-id at later))) - - (try - (xt/execute-tx - node - (cond-> [] - request-id (conj (assert-unused contract-id request-id)) - - (:state result) (conj (assert-no-later-state contract-id at)) - - :always - (conj [:put-docs execution-table - (cond-> {:xt/id (random-uuid) - :contract contract-id - :at now - :request (k/->xtdb request) - :response (k/->xtdb response) - :clause src} - request-id (assoc :request-id request-id) - by (assoc :by by))]) - - ;; The new state is valid from the moment the request took effect, - ;; not from when the database learned of it. That separation is the - ;; whole point: a back-dated execution lands in valid time while - ;; system time still records when we believed it -- provided it is - ;; not behind a later state change, see assert-no-later-state. - (:state result) - (conj [:put-docs {:into state-table :valid-from at} - (k/instance->doc (:state result) - :identity-field (mm/identity-field - reg (get (:state result) "$class")))]))) - (catch Exception e - (if (assert-failed? e) - ;; Two assertions can fail; whichever condition now holds says - ;; which one lost the race. - (if-let [prior (execution-for-request node contract-id request-id)] - (duplicate! contract-id request-id prior) - (out-of-order! contract-id at (later-state-change node contract-id at))) - (throw e)))) - - {:response response - :state (:state result) - :at now - :request-id request-id}))) + (< attempt max-attempts) (recur (inc attempt)) + + :else + (throw (ex-info "The contract's state kept changing while this request ran" + {:trustblocks/error :state-conflict + :contract contract-id + :attempts attempt})))))))) diff --git a/test/com/trustblocks/xtdb/execute_test.clj b/test/com/trustblocks/xtdb/execute_test.clj index 68362cf..0e3e253 100644 --- a/test/com/trustblocks/xtdb/execute_test.clj +++ b/test/com/trustblocks/xtdb/execute_test.clj @@ -231,16 +231,53 @@ "the clause saw the state as of Jan 15 (nothing paid), not today's") (is (= 1 (visits-paid node "snow-g")))))) -(deftest the-in-transaction-guard-catches-a-late-state-change-on-its-own - ;; trigger! checks for a later state change before writing, so the ASSERT - ;; is only reached when two triggers race. Exercise it directly, so a - ;; mistake in its SQL cannot hide behind the query that usually runs first. +(deftest the-in-transaction-guard-catches-any-state-written-since-the-read + ;; trigger! checks the order before writing, so this ASSERT is only reached + ;; when triggers race. Exercise it directly, so a mistake in its SQL cannot + ;; hide behind the query that usually runs first. (with-open [node (xt.node/start-node)] (populate! node ["snow-h"]) - (execute/trigger! node "snow-h" (visit-at "v-2" "2026-02-20T06:00:00Z" 3.5) - "2026-02-20T06:00:00Z") - (let [guard #'execute/assert-no-later-state] - (is (thrown? Exception - (xt/execute-tx node [(guard "snow-h" (java.time.Instant/parse "2026-01-15T06:00:00Z"))]))) - (is (some? (xt/execute-tx node [(guard "snow-h" (java.time.Instant/parse "2026-02-20T06:00:00Z"))]))) - (is (some? (xt/execute-tx node [(guard "snow-other" (java.time.Instant/parse "2026-01-15T06:00:00Z"))])))))) + (let [guard #'execute/assert-state-unchanged + version #(#'execute/state-version node "snow-h")] + (testing "before any state, any state write since fails it" + (is (nil? (version))) + (is (some? (xt/execute-tx node [(guard "snow-h" nil)])))) + (execute/trigger! node "snow-h" (visit-at "v-2" "2026-02-20T06:00:00Z" 3.5) + "2026-02-20T06:00:00Z") + (is (thrown? Exception (xt/execute-tx node [(guard "snow-h" nil)]))) + (let [read-version (version)] + (is (some? (xt/execute-tx node [(guard "snow-h" read-version)]))) + (testing "a write dated earlier than the latest still counts as a change" + (execute/trigger! node "snow-h" (visit-at "v-3" "2026-02-20T06:00:00Z" 3.5) + "2026-02-20T06:00:00Z") + (is (thrown? Exception (xt/execute-tx node [(guard "snow-h" read-version)])))))))) + +(deftest concurrent-state-changes-are-never-lost + ;; Triggers dated at or before one another, all reading the same state at + ;; once: without a guard, several compute "one more than what I read" and + ;; the last write wins, dropping visits. Every execution that lands must be + ;; counted; the rest are refused as out of order, never silently lost. + (with-open [node (xt.node/start-node)] + (populate! node ["snow-i"]) + (let [attempts (->> (range 8) + (mapv (fn [i] + (let [at (format "2026-02-%02dT06:00:00Z" (inc i))] + (future + (error-tag + #(execute/trigger! node "snow-i" + (visit-at (str "v-" i) at 3.5) + at :request-id (str "r-" i))))))) + (mapv deref)) + landed (count (execute/history node "snow-i"))] + (is (every? #{:out-of-order} (filter some? attempts)) (pr-str attempts)) + (is (= landed (count (remove some? attempts)))) + (is (= landed (visits-paid node "snow-i")))))) + +(deftest a-java-util-date-is-accepted-as-now + (with-open [node (xt.node/start-node)] + (populate! node ["snow-j"]) + (execute/trigger! node "snow-j" (visit-at "v-1" "2026-01-15T06:00:00Z" 3.5) + #inst "2026-01-15T06:00:00Z") + (execute/trigger! node "snow-j" (visit-at "v-2" "2026-02-02T06:00:00Z" 3.5) + #inst "2026-02-02T06:00:00Z") + (is (= 2 (visits-paid node "snow-j"))))) From 69b5c5d4f9bd85a8d9b4e9b76cb189ee5c087665 Mon Sep 17 00:00:00 2001 From: Tom Brooke Date: Thu, 24 Sep 2026 17:57:26 -0400 Subject: [PATCH 3/3] Check the request id on every trigger attempt, ahead of the order check After a concurrency retry, a concurrent copy of the same request may be the write that moved the state; its caller should hear duplicate-request with when it ran, not out-of-order. The ordinary resend case was already answered as a duplicate by trigger!'s early check; a test now pins that. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- src/com/trustblocks/xtdb/execute.clj | 5 +++++ test/com/trustblocks/xtdb/execute_test.clj | 13 +++++++++++++ 3 files changed, 19 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 3e3fac9..2708b3c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -426,7 +426,7 @@ identity for this email". ```sh bin/clj -M:run test ``` -Currently 52 tests / 181 assertions, 0 failures. Focus a namespace or var +Currently 53 tests / 182 assertions, 0 failures. Focus a namespace or var with `--focus `. **Lint + format + test together** (what to run before considering any change diff --git a/src/com/trustblocks/xtdb/execute.clj b/src/com/trustblocks/xtdb/execute.clj index 7956603..e4c5a03 100644 --- a/src/com/trustblocks/xtdb/execute.clj +++ b/src/com/trustblocks/xtdb/execute.clj @@ -311,6 +311,11 @@ or ::stale if another execution wrote this contract's state in between -- see assert-state-unchanged." [node reg ctx contract-id contract src runtime request now at request-id by] + ;; Asked again on every attempt, ahead of the order check: after a retry, a + ;; concurrent copy of this same request may be the write that moved the + ;; state, and its caller should hear "already executed", not "out of order". + (when-let [prior (execution-for-request node contract-id request-id)] + (duplicate! contract-id request-id prior)) (let [version (state-version node contract-id) state (current-state node contract-id :reg reg :at at) result (run-clause runtime ctx src {:data contract :request request diff --git a/test/com/trustblocks/xtdb/execute_test.clj b/test/com/trustblocks/xtdb/execute_test.clj index 0e3e253..1a1ecb4 100644 --- a/test/com/trustblocks/xtdb/execute_test.clj +++ b/test/com/trustblocks/xtdb/execute_test.clj @@ -281,3 +281,16 @@ (execute/trigger! node "snow-j" (visit-at "v-2" "2026-02-02T06:00:00Z" 3.5) #inst "2026-02-02T06:00:00Z") (is (= 2 (visits-paid node "snow-j"))))) + +(deftest a-retried-request-is-a-duplicate-even-behind-a-later-state-change + ;; A resent request that already landed is answered as a duplicate, with + ;; when it ran -- not as out of order, which would suggest it never did. + (with-open [node (xt.node/start-node)] + (populate! node ["snow-k"]) + (execute/trigger! node "snow-k" (visit-at "v-1" "2026-01-15T06:00:00Z" 3.5) + "2026-01-15T06:00:00Z" :request-id "r-1") + (execute/trigger! node "snow-k" (visit-at "v-2" "2026-02-02T06:00:00Z" 3.5) + "2026-02-02T06:00:00Z" :request-id "r-2") + (is (= :duplicate-request + (error-tag #(execute/trigger! node "snow-k" (visit-at "v-1" "2026-01-15T06:00:00Z" 3.5) + "2026-01-15T06:00:00Z" :request-id "r-1"))))))