Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -848,6 +848,34 @@ trimmedPatchData:

You can also provide explicit `start` and `end` byte positions (end is exclusive). Negative indexes count from the end of the byte array, so `end: -1` trims the last byte and `start: -32` keeps the final 32 bytes. `range` accepts either `start:end` or the bracket form `[start:end]`.

### `read-file`
Read the raw contents of a file as a value. This is the home for large, opaque, per-execution operational blobs (e.g. packed multisig signatures) that don't belong in `constants` and aren't typed build artifacts. The path is resolved relative to the directory of the job/template that uses it and is confined to the project root (absolute paths and `..` escapes are rejected):

```yaml
# Read packed Safe owner signatures written to a gitignored file next to the job
signatures:
type: "read-file"
arguments:
path: "signatures.hex"
encoding: "hex" # "utf8" (default), "hex", or "json"
```

`encoding: "utf8"` returns the text with a single trailing newline trimmed; `"hex"` validates and normalizes to a `0x`-prefixed lowercase hex string; `"json"` parses the file and returns the resulting value (composes with `read-json`).

### `concat`
Join resolved parts into a single string. Use this for URL/path templating instead of embedding `{{...}}` inside a longer literal (which is not interpolated — a `{{ref}}` is only resolved when it is the entire value):

```yaml
url:
type: "concat"
arguments:
values:
- "https://safe-transaction-mainnet.safe.global/api/v1/multisig-transactions/"
- "{{safe-tx-hash}}"
- "/"
separator: "" # optional, defaults to "" (direct concatenation)
```

Verify deployed contracts on block explorers:

```yaml
Expand Down Expand Up @@ -955,8 +983,58 @@ Catapult includes several standard templates:
- **`erc-2470`** and raw variant: CREATE2 Deployer (singleton factory)
- **`assured-deployment`**: Helper to ensure a contract is deployed at a specific address
- **`min-balance`**: Ensure minimum balance for any given address
- **`safe-exec-transaction`**: Assemble and broadcast a fully-signed Gnosis Safe `execTransaction` (see below)
- Raw building blocks: `raw-sequence-universal-deployer-2`, `raw-nano-universal-deployer`, `raw-erc-2470`

### `safe-exec-transaction`

Broadcasts a fully-signed Gnosis Safe transaction on-chain ("Shape 1" relay) instead of stopping at calldata for a human to paste into the Safe UI. It packs the owner signatures the Safe already collected into `execTransaction` and sends the outer transaction with the configured EOA as relayer.

The packed signatures are passed as the `signatures` argument. Where they come from is the caller's choice — the two common sources:

Offline / air-gapped (signatures collected into a gitignorable file):

```yaml
- name: "relay"
template: "safe-exec-transaction"
arguments:
safe: "{{safe_address}}"
to: "{{target}}"
data: "{{inner_calldata}}"
operation: "0" # 0 = CALL, 1 = DELEGATECALL
signatures:
type: "read-file"
arguments: { path: "signatures.hex", encoding: "hex" }
```

Safe Transaction Service (hosted flow — the service returns a top-level pre-packed `signatures` field):

```yaml
- name: "fetch-safe-tx"
type: "json-request"
arguments:
url:
type: "concat"
arguments:
values:
- "https://safe-transaction-mainnet.safe.global/api"
- "/v1/multisig-transactions/"
- "{{safe_tx_hash}}"
- "/"
- name: "relay"
template: "safe-exec-transaction"
arguments:
safe: "{{safe_address}}"
to: "{{target}}"
data: "{{inner_calldata}}"
operation: "0"
signatures:
type: "read-json"
arguments: { json: "{{fetch-safe-tx.response}}", path: "signatures" }
```

The template only assembles and broadcasts; it does not impose a post-execution skip condition, because the desired state of a Safe relay is the effect of the *inner* call. Wrap the calling job with `skip_if` observing that on-chain effect (e.g. `owner() == newOwner`) for idempotent, re-runnable convergence.

## Contract Resolution

Catapult automatically discovers and indexes contract artifacts in your project. It supports:
Expand Down
189 changes: 189 additions & 0 deletions notes/read-file-and-concat-value-resolvers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
# `read-file` + `concat` value resolvers (and why not a blob registry)

*Design note, branch `feat/read-file-value` (cut from `origin/master`, v1.5.0). Motivated by building a "Shape 1" Safe relay in `0xsequence/live-contracts` — a catapult job that broadcasts a fully-signed Gnosis Safe `execTransaction` on-chain, rather than emitting calldata for a human to paste into the Safe UI.*

## The problem

Building the relay we hit two real gaps:

1. **No home for an opaque per-execution blob.** The packed Safe owner signatures are a large chunk of hex that changes every execution and is pure operational data. The only place to put a value today is `constants` YAML. That works but is the wrong shelf: constants are meant to be small, stable, shared configuration, not big per-run payloads, and they can't be `.gitignore`d cleanly.

2. **`{{ref}}` only resolves when it is the *entire* value.** In `src/lib/core/resolver.ts` the reference match is anchored:

```ts
const refMatch = value.match(/^{{(.*)}}$/)
```

So `".../multisig-transactions/{{tx-hash}}/"` is sent **literally** — which 404'd us against the Safe Transaction Service. `networks.yaml` gets embedded interpolation, but only via a special-case regex in the network loader (`resolveRpcUrlTokens`, `RPC*` tokens only), not the general resolver.

We shipped the live-contracts job by working around both: a full-URL constant (couldn't template it) plus `json-request` + `read-json` to fetch the pre-packed `signatures` field from the tx-service at run time.

## Options considered

### A. Generic "blob registry" (a free-form `build-info`) — **rejected**

The original ask was: should catapult have a place to store "blobs of data" that isn't constants — a free-form version of build-info?

No. build-info earns its keep precisely because it is **typed and validated** (abi/bytecode, discoverable by hash) and referenced *semantically* via `Contract(name)`. A free-form analog has none of that — it is just "constants, but a second bag," with its own discovery/merge/duplicate-key machinery to build and maintain, and no added safety.

This repo's own `notes/roadmap-thinking.md` (§3) already diagnosed the adjacent pain — build-info blobs hand-copied from other repos with the source link lost — and concluded the right frame is **provenance metadata about a canonical artifact**, not a generic blob bucket. A blob registry pulls in the opposite direction. The blob problem is better served by the smallest primitive that lets a blob live in *its own file*: `read-file`.

### B. `read-file` value resolver — **built**

A blob lives in its own file, referenced by path, resolved relative to the job dir, gitignorable, with an encoding hint. Small, general, and it composes with the resolvers that already exist (`read-json`, `slice-bytes`, `abi-encode`, …).

### C. `concat` value resolver — **built**

An explicit string-join, chosen over implicit whole-string interpolation. Implicit interpolation risks mangling values that legitimately contain `{{` and forces the resolver to guess intent; an explicit `concat` is unambiguous and self-documenting. Solves URL/path templating.

### D. `safe-exec-transaction` std template — **built**

Encapsulates the Shape-1 relay on top of the primitives + existing `json-request`/`read-json`/`abi-encode`/`send-transaction`. See "The template" section below for the final shape and the one design decision that fell out of it (a single `signatures` argument rather than baking the file-vs-service choice into the template).

## What was built

Two pure value resolvers, wired into the existing `ValueResolver` union and the resolver dispatch switch, matching the surrounding code's style and error conventions.

### `read-file`

```yaml
signatures:
type: "read-file"
arguments:
path: "signatures.hex" # relative to the job/template directory
encoding: "hex" # "utf8" (default) | "hex" | "json"
```

- `utf8` — returns text as-is with a single trailing newline trimmed (so file contents compare cleanly against inline strings).
- `hex` — validates and normalizes to a `0x`-prefixed lowercase hex string (accepts input with or without `0x`).
- `json` — parses the file and returns the value; composes with `read-json` to pull a nested field.

### `concat`

```yaml
url:
type: "concat"
arguments:
values:
- "https://safe-transaction-mainnet.safe.global/api/v1/multisig-transactions/"
- "{{safe-tx-hash}}"
- "/"
separator: "" # optional, default "" (direct concatenation)
```

Each part is resolved then coerced to a string (numbers/booleans allowed; objects/null/undefined rejected with a clear error) and joined by `separator`.

### Files touched

- `src/lib/types/values.ts` — `ReadFileValue`, `ConcatValue` interfaces + added to the `ValueResolver` union.
- `src/lib/core/resolver.ts` — imports, two dispatch cases, `resolveReadFile` / `resolveConcat` implementations.
- `src/lib/core/context.ts` — optional `projectRoot` constructor arg + `getProjectRoot()` (used to confine reads). Backwards-compatible: existing 5-arg callers and test mocks are unaffected.
- `src/lib/deployer.ts` — passes `options.projectRoot` into the context.
- `src/lib/std/templates/safe-exec-transaction.yaml` — the Shape-1 relay std template.
- `src/lib/core/__tests__/resolver.spec.ts` — `read-file` (11 cases) and `concat` (6 cases) describe blocks.
- `src/lib/core/__tests__/safe-exec-transaction.spec.ts` — parses the shipped template and runs it end-to-end (2 cases).
- `README.md` — `read-file` and `concat` sections under Value Resolvers, plus a `safe-exec-transaction` section under Standard Templates.

## Security considerations

`read-file` reads from disk driven by YAML, so path handling is the whole risk surface:

- **No absolute paths.** Rejected outright — a job must not name `/etc/...` or a home-dir key file.
- **Confined to the project root.** The path is resolved against the job/template directory, then checked with `path.relative(projectRoot, resolved)`; anything that starts with `..` or is absolute (i.e. escapes the root) is refused. `../../secrets` cannot climb out. (When `projectRoot` is unknown — e.g. a bare unit-test context — the resolver still rejects absolute paths and resolves relative to the context dir; the deployer always supplies `projectRoot` in real runs.)
- **No secret auto-discovery.** The resolver only reads the exact file named. It never scans, globs, or reads `.env`/keystores implicitly. Secrets (the deployer key) continue to arrive via env/CLI, never through this path.
- **Blobs are meant to be gitignorable.** Operational payloads (signatures) live in their own file the operator can `.gitignore`, keeping large per-run data out of git and out of `constants`.

`concat` has no I/O and no injection surface beyond producing a string; object parts are rejected so a misresolved reference fails loudly instead of emitting `[object Object]`.

## How live-contracts consumes it

**Before** (what we actually shipped as a workaround): the endpoint URL had to be a single full-URL constant because it couldn't be templated, and the packed signatures came from a run-time `json-request` to the Safe Transaction Service because there was nowhere good to store them:

```yaml
constants:
# Whole URL as one opaque constant — the tx hash could not be interpolated.
safe_tx_url: "https://safe-transaction-mainnet.safe.global/api/v1/multisig-transactions/0xabc.../"

actions:
- name: fetch-sigs
type: "json-request"
arguments:
url: "{{safe_tx_url}}"
- name: relay
type: "send-transaction"
arguments:
to: "{{safe_address}}"
data:
type: "abi-encode"
arguments:
signature: "execTransaction(address,uint256,bytes,uint8,uint256,uint256,uint256,address,address,bytes)"
values: [ ..., { type: "read-json", arguments: { json: "{{fetch-sigs.response}}", path: "signatures" } } ]
```

**After** — the URL is templated with `concat`, and the collected signatures live in a gitignorable file read with `read-file` (offline, deterministic, no dependency on the tx-service being reachable at relay time):

```yaml
constants:
safe_tx_service: "https://safe-transaction-mainnet.safe.global/api/v1"

actions:
# Option 1: still fetch from the service, but build the URL with concat
- name: fetch-sigs
type: "json-request"
arguments:
url:
type: "concat"
values:
- "{{safe_tx_service}}"
- "/multisig-transactions/"
- "{{safe-tx-hash}}"
- "/"

# Option 2: read pre-collected packed signatures straight from a file
- name: relay
type: "send-transaction"
arguments:
to: "{{safe_address}}"
data:
type: "abi-encode"
arguments:
signature: "execTransaction(address,uint256,bytes,uint8,uint256,uint256,uint256,address,address,bytes)"
values:
- # ...to, value, data, operation, safeTxGas, baseGas, gasPrice, gasToken, refundReceiver...
- { type: "read-file", arguments: { path: "signatures.hex", encoding: "hex" } }
```

## The template: `safe-exec-transaction`

Shipped at `src/lib/std/templates/safe-exec-transaction.yaml`. It takes the Safe address, the ten `execTransaction` parameters (`to`, `value`, `data`, `operation`, `safeTxGas`, `baseGas`, `gasPrice`, `gasToken`, `refundReceiver`) and the packed owner `signatures`, then:

1. `abi-encode`s `execTransaction(address,uint256,bytes,uint8,uint256,uint256,uint256,address,address,bytes)` with those values.
2. `send-transaction`s that calldata to the Safe (the configured EOA is the relayer).
3. Exposes the outer tx hash as the template output `hash`.

```yaml
- name: "relay"
template: "safe-exec-transaction"
arguments:
safe: "{{safe_address}}"
to: "{{target}}"
data: "{{inner_calldata}}"
operation: "0"
signatures:
type: "read-file" # offline: a gitignorable signatures file
arguments: { path: "signatures.hex", encoding: "hex" }
```

For the hosted flow, the caller fetches the signatures with a sibling `json-request` (URL built with `concat`) and passes `read-json` of the response's `signatures` field — see README for that variant.

**Design decision — one `signatures` argument, not a file-vs-service switch inside the template.** My first cut tried to make the template pick the source (fetch from the tx-service *or* read the file). That can't be expressed cleanly today: catapult has no conditional/coalesce value resolver, so a `read-file` with an empty path just errors — there's no "use A else B". Forcing it would have meant either a new `conditional` resolver (out of scope) or an ugly always-fetch-then-maybe-ignore. Instead the template takes a single resolved `signatures` value and the *caller* chooses the source with the resolver they want (`read-file` or `json-request`+`read-json`). This keeps the template single-responsibility (assemble + broadcast), mirrors how `min-balance` takes an already-resolved `balance`, and makes both source patterns first-class in the docs rather than one being privileged.

**No post-execution skip condition in the template.** The desired state of a Safe relay is the effect of the *inner* call, which only the caller knows, so the template must not impose a generic post-check (catapult's template `skip_condition` is post-checked and would fail spuriously). The caller wraps the job with `skip_if` (a pure pre-gate that already exists for exactly the multisig-payload case — see `Job.skip_if` and `roadmap-thinking.md` §2) observing the on-chain effect, giving the same idempotent, re-runnable convergence as deployments.

Open design point: this overlaps with the `propose-transaction` + `pending`-state model in `roadmap-thinking.md` §2 — a fuller lifecycle where catapult also *collects* signatures. `safe-exec-transaction` covers the "signatures already gathered, broadcast them" half; the primitives (`read-file`, `concat`) are useful regardless of how the collection half lands.

## Test results

`resolver.spec.ts`: 216/216 pass (includes the 11 new `read-file` + 6 new `concat` cases). `safe-exec-transaction.spec.ts`: 2/2 pass — it parses the actually-shipped YAML, runs the template through the engine, and asserts the broadcast transaction goes to the Safe with `execTransaction` calldata that matches an independently-computed ethers encoding (an EOA stands in for the Safe so no real Safe is needed). Build is clean (`pnpm build`), lint has 0 errors (only the pre-existing repo-wide `no-explicit-any` warnings).

The full suite has 4–5 pre-existing failures in `engine.spec.ts` (`send-signed-transaction`, `test-nicks-method`) — these fail identically on a clean `origin/master` tree and are environmental: the local node at `127.0.0.1:8545` is a Polygon mainnet fork (chainId `0x89`), not a clean instant-mining anvil with an unlocked funded account. They are unrelated to this change, which adds only two pure value resolvers and one optional constructor argument.
Loading
Loading