Skip to content

feat: add embedded ms3t S3 listener backed by Forge #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
frrist wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

feat: add embedded ms3t S3 listener backed by Forge #34

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
79134c2
feat: add embedded ms3t S3 listener backed by Forge
frrist Apr 29, 2026
6e643a9
feat: lsm tree style buffering/aggregation
frrist May 1, 2026
afb7f97
docs: update architectural.md
frrist May 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions config.example.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,3 +108,43 @@ storage:
log:
# Log level: debug, info, warn, error
level: "info"

# ms3t — embedded S3-compatible HTTP server backed by an MST. When
# enabled, sprue exposes an S3 listener alongside its UCAN endpoint;
# objects are stored as content-addressed blocks in a local SQLite
# blockstore and (optionally) shipped to a Forge stack via guppy.
ms3t:
enabled: false
# host:port for the S3 listener. Path-style addressing only; clients
# must set forcePathStyle=true and disable streaming uploads
# (request_checksum_calculation=when_required for AWS CLI v2.23+).
addr: ":9000"
# ms3t persists its SQLite database and (when forge is disabled)
# CAR files under this directory. Created if missing.
data_dir: "./ms3t-data"
# Body chunk size in bytes for new objects. Default 1 MiB.
chunk_size: 1048576
# Uploader: flush a CAR after this many buffered bytes. Default 64 MiB.
batch_bytes: 67108864
# Uploader: flush a CAR after this idle duration. Default 5s.
batch_age: "5s"

forge:
# When true, every batched CAR is shipped to piri through
# sprue's routing, piriclient, and indexerclient — no
# UCAN-over-HTTP loopback. ms3t reuses sprue's identity as the
# signer; no separate principal or delegation file is needed.
# When false, CARs are written to data_dir/cars only.
enabled: false
# Path to ms3t's persisted space keypair. Generated on first
# run if missing. ms3t is the root UCAN authority over its own
# space, which lets self-issued retrieval delegations validate
# against piri's auth on index-blob fetches.
# Defaults to <data_dir>/space.key.
space_key_file: ""
# When true, all block reads (MST nodes, manifests, body
# chunks) go through indexer→piri instead of a local SQLite
# cache, AND writes go synchronously to Forge (Batched is
# bypassed). Closes the read-after-write race; raises per-PUT
# latency to the Forge round trip cost. Requires enabled = true.
no_cache: false
42 changes: 27 additions & 15 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ require (
github.com/ipfs/go-log/v2 v2.9.0
github.com/ipld/go-ipld-prime v0.21.1-0.20240917223228-6148356a4c2e
github.com/jackc/pgx/v5 v5.9.1
github.com/labstack/echo/v4 v4.14.0
github.com/labstack/echo/v4 v4.15.0
github.com/multiformats/go-multiaddr v0.16.0
github.com/multiformats/go-multibase v0.2.0
github.com/multiformats/go-multihash v0.2.3
Expand All @@ -23,29 +23,42 @@ require (
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
github.com/storacha/go-libstoracha v0.7.5
github.com/storacha/go-ucanto v0.7.2
github.com/storacha/go-ucanto v0.8.2
github.com/storacha/guppy v0.7.0
github.com/stretchr/testify v1.11.1
github.com/testcontainers/testcontainers-go v0.42.0
github.com/testcontainers/testcontainers-go/modules/dynamodb v0.41.0
github.com/testcontainers/testcontainers-go/modules/minio v0.40.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0
go.uber.org/fx v1.24.0
go.uber.org/zap v1.27.0
go.uber.org/zap/exp v0.3.0
modernc.org/sqlite v1.46.1
)

require (
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/ipfs/boxo v0.30.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/mfridman/interpolate v0.0.2 // indirect
github.com/moby/moby/api v1.54.1 // indirect
github.com/moby/moby/client v0.4.0 // indirect
github.com/moby/sys/atomicwriter v0.1.0 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/sethvargo/go-retry v0.3.0 // indirect
github.com/storacha/indexing-service v1.12.2 // indirect
github.com/stretchr/objx v0.5.3 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0 // indirect
go.opentelemetry.io/otel/sdk v1.43.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.43.0 // indirect
golang.org/x/sync v0.20.0 // indirect
modernc.org/libc v1.68.0 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
)

require (
Expand Down Expand Up @@ -86,41 +99,41 @@ require (
github.com/ebitengine/purego v0.10.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/filecoin-project/go-data-segment v0.0.1 // indirect
github.com/filecoin-project/go-fil-commcid v0.2.0 // indirect
github.com/filecoin-project/go-fil-commcid v0.3.1 // indirect
github.com/filecoin-project/go-fil-commp-hashhash v0.2.0 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/ipfs/bbloom v0.0.4 // indirect
github.com/ipfs/go-block-format v0.2.0 // indirect
github.com/ipfs/go-block-format v0.2.1
github.com/ipfs/go-blockservice v0.5.2 // indirect
github.com/ipfs/go-datastore v0.9.0 // indirect
github.com/ipfs/go-ipfs-blockstore v1.3.1 // indirect
github.com/ipfs/go-ipfs-ds-help v1.1.1 // indirect
github.com/ipfs/go-ipfs-exchange-interface v0.2.1 // indirect
github.com/ipfs/go-ipfs-util v0.0.3 // indirect
github.com/ipfs/go-ipld-cbor v0.1.0 // indirect
github.com/ipfs/go-ipld-format v0.6.0 // indirect
github.com/ipfs/go-ipld-cbor v0.2.0
github.com/ipfs/go-ipld-format v0.6.1 // indirect
github.com/ipfs/go-ipld-legacy v0.2.1 // indirect
github.com/ipfs/go-log v1.0.5 // indirect
github.com/ipfs/go-merkledag v0.11.0 // indirect
github.com/ipfs/go-metrics-interface v0.0.1 // indirect
github.com/ipfs/go-metrics-interface v0.3.0 // indirect
github.com/ipfs/go-verifcid v0.0.3 // indirect
github.com/ipld/go-car v0.6.2 // indirect
github.com/ipld/go-codec-dagpb v1.6.0 // indirect
github.com/ipld/go-codec-dagpb v1.7.0 // indirect
github.com/ipni/go-libipni v0.6.18 // indirect
github.com/klauspost/compress v1.18.5 // indirect
github.com/klauspost/cpuid/v2 v2.2.10 // indirect
github.com/labstack/gommon v0.4.2 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
github.com/libp2p/go-libp2p v0.41.1 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/lufia/plan9stats v0.0.0-20240513124658-fba389f38bae // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
Expand All @@ -136,14 +149,13 @@ require (
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/multiformats/go-base32 v0.1.0 // indirect
github.com/multiformats/go-base36 v0.2.0 // indirect
github.com/multiformats/go-multicodec v0.9.1 // indirect
github.com/multiformats/go-multicodec v0.9.2
github.com/multiformats/go-varint v0.1.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/pion/datachannel v1.6.0 // indirect
github.com/pion/logging v0.2.4 // indirect
github.com/pion/sctp v1.9.2 // indirect
github.com/pion/webrtc/v4 v4.2.9 // indirect
github.com/pkg/errors v0.9.1 // indirect
Expand All @@ -164,7 +176,7 @@ require (
github.com/ucan-wg/go-ucan v0.0.0-20240916120445-37f52863156c // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasttemplate v1.2.2 // indirect
github.com/whyrusleeping/cbor-gen v0.3.1 // indirect
github.com/whyrusleeping/cbor-gen v0.3.1
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 // indirect
Expand All @@ -176,12 +188,12 @@ require (
go.uber.org/multierr v1.11.0 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.49.0 // indirect
golang.org/x/exp v0.0.0-20260218203240-3dfff04db8fa // indirect
golang.org/x/exp v0.0.0-20260312153236-7ab1446f8b90 // indirect
golang.org/x/net v0.52.0 // indirect
golang.org/x/sys v0.42.0 // indirect
golang.org/x/text v0.35.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 // indirect
google.golang.org/protobuf v1.36.11 // indirect
Expand Down
Loading
Loading