feat(network): durable eRPC config via erpc set|status|reset

CLAUDE.md

@@ -227,7 +227,7 @@ obol sell delete ollama-gated -n llm
 
 ## RPC Gateway
 
-`obol network add|remove|status` manages remote RPCs via eRPC ConfigMap. Default: read-only (blocks `eth_sendRawTransaction`, `eth_sendTransaction`). `--allow-writes` flips readOnly → route `eth_sendRawTransaction` to the user-chosen upstream via eRPC per-method selection policy. `--endpoint` adds a custom RPC directly (skips ChainList). Key functions in `internal/network/rpc.go`: `AddPublicRPCs()` (ChainList), `AddCustomRPC()`, `ListRPCNetworks()`. Record-on-write: add/remove update `$CONFIG_DIR/rpc/recorded-upstreams.yaml` (`internal/network/record.go`); `obol stack up` replays via `ReconcileRecordedRPCs()` so remote RPCs survive cluster recreation. Local-node upstreams are NOT recorded (`network sync` re-registers them).
+`obol network add|remove|status` manages remote RPCs via eRPC ConfigMap. Default: read-only (blocks `eth_sendRawTransaction`, `eth_sendTransaction`). `--allow-writes` flips readOnly → route `eth_sendRawTransaction` to the user-chosen upstream via eRPC per-method selection policy. `--endpoint` adds a custom RPC directly (skips ChainList). Key functions in `internal/network/rpc.go`: `AddPublicRPCs()` (ChainList), `AddCustomRPC()`, `ListRPCNetworks()`. Record-on-write: add/remove update `$CONFIG_DIR/rpc/recorded-upstreams.yaml` (`internal/network/record.go`); `obol stack up` replays via `ReconcileRecordedRPCs()` so remote RPCs survive cluster recreation. Local-node upstreams are NOT recorded (`network sync` re-registers them). **Durable eRPC operator config** (so local baskets are not lost on stack up): `obol network erpc set|status|reset` persists `$CONFIG_DIR/rpc/erpc-overlay.yaml` (merge-by-id into the live ConfigMap; not a full replace) and re-applies after recorded remotes on stack up via `ReconcileERPCOverlay()` (`internal/network/overlay.go`, #763). Example: `examples/erpc-overlay-hyperevm.yaml`.
 
 ## Network Management
 
@@ -242,7 +242,7 @@ Two-stage templating: `values.yaml.gotmpl` annotated with `@enum`/`@default`/`@d
 | Command | Action |
 |---------|--------|
 | `obol stack init` | Generate cluster ID (petname), resolve absolute paths, write k3d.yaml, copy infrastructure |
-| `obol stack up` | `k3d cluster create`, export kubeconfig, k3s auto-applies manifests, auto-configures LiteLLM with Ollama models (preserves Ollama modified-time order; `:cloud` aliases demoted behind local chat models; embedding-only models last; warns + suggests `ollama pull qwen3.5:4b` when empty or all-`:cloud`), re-imposes recorded model config (`model.ReconcileRecorded` — operator's `obol model` choices win over auto-detect), deploys default Hermes agent, applies agent capabilities, starts persistent Cloudflare tunnel, then replays recorded state: RPC upstreams (`network.ReconcileRecordedRPCs`) → Agent CRs (`agentcrd.ResumeAll`) → sell offers (`resumeSellOffers`; agent-type offers ride the sell-http store). Guard: `cmd/obol/stackup_resume_guard_test.go` |
+| `obol stack up` | `k3d cluster create`, export kubeconfig, k3s auto-applies manifests, auto-configures LiteLLM with Ollama models (preserves Ollama modified-time order; `:cloud` aliases demoted behind local chat models; embedding-only models last; warns + suggests `ollama pull qwen3.5:4b` when empty or all-`:cloud`), re-imposes recorded model config (`model.ReconcileRecorded` — operator's `obol model` choices win over auto-detect), deploys default Hermes agent, applies agent capabilities, starts persistent Cloudflare tunnel, then replays recorded state: RPC upstreams (`network.ReconcileRecordedRPCs`) → eRPC operator overlay (`network.ReconcileERPCOverlay`) → Agent CRs (`agentcrd.ResumeAll`) → sell offers (`resumeSellOffers`; agent-type offers ride the sell-http store). Guard: `cmd/obol/stackup_resume_guard_test.go` |
 | `obol stack down` | `k3d cluster stop` (delete fallback; preserves config + data) |
 | `obol stack purge [-f]` | Delete config; `-f` also deletes root-owned PVCs; `-f` offers a full `stack export` first (fallback: OpenClaw wallet prompt) |
 | `obol stack export` | Full backup archive: config dir (minus kubeconfig/defaults), agent data dirs (brains + keystores, deployments quiesced for consistency), encrypted wallet backups, etcd-drift resources (Agent CRs, ServiceOffers, LiteLLM/eRPC CMs). `internal/stackbackup/` |

cmd/obol/main.go

@@ -212,6 +212,11 @@ GLOBAL OPTIONS:{{template "visibleFlagTemplate" .}}{{end}}
 							// Replay recorded remote RPC upstreams into the
 							// (possibly fresh) eRPC ConfigMap. Best-effort.
 							network.ReconcileRecordedRPCs(cfg, u)
+							// Re-apply durable multi-upstream eRPC operator
+							// overlays (baskets/scoring/rate-limits) AFTER
+							// simple recorded remotes. Best-effort.
+							// See ObolNetwork/obol-stack#763.
+							network.ReconcileERPCOverlay(cfg, u)
 							// Re-apply recorded Agent CRs BEFORE sell offers:
 							// agent-backed ServiceOffers resolve agent.ref and
 							// would dangle without their Agent. Best-effort.

cmd/obol/network.go

@@ -76,6 +76,7 @@ func networkCommand(cfg *config.Config) *cli.Command {
 			networkAddCommand(cfg),
 			networkRemoveCommand(cfg),
 			networkStatusCommand(cfg),
+			networkERPCCommand(cfg),
 		},
 	}
 }
@@ -534,6 +535,81 @@ func networkStatusCommand(cfg *config.Config) *cli.Command {
 	}
 }
 
+// network erpc — durable operator eRPC config that survives stack up (#763).
+// Verbs match sell-info set/reset (host-side intent) + status.
+// ---------------------------------------------------------------------------
+
+func networkERPCCommand(cfg *config.Config) *cli.Command {
+	return &cli.Command{
+		Name:  "erpc",
+		Usage: "Manage durable operator eRPC config (host-side; re-applied on stack up so local baskets are not lost)",
+		Commands: []*cli.Command{
+			{
+				Name:  "set",
+				Usage: "Set durable eRPC config from a YAML file and merge it into the live ConfigMap",
+				Flags: []cli.Flag{
+					&cli.StringFlag{
+						Name:     "file",
+						Aliases:  []string{"f"},
+						Usage:    "YAML fragment: networks, upstreams, rateLimiters, cachePoliciesAdd (saved to $CONFIG_DIR/rpc/erpc-overlay.yaml)",
+						Required: true,
+					},
+				},
+				Action: func(_ context.Context, cmd *cli.Command) error {
+					return network.SetERPC(cfg, getUI(cmd), cmd.String("file"))
+				},
+			},
+			{
+				Name:  "status",
+				Usage: "Show the durable eRPC config on disk",
+				Action: func(_ context.Context, cmd *cli.Command) error {
+					u := getUI(cmd)
+					st, err := network.StatusERPC(cfg)
+					if err != nil {
+						return err
+					}
+					u.Printf("eRPC operator config\n")
+					u.Printf("====================\n\n")
+					u.Printf("Path: %s\n", st.Path)
+					if !st.Present {
+						u.Info("Status: not set")
+						u.Info("Set with: obol network erpc set -f <file.yaml>")
+						return nil
+					}
+					u.Printf("Status: set (v%d, hash %s)\n", st.Version, st.ContentHash)
+					u.Printf("Networks (%d):\n", st.NetworkCount)
+					for _, k := range st.NetworkKeys {
+						u.Printf("  - %s\n", k)
+					}
+					if st.NetworkCount == 0 {
+						u.Info("  (none)")
+					}
+					u.Printf("Upstreams (%d):\n", st.UpstreamCount)
+					for _, id := range st.UpstreamIDs {
+						u.Printf("  - %s\n", id)
+					}
+					if st.UpstreamCount == 0 {
+						u.Info("  (none)")
+					}
+					u.Printf("Rate-limit budgets: %d\n", st.BudgetCount)
+					u.Printf("Cache policies to add: %d\n", st.CachePolicyAdd)
+					return nil
+				},
+			},
+			{
+				Name:  "reset",
+				Usage: "Reset durable eRPC config (strip from live ConfigMap and delete host file)",
+				Action: func(_ context.Context, cmd *cli.Command) error {
+					return network.ResetERPC(cfg, getUI(cmd))
+				},
+			},
+		},
+		Action: func(ctx context.Context, cmd *cli.Command) error {
+			return cli.ShowSubcommandHelp(cmd)
+		},
+	}
+}
+
 // chainIDToName returns a human-readable name for a chain ID.
 func chainIDToName(chainID int) string {
 	names := map[int]string{
@@ -550,6 +626,7 @@ func chainIDToName(chainID int) string {
 		43114:    "Avalanche",
 		59144:    "Linea",
 		84532:    "Base Sepolia",
+		999:      "HyperEVM",
 		534352:   "Scroll",
 		560048:   "Hoodi",
 		11155111: "Sepolia",

cmd/obol/stackup_resume_guard_test.go

@@ -21,13 +21,17 @@ func TestStackUpAction_ReplaysRecordedState(t *testing.T) {
 
 	upIdx := strings.Index(body, "stack.Up(cfg")
 	rpcIdx := strings.Index(body, "network.ReconcileRecordedRPCs(")
+	overlayIdx := strings.Index(body, "network.ReconcileERPCOverlay(")
 	agentsIdx := strings.Index(body, "agentcrd.ResumeAll(")
 	appsIdx := strings.Index(body, "app.ResumeAll(")
 	offersIdx := strings.Index(body, "resumeSellOffers(")
 
 	if rpcIdx < 0 {
 		t.Fatal("cmd/obol/main.go must call network.ReconcileRecordedRPCs — without it recorded remote RPCs never reach a freshly-recreated cluster")
 	}
+	if overlayIdx < 0 {
+		t.Fatal("cmd/obol/main.go must call network.ReconcileERPCOverlay — without it durable eRPC baskets (e.g. HyperEVM) never re-apply after stack up (#763)")
+	}
 	if agentsIdx < 0 {
 		t.Fatal("cmd/obol/main.go must call agentcrd.ResumeAll — without it recorded Agent CRs never reach a freshly-recreated cluster")
 	}
@@ -37,9 +41,12 @@ func TestStackUpAction_ReplaysRecordedState(t *testing.T) {
 	if upIdx < 0 || offersIdx < 0 {
 		t.Fatalf("expected stack.Up and resumeSellOffers in main.go; upIdx=%d offersIdx=%d", upIdx, offersIdx)
 	}
-	if rpcIdx < upIdx || agentsIdx < upIdx || appsIdx < upIdx {
+	if rpcIdx < upIdx || overlayIdx < upIdx || agentsIdx < upIdx || appsIdx < upIdx {
 		t.Error("recorded-state replay must run AFTER stack.Up — before it there is no kubeconfig/cluster")
 	}
+	if overlayIdx < rpcIdx {
+		t.Error("ReconcileERPCOverlay must run AFTER ReconcileRecordedRPCs — overlay merges onto base+recorded remotes")
+	}
 	if agentsIdx > offersIdx {
 		t.Error("agentcrd.ResumeAll must run BEFORE resumeSellOffers — agent-backed ServiceOffers need their Agent CR first")
 	}

examples/erpc-overlay-hyperevm.yaml

@@ -0,0 +1,103 @@
+# Example eRPC operator overlay — HyperEVM (chain 999) basket.
+#
+# Durable operator eRPC config so local baskets survive `obol stack up`. Set with:
+#
+#   obol network erpc set -f examples/erpc-overlay-hyperevm.yaml
+#   obol network erpc status
+#   
+# Host copy lives at $OBOL_CONFIG_DIR/rpc/erpc-overlay.yaml (0600) and is
+# re-merged after recorded remotes on every stack up (see #763).
+#
+# Edit the local endpoint for your co-located hl-node before applying.
+
+version: 1
+
+networks:
+  - alias: hyperevm
+    architecture: evm
+    evm:
+      chainId: 999
+    failsafe:
+      timeout:
+        duration: 10s
+      retry:
+        maxAttempts: 3
+        delay: 100ms
+      hedge:
+        delay: 150ms
+        maxCount: 1
+
+upstreams:
+  # Prefer a co-located non-validator HyperEVM RPC when present.
+  - id: local-hl-node
+    endpoint: http://192.168.50.21:3001/evm
+    evm:
+      chainId: 999
+    routing:
+      scoreMultipliers:
+        - network: "*"
+          method: "*"
+          overall: 5.0
+
+  - id: hyperevm-official
+    endpoint: https://rpc.hyperliquid.xyz/evm
+    evm:
+      chainId: 999
+    tags: ["tier:fallback"]
+    rateLimitBudget: hyperevm-official
+    routing:
+      scoreMultipliers:
+        - network: "*"
+          method: "*"
+          overall: 0.3
+
+  - id: hyperevm-drpc
+    endpoint: https://hyperliquid.drpc.org
+    evm:
+      chainId: 999
+    tags: ["tier:fallback"]
+    rateLimitBudget: hyperevm-drpc
+    routing:
+      scoreMultipliers:
+        - network: "*"
+          method: "*"
+          overall: 0.3
+
+  - id: hyperevm-onfinality
+    endpoint: https://hyperliquid.api.onfinality.io/evm/public
+    evm:
+      chainId: 999
+    tags: ["tier:fallback"]
+    rateLimitBudget: hyperevm-onfinality
+    routing:
+      scoreMultipliers:
+        - network: "*"
+          method: "*"
+          overall: 0.3
+
+rateLimiters:
+  budgets:
+    - id: hyperevm-official
+      rules:
+        - method: "*"
+          maxCount: 100
+          period: second
+    - id: hyperevm-drpc
+      rules:
+        - method: "*"
+          maxCount: 30
+          period: second
+    - id: hyperevm-onfinality
+      rules:
+        - method: "*"
+          maxCount: 25
+          period: second
+
+# eth_call @ latest (HyperCore precompiles) needs realtime caching; the chart
+# default finality:unfinalized can serve stale precompile reads under burst.
+cachePoliciesAdd:
+  - network: "*"
+    method: eth_call
+    finality: realtime
+    connector: memory-cache
+    ttl: 2s