Skip to content

Broadcast to every server at once, and stop leaking the NOWNodes key - #458

Merged
peachbits merged 3 commits into
masterfrom
matthew/broadcast-all-servers
Sep 8, 2026
Merged

peachbits merged 3 commits into
masterfrom
matthew/broadcast-all-servers

Conversation

@peachbits

@peachbits peachbits commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

CHANGELOG

Does this branch warrant an entry to the CHANGELOG?

  • Yes
  • No

Dependencies

none

Description

Engine half of the "send failed in the UI but the money moved" incident. Asana: https://app.asana.com/1/9976422036640/project/1213880789473005/task/1217135300337949

GUI companion (independent, neither blocks the other): EdgeApp/edge-react-gui#6190

Three commits, review fixups squashed in. The first commit stands alone and can be reviewed, or landed, without the rest.


1. Send the NOWNodes API key only to NOWNodes servers

ServerConfig had a single type, blockbook-nownode, so when Edge's own Blockbook hosts were added in Dec 2024 (9962d7e) they were filed under it alongside the real NOWNodes entries. Every HTTP broadcast therefore sent the NOWNodes key to Edge's servers too.

They are not the same kind of server:

GET https://btc-wusa1.edge.app/api/v2   -> 200, no key needed
GET https://btcbook.nownodes.io/api/v2  -> 401 Unknown API_key

The type becomes blockbook | blockbook-nownode. The ten edge.app HTTP entries are now plain blockbook, the nownodes.io entries are unchanged, and the key is attached per target instead of to the whole batch.

This also fixes the no-key case. A missing nowNodesApiKey previously rejected the entire HTTP path with "Missing connection key for fallback servers", even though the public servers need no key. NOWNodes targets are now skipped with a warning and the public servers still carry the broadcast.

2. Reject socket requests when the socket fails

Found in review. promisifyWsMessage wraps each request in a Deferred that was only ever resolved from inside its own generator. The socket reports every failure (error response, 30s expiry, socket close) by throwing into that generator, which had no try/catch, so the deferred stayed pending forever and the caller hung. That applied to fetchInfo, fetchAddress, fetchAddressUtxos, fetchTransaction and ping, not only broadcastTx. The generator now rejects its deferred on throw.

This has to precede the fan-out: once every socket attempt is counted, an all-fail broadcast with a cached socket could never settle without it.

3. Broadcast to every server at once

The HTTP servers were used only when no WebSocket reported itself connected. A socket that looks connected but never answers sendTransaction therefore failed the whole broadcast, after the 30 second request timeout, without a single HTTP attempt being made. That is the shape of the incident behind this task.

The gate is worse than it looks: the NOWNodes WebSocket URI is docked 400 points at score load (ServerScores.serverScoresLoad penalises any URI carrying key params) and so rarely wins one of the two connection slots. HTTP is the realistic route to those servers, and it was reachable only when every socket was visibly down.

broadcastTx now fires in two waves. The first goes out immediately to every cached blockbook (sockets still connecting included, since a queued request transmits as soon as the socket opens) and to every blockbook HTTP server, which are Edge's own. The second wave, the blockbook-nownode HTTP servers, fires after NOWNODES_BROADCAST_DELAY_MS (2s), or as soon as every first-wave attempt has failed, whichever comes first. The first success resolves; it rejects only once every attempt has failed, logging each failure against its server. Every attempt is bounded by BROADCAST_ATTEMPT_TIMEOUT_MS (30s, mirroring the socket request expiry), so a server that accepts the connection and never answers cannot hold the broadcast open; the socket layer already expired its own requests, HTTP had no bound until Bugbot pointed it out.

The two waves are the answer to the review thread on unconditional HTTP. NOWNodes is a third party, so it only sees a transaction when Edge's own infrastructure has not already carried it. In the incident every socket was down and the HTTP fallback that saved the broadcast included Edge's own servers, so the first wave would very likely have succeeded on its own; the delay costs anything only in the double-failure case, and two seconds against the old thirty-second hang is nothing. attempts counts both waves up front, so an all-fail first wave cannot reject before the second has run.

Testing

test/common/utxobased/engine/ServerStates.spec.ts (new) and Blockbook.spec.ts:

  • broadcasting over every HTTP server when no sockets are cached
  • first success wins when one server fails and another succeeds
  • rejecting only after every attempt fails
  • rejecting immediately when there is nothing to broadcast to
  • NOWNodes is not contacted at all when Edge's own servers carry the broadcast (waits out the delay to prove the timer was cancelled)
  • NOWNodes fires at once, not after the delay, when Edge's servers fail fast, with the api-key header on the NOWNodes request and not on the public one
  • skipping NOWNodes but still using public servers when no key is configured
  • rejecting when only NOWNodes servers exist and no key is configured
  • fetchInfo rejects when the server answers with an error (commit 2)
  • a real WebSocket that accepts sendTransaction and never answers: NOWNodes is held back for the delay, then the broadcast resolves through it well inside the 30s socket timeout
  • a socket that refuses alongside failing Edge servers: NOWNodes fires at once and the broadcast resolves in under the delay
  • an HTTP server that accepts and never answers times out and the broadcast rejects
  • a refusing socket plus a hung HTTP server rejects instead of hanging, Bugbot's case
  • the default attempt timeout equals the socket layer's 30 seconds
  • a socket that refuses sendTransaction with no HTTP server rejects instead of hanging
  • a socket that refuses alongside a failing HTTP server rejects instead of hanging

Full suite: 1261 passing at commit 1, 1262 at commit 2, 1270 at HEAD (re-run after the squash; tree identical to the reviewed head). tsc and eslint clean.

Note for reviewers

This deliberately does not include the broadcast-failure classification from #455. Per the direction agreed for this task, the app now hard-fails an ambiguous broadcast and says so, rather than trying to infer whether the transaction landed. The saveTx "No addresses to process" fix from #455 is still valid and is not in this branch.


Note

High Risk
Changes core transaction broadcast paths, third-party exposure timing, and API key handling—directly affects whether sends succeed or fail and whether txs leak to NOWNodes early.

Overview
Fixes UTXO sends that could fail in the UI while the transaction still propagated, by changing how broadcastTx reaches Blockbook.

broadcastTx now fans out in two waves. The first wave immediately hits every cached Blockbook WebSocket (including sockets still connecting) and every Edge blockbook HTTP host. NOWNodes blockbook-nownode HTTP targets run in a second wave after 2 seconds, or immediately once the first wave has all failed—whichever comes first. The first success wins; the call rejects only when every attempt fails or hits a 30s per-attempt timeout (HTTP previously could hang indefinitely). This replaces the old rule that skipped HTTP whenever any socket looked connected, which let one silent socket block fallback entirely.

Server typing and credentials: ServerConfig is split into blockbook (no headers) vs blockbook-nownode (api-key only on NOWNodes). Edge *.edge.app HTTP entries across coin configs are reclassified as blockbook, so the NOWNodes key is no longer sent to Edge. Without a key, NOWNodes servers are skipped with a warning while public Edge HTTP still broadcasts.

Socket hang fix: Blockbook.promisifyWsMessage now rejects when the socket throws (errors, timeout, close), so broadcast and other WS calls settle instead of pending forever—required for counting all broadcast attempts.

New ServerStates.broadcastTx tests cover wave timing, header placement, timeouts, and socket+HTTP combinations; Blockbook.spec adds error-response rejection.

Reviewed by Cursor Bugbot for commit b1d77cf. Bugbot is set up for automated code reviews on this repo. Configure here.

`ServerConfig` had a single type, `blockbook-nownode`, so when Edge's own
Blockbook hosts were added in Dec 2024 (9962d7e) they were filed under it
alongside the real NOWNodes entries. Every HTTP broadcast therefore sent
the NOWNodes key to Edge's servers as well.

They are not the same kind of server:

    GET https://btc-wusa1.edge.app/api/v2   -> 200, no key needed
    GET https://btcbook.nownodes.io/api/v2  -> 401 Unknown API_key

The type becomes `blockbook | blockbook-nownode`. The ten edge.app HTTP
entries are now plain `blockbook`, the nownodes.io entries are unchanged,
and the key is attached per target rather than to the whole batch.

This also fixes the no-key case. A missing `nowNodesApiKey` previously
rejected the entire HTTP path with "Missing connection key for fallback
servers", even though the public servers need no key. NOWNodes targets
are now skipped with a warning and the public servers still carry the
broadcast; the reject is reserved for having no usable server at all.

Adds test/common/utxobased/engine/ServerStates.spec.ts covering the
header routing, both no-key cases, and the existing HTTP broadcast
behavior it has to preserve.
@peachbits
peachbits force-pushed the matthew/broadcast-all-servers branch from fa5f837 to f809273 Compare September 3, 2026 05:09
@peachbits
peachbits marked this pull request as ready for review September 3, 2026 19:52

@j0ntz j0ntz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed alongside EdgeApp/edge-react-gui#6190; the findings are inline.

Nit: the commit subject Broadcast to every server at once instead of gating HTTP on the sockets is 71 characters, over Edge's 50-character limit. Broadcast to every server at once is 33 and the body already carries the rest. (Send the NOWNodes API key only to NOWNodes servers is exactly 50 and fine.)

Comment thread src/common/utxobased/engine/ServerStates.ts Outdated
Comment thread src/common/utxobased/engine/ServerStates.ts Outdated

@j0ntz j0ntz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correcting the verdict on my earlier review: those findings are blocking, not advisory.

The broadcast hang is the one that gates this. An all-fail broadcast with any entry in serverStatesCache settles neither way, and paired with EdgeApp/edge-react-gui#6190 the send scene has no error to show and no finally to unlock the slider.

promisifyWsMessage wraps every request in a Deferred that is only ever
resolved from inside its own generator, on the happy path. The socket
reports every failure by throwing into that generator:

- an error response in Socket.onMessage
- the 30 second request expiry in Socket.onTimer
- the socket closing in Socket.onSocketClose

The generator had no try/catch, so the throw was logged by the socket
and the deferred was left pending forever. Every caller of
promisifyWsMessage (fetchInfo, fetchAddress, fetchAddressUtxos,
fetchTransaction, ping, broadcastTx) therefore hung on a failed request
instead of seeing the error.

The generator now catches the throw and rejects its deferred. Found in
review of the broadcast fan-out that follows: once every socket attempt
is counted, an all-fail broadcast with a cached socket could never
settle without this.
@peachbits
peachbits force-pushed the matthew/broadcast-all-servers branch from f809273 to 0d32978 Compare September 7, 2026 18:42
@peachbits

Copy link
Copy Markdown
Contributor Author

Subject reworded to Broadcast to every server at once (33 chars). The branch is now four commits: the key fix first as requested, then the promisifyWsMessage rejection fix found via your hang finding, then the fan-out, then a fixup with the all-fail specs. The unconditional-HTTP question on the other thread is with Matthew as a product decision; I will follow up there.

@peachbits
peachbits force-pushed the matthew/broadcast-all-servers branch from 0d32978 to 32c5992 Compare September 7, 2026 19:13

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 32c5992. Configure here.

Comment thread src/common/utxobased/engine/ServerStates.ts
@peachbits
peachbits force-pushed the matthew/broadcast-all-servers branch from ce3c96a to b1d77cf Compare September 7, 2026 22:53
`broadcastTx` used the HTTP servers only when no WebSocket reported
itself connected. A socket that looks connected but never answers
`sendTransaction` therefore failed the entire broadcast, after the 30
second request timeout, without a single HTTP attempt being made. That
is the shape of the incident this task exists for (Asana
1217135300337949): the send failed in the UI while the transaction was
already on the network.

The gate is worse than it looks, because the NOWNodes WebSocket URI is
docked 400 points at score load (ServerScores.serverScoresLoad penalises
any URI carrying key params) and so rarely wins one of the two
connection slots. HTTP is the realistic route to those servers, and it
was reachable only when every socket was visibly down.

broadcastTx now fires in two waves. The first goes out immediately to
every cached blockbook (sockets still connecting included, since a
queued request transmits as soon as the socket opens) and to every
`blockbook` HTTP server, which are Edge's own. The second wave, the
`blockbook-nownode` HTTP servers, fires after NOWNODES_BROADCAST_DELAY_MS
or as soon as every first-wave attempt has failed, whichever comes
first. NOWNodes is a third party, so it only sees a transaction when
Edge's own infrastructure has not already carried it. The first success
resolves; the promise rejects only once every attempt has failed,
logging each failure against its server.

Every attempt is bounded by BROADCAST_ATTEMPT_TIMEOUT_MS (30 seconds,
the same figure the socket layer uses to expire a request). Because
each attempt counts toward "all attempts failed", each one has to
settle; without the bound an HTTP server that accepted the connection
and never answered could hold the broadcast open even after every
socket had failed. The fetch itself is left to finish; only the attempt
is settled as a failure. ServerStateConfig.broadcastTimeoutMs overrides
the bound for tests.

Extends the ServerStates spec with a real WebSocket server: one that
accepts `sendTransaction` and never answers (NOWNodes is held back for
the delay, then the broadcast resolves over HTTP well inside the socket
timeout), one that refuses it alongside failing and hung HTTP servers
(rejects instead of hanging), plus the two-wave timing in both
directions and the api-key header reaching only NOWNodes.
@peachbits
peachbits force-pushed the matthew/broadcast-all-servers branch from b1d77cf to d04bef3 Compare September 7, 2026 22:53
@peachbits
peachbits merged commit d645f75 into master Sep 8, 2026
4 checks passed
@peachbits
peachbits deleted the matthew/broadcast-all-servers branch September 8, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants