Skip to content

feat: bigdb-proxy, one address in front of many nodes - #21

Merged
9bany merged 1 commit into
masterfrom
feat/bigdb-proxy
Sep 4, 2026
Merged

9bany merged 1 commit into
masterfrom
feat/bigdb-proxy

Conversation

@9bany

@9bany 9bany commented Sep 4, 2026

Copy link
Copy Markdown
Member

A cluster published one node's port, with a comment explaining that a client does not have to choose which node answers. That was right and incomplete: clients/go holds one connection to one address with no failover and no view of the topology, so the published node was a single point of failure for machines that were all healthy. Publishing all three would have moved the choice to the client, which has nowhere to put it.

bigproxy --cluster cluster.toml asks every node whether it is serving, sends each request to the least busy one that says yes, and carries nothing that is not on a route table written down in advance.

It deliberately does not route by key. Which node serves a range is a value the nodes agree on, and a proxy computing the same answer would be a second copy of ownership.rs that is wrong for as long as it takes a moved range to reach it -- and that the cluster cannot correct, because the proxy is not in the agreement. The question it answers has a published answer: GET /ready reports serving, which the daemon documents as the field a load balancer may act on.

Two properties of that route shape the poller. It is always 200, so the body is the signal and not the status line. And serving is absent on a node with no agreement -- absent means serving, or the rotation empties for every unreplicated deployment.

The allowlist is an allowlist because a denylist is a list somebody can forget to extend; the twenty-one /internal/* routes are excluded by not appearing. Matching is positional against decoded segments and the upstream path is rebuilt from what matched, so .., %2f and collapsed slashes cannot produce a request the table did not intend. A route that is not in it answers 404 no_such_route -- the same sentence a typo gets, because a distinguishable refusal would confirm the peer surface exists.

Three independent things keep that surface closed, and no flag turns any off: the table does not name those paths; x-big-wire and x-big-cluster are stripped from every client request; and the proxy presents no client certificate, so it lands as Identity::None and is refused with 403 not_a_peer. --upstream-ca may point at the cluster's own peer-ca.pem: a CA certificate is public, and trusting a CA is not being trusted by it. There is deliberately no --upstream-cert.

Retries are the narrow rule: a request that never reached the socket may go to another node, for any route including an import, because the second attempt is the first. A repeatable route may retry on a failure carrying no answer. POST /sql is not repeatable -- it carries CREATE TABLE too, and this proxy does not parse SQL to find out. A 5xx is never retried; it is an answer.

Also splits big-wire out of big-http. The proxy speaks HTTP and touches no data, but while the parser lived beside the router, linking it meant linking big-engine, big-sql and argon2 -- and "this process never verifies a password" stayed a sentence in a readme. The shipped tree is now big-proxy -> big-wire -> big-tls, and CI greps cargo tree -e normal for the names that must not appear. Response::from_error could not follow the parser out and is status::response_for.

Three constants are copied rather than imported for the same reason, and tests/agreement.rs holds each to the original with big-cluster as a dev-dependency: a copy nobody checks is a copy that drifts.

deploy/cluster now publishes the proxy and nothing else. docker compose stop a and the queries keep working, which is the only demonstration that matters.

A cluster published one node's port, with a comment explaining that a client
does not have to choose which node answers. That was right and incomplete:
`clients/go` holds one connection to one address with no failover and no view
of the topology, so the published node was a single point of failure for
machines that were all healthy. Publishing all three would have moved the
choice to the client, which has nowhere to put it.

`bigproxy --cluster cluster.toml` asks every node whether it is serving, sends
each request to the least busy one that says yes, and carries nothing that is
not on a route table written down in advance.

It deliberately does not route by key. Which node serves a range is a value the
nodes agree on, and a proxy computing the same answer would be a second copy of
ownership.rs that is wrong for as long as it takes a moved range to reach it --
and that the cluster cannot correct, because the proxy is not in the agreement.
The question it answers has a published answer: GET /ready reports `serving`,
which the daemon documents as the field a load balancer may act on.

Two properties of that route shape the poller. It is always 200, so the body is
the signal and not the status line. And `serving` is absent on a node with no
agreement -- absent means serving, or the rotation empties for every
unreplicated deployment.

The allowlist is an allowlist because a denylist is a list somebody can forget
to extend; the twenty-one /internal/* routes are excluded by not appearing.
Matching is positional against decoded segments and the upstream path is rebuilt
from what matched, so `..`, `%2f` and collapsed slashes cannot produce a request
the table did not intend. A route that is not in it answers 404 no_such_route --
the same sentence a typo gets, because a distinguishable refusal would confirm
the peer surface exists.

Three independent things keep that surface closed, and no flag turns any off:
the table does not name those paths; x-big-wire and x-big-cluster are stripped
from every client request; and the proxy presents no client certificate, so it
lands as Identity::None and is refused with 403 not_a_peer. --upstream-ca may
point at the cluster's own peer-ca.pem: a CA certificate is public, and trusting
a CA is not being trusted by it. There is deliberately no --upstream-cert.

Retries are the narrow rule: a request that never reached the socket may go to
another node, for any route including an import, because the second attempt is
the first. A repeatable route may retry on a failure carrying no answer. POST
/sql is not repeatable -- it carries CREATE TABLE too, and this proxy does not
parse SQL to find out. A 5xx is never retried; it is an answer.

Also splits big-wire out of big-http. The proxy speaks HTTP and touches no data,
but while the parser lived beside the router, linking it meant linking
big-engine, big-sql and argon2 -- and "this process never verifies a password"
stayed a sentence in a readme. The shipped tree is now big-proxy -> big-wire ->
big-tls, and CI greps `cargo tree -e normal` for the names that must not appear.
Response::from_error could not follow the parser out and is status::response_for.

Three constants are copied rather than imported for the same reason, and
tests/agreement.rs holds each to the original with big-cluster as a
dev-dependency: a copy nobody checks is a copy that drifts.

deploy/cluster now publishes the proxy and nothing else. `docker compose stop a`
and the queries keep working, which is the only demonstration that matters.
@9bany
9bany merged commit f8fee23 into master Sep 4, 2026
17 of 22 checks passed
@9bany
9bany deleted the feat/bigdb-proxy branch September 7, 2026 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant