Repository navigation
feat: bigdb-proxy, one address in front of many nodes - #21
Merged
Merged
Conversation
A cluster published one node's port, with a comment explaining that a client does not have to choose which node answers. That was right and incomplete: `clients/go` holds one connection to one address with no failover and no view of the topology, so the published node was a single point of failure for machines that were all healthy. Publishing all three would have moved the choice to the client, which has nowhere to put it. `bigproxy --cluster cluster.toml` asks every node whether it is serving, sends each request to the least busy one that says yes, and carries nothing that is not on a route table written down in advance. It deliberately does not route by key. Which node serves a range is a value the nodes agree on, and a proxy computing the same answer would be a second copy of ownership.rs that is wrong for as long as it takes a moved range to reach it -- and that the cluster cannot correct, because the proxy is not in the agreement. The question it answers has a published answer: GET /ready reports `serving`, which the daemon documents as the field a load balancer may act on. Two properties of that route shape the poller. It is always 200, so the body is the signal and not the status line. And `serving` is absent on a node with no agreement -- absent means serving, or the rotation empties for every unreplicated deployment. The allowlist is an allowlist because a denylist is a list somebody can forget to extend; the twenty-one /internal/* routes are excluded by not appearing. Matching is positional against decoded segments and the upstream path is rebuilt from what matched, so `..`, `%2f` and collapsed slashes cannot produce a request the table did not intend. A route that is not in it answers 404 no_such_route -- the same sentence a typo gets, because a distinguishable refusal would confirm the peer surface exists. Three independent things keep that surface closed, and no flag turns any off: the table does not name those paths; x-big-wire and x-big-cluster are stripped from every client request; and the proxy presents no client certificate, so it lands as Identity::None and is refused with 403 not_a_peer. --upstream-ca may point at the cluster's own peer-ca.pem: a CA certificate is public, and trusting a CA is not being trusted by it. There is deliberately no --upstream-cert. Retries are the narrow rule: a request that never reached the socket may go to another node, for any route including an import, because the second attempt is the first. A repeatable route may retry on a failure carrying no answer. POST /sql is not repeatable -- it carries CREATE TABLE too, and this proxy does not parse SQL to find out. A 5xx is never retried; it is an answer. Also splits big-wire out of big-http. The proxy speaks HTTP and touches no data, but while the parser lived beside the router, linking it meant linking big-engine, big-sql and argon2 -- and "this process never verifies a password" stayed a sentence in a readme. The shipped tree is now big-proxy -> big-wire -> big-tls, and CI greps `cargo tree -e normal` for the names that must not appear. Response::from_error could not follow the parser out and is status::response_for. Three constants are copied rather than imported for the same reason, and tests/agreement.rs holds each to the original with big-cluster as a dev-dependency: a copy nobody checks is a copy that drifts. deploy/cluster now publishes the proxy and nothing else. `docker compose stop a` and the queries keep working, which is the only demonstration that matters.
9bany
force-pushed
the
feat/bigdb-proxy
branch
from
September 4, 2026 11:32
045f5db to
b211358
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A cluster published one node's port, with a comment explaining that a client does not have to choose which node answers. That was right and incomplete:
clients/goholds one connection to one address with no failover and no view of the topology, so the published node was a single point of failure for machines that were all healthy. Publishing all three would have moved the choice to the client, which has nowhere to put it.bigproxy --cluster cluster.tomlasks every node whether it is serving, sends each request to the least busy one that says yes, and carries nothing that is not on a route table written down in advance.It deliberately does not route by key. Which node serves a range is a value the nodes agree on, and a proxy computing the same answer would be a second copy of ownership.rs that is wrong for as long as it takes a moved range to reach it -- and that the cluster cannot correct, because the proxy is not in the agreement. The question it answers has a published answer: GET /ready reports
serving, which the daemon documents as the field a load balancer may act on.Two properties of that route shape the poller. It is always 200, so the body is the signal and not the status line. And
servingis absent on a node with no agreement -- absent means serving, or the rotation empties for every unreplicated deployment.The allowlist is an allowlist because a denylist is a list somebody can forget to extend; the twenty-one /internal/* routes are excluded by not appearing. Matching is positional against decoded segments and the upstream path is rebuilt from what matched, so
..,%2fand collapsed slashes cannot produce a request the table did not intend. A route that is not in it answers 404 no_such_route -- the same sentence a typo gets, because a distinguishable refusal would confirm the peer surface exists.Three independent things keep that surface closed, and no flag turns any off: the table does not name those paths; x-big-wire and x-big-cluster are stripped from every client request; and the proxy presents no client certificate, so it lands as Identity::None and is refused with 403 not_a_peer. --upstream-ca may point at the cluster's own peer-ca.pem: a CA certificate is public, and trusting a CA is not being trusted by it. There is deliberately no --upstream-cert.
Retries are the narrow rule: a request that never reached the socket may go to another node, for any route including an import, because the second attempt is the first. A repeatable route may retry on a failure carrying no answer. POST /sql is not repeatable -- it carries CREATE TABLE too, and this proxy does not parse SQL to find out. A 5xx is never retried; it is an answer.
Also splits big-wire out of big-http. The proxy speaks HTTP and touches no data, but while the parser lived beside the router, linking it meant linking big-engine, big-sql and argon2 -- and "this process never verifies a password" stayed a sentence in a readme. The shipped tree is now big-proxy -> big-wire -> big-tls, and CI greps
cargo tree -e normalfor the names that must not appear. Response::from_error could not follow the parser out and is status::response_for.Three constants are copied rather than imported for the same reason, and tests/agreement.rs holds each to the original with big-cluster as a dev-dependency: a copy nobody checks is a copy that drifts.
deploy/cluster now publishes the proxy and nothing else.
docker compose stop aand the queries keep working, which is the only demonstration that matters.