Repository navigation
feat: a Python client, written against the HTTP surface - #19
Merged
Merged
Conversation
`clients/` had a readme and an empty directory. The three client-shaped things in
the tree are `bigctl`, `contrib/big-message`, and a console API layer that does
no networking at all - so anyone reaching for bigdb from Python had to write
their own HTTP and their own escaping, and escaping is where a mistake is a
security bug rather than a wrong answer.
Data plane and schema: `/sql`, `/table/{t}/query`, `/table/{t}/import`,
`/table/{t}/delete`, `/table/{t}/records`, `/schema`, `/health`, `/ready`, and
the DDL routes. Not the operator surface - `/metrics`, `/verify`, `/repair`,
`/admin/backup`, `/cluster/*` - which is `bigctl`'s and wants a person reading
the answers. A sync client, an async one, and a PEP 249 driver so
`pandas.read_sql` works.
No dependencies, the same claim `contrib/big-message` makes and for the same
reason: an empty manifest is the only form of it a build can check. CI checks it
twice - the wheel declares no `Requires-Dist`, and importing the package pulls in
nothing outside the standard library.
`transport/base.py`'s `RawResponse` is the seam. Everything about the protocol
lives above it and is written once - the route table, the decoders, the escaping,
the retry policy; sockets live below it and are written twice, because there is
no way to write them once. `test_transport_conformance.py` runs one table of
cases against both transports, so a difference between them is a bug in one
rather than a design decision.
What is ported rather than reinvented, because a client with its own idea of
these agrees with the server until it does not:
- `sql.py` is `contrib/big-message/src/sql.rs` function for function, with its
test vectors. Identifiers are always quoted, numbers are checked against
`big_sql::lex`'s own grammar - no exponent form, `units / 10^scale` in a `u64`
- and `None` is refused because the dialect has no `NULL` literal. A 20,000
case differential fuzz holds the `?` scanner against an independently written
oracle.
- The retry rules and the connection lifetime from `big-message`'s readme:
failures split by what the caller can do about them, so `NotSent` is always
safe to repeat and `Unknown` is never repeated on a route that allocates. The
keep-alive connection is retired at 900 requests and 3s idle against the
server's 1000 and 5s, which makes the idle-close race unreachable rather than
handled.
- Response shapes and error codes from `crates/big-http/src/json.rs`, with bodies
copied out of the server's own tests.
Two vocabularies are kept as two, not normalised: `?kind=signed` on the way in
comes back as `signedint`, and inventing a third spelling neither side speaks
would be worse than the asymmetry. The console's `types.ts` is stale on this and
on the error envelope key, and is not the reference used here.
Gated by `make py-check`, kept out of `make check` deliberately: that one is
`lint test docs` and Rust-only, and adding a Python toolchain to the command
every contributor runs would break it on a machine with no `pip`.
9bany
added a commit
that referenced
this pull request
Sep 4, 2026
`clients/go` was an empty directory. The Python client landed in #19 and the same argument applies here: anyone reaching for bigdb from Go had to write their own HTTP and their own escaping, and escaping is where a mistake is a security bug rather than a wrong answer. `archived/featurebase` is Go and is not a starting point - it speaks a different server's routes. Same scope as the Python client: `/sql`, `/table/{t}/query`, `/table/{t}/import`, `/table/{t}/delete`, `/table/{t}/records`, `/schema`, `/health`, `/ready`, and the DDL routes. Not the operator surface. A `Client`, an `iter.Seq2` walk over record ids, and a `database/sql` driver so `sqlx` and anything else that speaks it works. No dependencies. `go.mod` has no `require` block, there is no `go.sum`, and CI fails if either changes - `make deps` walks `go list -deps` and asserts every package is either stdlib or this module. What is ported rather than reinvented, because a client with its own idea of these agrees with the server until it does not: - `sql.go` is `contrib/big-message/src/sql.rs` function for function, with its test vectors. Identifiers are always quoted, numbers are checked against `big_sql::lex`'s own grammar, and `nil` is refused because the dialect has no `NULL` literal. One thing is shorter here: `strconv.FormatFloat` with `'f'` and precision -1 never emits an exponent, so `push_float`'s search for a spelling has nothing to search for. `1e300` is still refused, by the same `u64` overflow it reaches in Rust. - The retry rules and the connection lifetime from `big-message`'s readme. `TransportError.Sent` is the split: false is always safe to repeat, true is never repeated on a route that allocates. The connection is retired at 900 requests and 3s idle against the server's 1000 and 5s, so the idle-close race is unreachable rather than handled. Through `database/sql` the same two rules are `driver.Validator.IsValid` and `driver.SessionResetter.ResetSession`, which the pool already calls at exactly those two moments. Three things Go forces that the Python client did not have to answer: - `net/http` is not the transport. It knows whether a request reached the wire and discards the fact: when a request will not be retried, `transport.go` unwraps `nothingWrittenError` and `transportReadFromServerError` to the underlying error, so "nothing was written" and "written in full, then silence" arrive identical. That one bit is the whole retry policy. Its own retrying is not the problem and the readme says so. `WithDoer` takes an implementation for anyone who needs a proxy more than the distinction. - `encoding/json` decodes a number into `any` as a `float64`, and record ids are `u64`: `18446744073709551615` comes back as `...552000`. So every shape is a typed struct and the escape hatch for a field this build has not heard of is `json.RawMessage`, not `map[string]any`. Two tests hold it, one at each end of the pipe. - `Client` takes a lock. The Python client declares `threadsafety = 1` and does not, on the grounds that a lock serialises calls without saying so. In Go a client that cannot be shared will be shared anyway; the readme says it is one connection and points at `database/sql` for parallelism. `?database=` is folded into the table name rather than sent as a parameter alone. #18 made the parameter scope the data routes, but the DDL routes read the path only - so a client that set a database would `CreateTable` in the default one and `Import` into another, with a 200 at every step. A qualified path works on every route and is documented to win over a disagreeing parameter, so `WithDatabase` has one meaning across the whole client. The parameter is still sent; the permission check reads it. Testing is 3,700 lines against 4,100 of source, and the vectors are the repository's own so the two sides are held to each other rather than each to itself. The fake server is a raw `net.Listen` rather than `httptest.Server`, which cannot emit a `Content-Length` that lies or a response cut off mid status line - the cases that matter. Integration is behind `-tags integration` and skips with instructions when no `big` binary is found; it does not build one, because a `go test` that shells out to cargo fails outright without Rust. The integration suite found the schema snapshot bug fixed in #18 - a create that answered 200, a field route that answered 200, and a write that said the table did not exist. `table()` asserts the schema admits a table exists rather than trusting the status code, which is what caught it, and a regression test pins drop-then-create.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
clients/had a readme and an empty directory. The three client-shaped things in the tree arebigctl,contrib/big-message, and a console API layer that does no networking at all - so anyone reaching for bigdb from Python had to write their own HTTP and their own escaping, and escaping is where a mistake is a security bug rather than a wrong answer.Data plane and schema:
/sql,/table/{t}/query,/table/{t}/import,/table/{t}/delete,/table/{t}/records,/schema,/health,/ready, and the DDL routes. Not the operator surface -/metrics,/verify,/repair,/admin/backup,/cluster/*- which isbigctl's and wants a person reading the answers. A sync client, an async one, and a PEP 249 driver sopandas.read_sqlworks.No dependencies, the same claim
contrib/big-messagemakes and for the same reason: an empty manifest is the only form of it a build can check. CI checks it twice - the wheel declares noRequires-Dist, and importing the package pulls in nothing outside the standard library.transport/base.py'sRawResponseis the seam. Everything about the protocol lives above it and is written once - the route table, the decoders, the escaping, the retry policy; sockets live below it and are written twice, because there is no way to write them once.test_transport_conformance.pyruns one table of cases against both transports, so a difference between them is a bug in one rather than a design decision.What is ported rather than reinvented, because a client with its own idea of these agrees with the server until it does not:
sql.pyiscontrib/big-message/src/sql.rsfunction for function, with its test vectors. Identifiers are always quoted, numbers are checked againstbig_sql::lex's own grammar - no exponent form,units / 10^scalein au64Noneis refused because the dialect has noNULLliteral. A 20,000 case differential fuzz holds the?scanner against an independently written oracle.big-message's readme: failures split by what the caller can do about them, soNotSentis always safe to repeat andUnknownis never repeated on a route that allocates. The keep-alive connection is retired at 900 requests and 3s idle against the server's 1000 and 5s, which makes the idle-close race unreachable rather than handled.crates/big-http/src/json.rs, with bodies copied out of the server's own tests.Two vocabularies are kept as two, not normalised:
?kind=signedon the way in comes back assignedint, and inventing a third spelling neither side speaks would be worse than the asymmetry. The console'stypes.tsis stale on this and on the error envelope key, and is not the reference used here.Gated by
make py-check, kept out ofmake checkdeliberately: that one islint test docsand Rust-only, and adding a Python toolchain to the command every contributor runs would break it on a machine with nopip.