Skip to content

Coln query/flir query program mapping - #137

Open
lstwn wants to merge 33 commits into
mainfrom
coln-query/flir-query-program-mapping
Open

Coln query/flir query program mapping#137
lstwn wants to merge 33 commits into
mainfrom
coln-query/flir-query-program-mapping

Conversation

@lstwn

@lstwn lstwn commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Sorry, another big one. Still lots of churn.

FLIR → query program mapping

Makes coln-query consume coln's FLIR directly: a FlatRealm is lowered into a runnable query program, compiled through the full pipeline, and driven through a transactional API intended for coln-store. Most of the non-FLIR churn is structure that this mapping demanded.

FLIR frontend

  • api::query::FlirProgram: lowers a FlatRealm into Stmt/Expr/RelExpr statements: one query per declared rule, atoms into sources or derived-view references, equalities into join variables, and per-rule metadata (rule kind, output schema) kept beside the query program.
  • coln-flir-rs::schema: BaseTableSchema carries one column view per engine (compiler, store, query) plus the index translations between them, so the query engine no longer has to guess at physical column order. FLIR JSON loading is now reusable by other crates (test_utils behind a feature, matching coln-query).
  • New GraphOfGraphs.json fixture; Graph.json/Prim.json reformatted and extended.

Frontend/backend decoupling

  • QueryProgram and Catalog traits: a frontend hands the pipeline two things: the QueryIr (moved out via take_code, so passes consume it) and a catalog describing what SourceExpr leaves name. A second frontend (Datalog, say) implements this without touching api/. SourceExpr leaves now reference a SourceId into the catalog instead of carrying their schema.
  • Three schema vocabularies, explicitly separated: The coln frontend's (BaseTableSchema), a backend-neutral relational::schema::TableSchema, and a backend-specific (incremental::schema::StreamSchema for DBSP. The latter uses the keyed (TupleKey, TupleValue) layout that follows from relying in OrdIndexedZSets. Pulls DBSP assumptions out of the shared layers.

Compile pipeline

  • New per-backend lowering stage (Backend::lower), between optimization and resolution. The DBSP backend uses it to fold each MultiWayEquiJoinExpr (new, variable-oriented N-ary join; what FLIR naturally emits) into a left-deep chain of binary EquiJoinExprs. Running before the resolver lets the pass mint VarExprs freely.
  • optimizer::rewrite: TransformationRule and RewriteDriver: the folding traversal (27 methods across the owned-visitor families) is written once, so a pass contributes only the interesting rule. The driver iterates to fixpoint, bounds rounds, and names the still-firing rules on exhaustion so a rule cycle surfaces as a legible error. Owned visitors take Box<XxxExpr> to avoid re-boxing untouched nodes.
  • host::walk: generic pre/post-order scan over the AST, so scans stop restating tree shape; folds stay on the visitor traits.
  • host::print: pretty printer rendering a program as an indented node tree with a precise addressing vocabulary (on(y in 2), select(out)), which is what makes the plan rewrites debuggable.

Transactional API

  • Tx as a typestate machine over five states: Prepare → Pending → {Committed, Aborted, Rejected}. try_commit runs the engine and checks hard constraints; the caller must commit explicitly or the transaction rolls back.
  • Violations are two types, not one: ViolationsSet (what is violated now) vs ViolationsDelta (what changed), distinguished by a marker. Same rows either way, so the type is the only thing stopping a consumer from reading a repaired constraint as a broken one.
  • unsafe_apply for the non-transactional path to support query engine restarts.

Misc

  • test_helper → test_utils (feature-gated, aligned with coln-flir-rs), substantially expanded.
  • Reworked justfile with comments, test filtering, aliases.

Tests cover the FLIR → query program translation end to end (including that N conditions collapse into one Selection with an AND'ed condition rather than N nested ones), the multi-way join lowering, the printer, and the transactional API.

lstwn added 30 commits August 10, 2026 11:35
…ariants as types) in an oblivious-to-JSON manner
…r crates to consume; Add end-to-end test for translating (but not running yet) FLIR to a query program
…tal backend to substitute multi way joins with a folded sequence of binary joins
…mpact structural transformations without AST boilerplate
…nditions into one selection whose condition ANDs the N conditions, as opposed to N individual Selections
…ery and have SourceExpr leaves reference a name in a Catalog of base tables
…chema, and per backend schemas of tables and views to decouple further from DBSP
@lstwn lstwn self-assigned this Sep 1, 2026
@lstwn
lstwn marked this pull request as ready for review September 1, 2026 16:05
@lstwn

lstwn commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

@incipit0 Do you quickly want to review the changes in coln-store? It's all harmless and mostly due to the transparent Equality struct in coln-flir now..

@incipit0 incipit0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, don't really care about changes in solver I am going to delete it anyway

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants