Coln query/flir query program mapping - #137
Open
lstwn wants to merge 33 commits into
Open
Conversation
…and fix propagated errors in coln-store
…ariants as types) in an oblivious-to-JSON manner
…e for old Code type alias
…r crates to consume; Add end-to-end test for translating (but not running yet) FLIR to a query program
…tal backend to substitute multi way joins with a folded sequence of binary joins
…mpact structural transformations without AST boilerplate
…nditions into one selection whose condition ANDs the N conditions, as opposed to N individual Selections
…ery and have SourceExpr leaves reference a name in a Catalog of base tables
…chema, and per backend schemas of tables and views to decouple further from DBSP
…ts for (soft) violations
…om the outside on ColnQuery
…tils, just like the feature name
lstwn
marked this pull request as ready for review
September 1, 2026 16:05
Collaborator
Author
|
@incipit0 Do you quickly want to review the changes in |
incipit0
approved these changes
Sep 1, 2026
incipit0
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, don't really care about changes in solver I am going to delete it anyway
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sorry, another big one. Still lots of churn.
FLIR → query program mapping
Makes coln-query consume coln's FLIR directly: a FlatRealm is lowered into a runnable query program, compiled through the full pipeline, and driven through a transactional API intended for coln-store. Most of the non-FLIR churn is structure that this mapping demanded.
FLIR frontend
api::query::FlirProgram: lowers aFlatRealmintoStmt/Expr/RelExprstatements: one query per declared rule, atoms into sources or derived-view references, equalities into join variables, and per-rule metadata (rule kind, output schema) kept beside the query program.coln-flir-rs::schema:BaseTableSchemacarries one column view per engine (compiler, store, query) plus the index translations between them, so the query engine no longer has to guess at physical column order. FLIR JSON loading is now reusable by other crates (test_utilsbehind a feature, matching coln-query).GraphOfGraphs.jsonfixture;Graph.json/Prim.jsonreformatted and extended.Frontend/backend decoupling
QueryProgramandCatalogtraits: a frontend hands the pipeline two things: theQueryIr(moved out viatake_code, so passes consume it) and a catalog describing whatSourceExprleaves name. A second frontend (Datalog, say) implements this without touchingapi/.SourceExprleaves now reference aSourceIdinto the catalog instead of carrying their schema.BaseTableSchema), a backend-neutralrelational::schema::TableSchema, and a backend-specific (incremental::schema::StreamSchemafor DBSP. The latter uses the keyed(TupleKey, TupleValue)layout that follows from relying inOrdIndexedZSets. Pulls DBSP assumptions out of the shared layers.Compile pipeline
Backend::lower), between optimization and resolution. The DBSP backend uses it to fold eachMultiWayEquiJoinExpr(new, variable-oriented N-ary join; what FLIR naturally emits) into a left-deep chain of binaryEquiJoinExprs. Running before the resolver lets the pass mintVarExprsfreely.optimizer::rewrite:TransformationRuleandRewriteDriver: the folding traversal (27 methods across the owned-visitor families) is written once, so a pass contributes only the interesting rule. The driver iterates to fixpoint, bounds rounds, and names the still-firing rules on exhaustion so a rule cycle surfaces as a legible error. Owned visitors takeBox<XxxExpr>to avoid re-boxing untouched nodes.host::walk: generic pre/post-order scan over the AST, so scans stop restating tree shape; folds stay on the visitor traits.host::print: pretty printer rendering a program as an indented node tree with a precise addressing vocabulary(on(y in 2), select(out)), which is what makes the plan rewrites debuggable.Transactional API
Txas a typestate machine over five states: Prepare → Pending → {Committed, Aborted, Rejected}.try_commitruns the engine and checks hard constraints; the caller must commit explicitly or the transaction rolls back.ViolationsSet(what is violated now) vsViolationsDelta(what changed), distinguished by a marker. Same rows either way, so the type is the only thing stopping a consumer from reading a repaired constraint as a broken one.unsafe_applyfor the non-transactional path to support query engine restarts.Misc
Tests cover the FLIR → query program translation end to end (including that N conditions collapse into one Selection with an AND'ed condition rather than N nested ones), the multi-way join lowering, the printer, and the transactional API.