Repository navigation
feat: add aliases option and decoder support for alias-compressed LDF - #14
Merged
Merged
Conversation
## Changes - Add `options["aliases"]` to `json_to_lineage_format` (default `True`). It has no effect on the output yet. - Add an `aliases` parameter to `convert_json_file_to_lineage`. - Add a `--no-aliases` flag to the `json-to-lineage` CLI command. - Teach `lineage_format_to_json` to read the new constructs, in addition to the legacy form: - `REF` section with `@Rn` references in `ATTR`, `HP` and `DSD`. An undefined `@...` stays a literal. - `*` default entry in `HP` and `DSD`. An empty entry means "none" and beats `*`. - `IKP` section with `ikp=` plus an `ik=` suffix. - `NDEF` section with default node tokens. An own token overrides the default, and an empty token (`origin=`) removes it. - Edges grouped by source in `E` (`N1>N2,N3`). - Default `EP` lines (`*>* etype=…`, `* etype=…`). An explicit entry wins, and an empty `etype=` means "no type". - Add `tests/test_aliases.py` for the new decoder paths, the CLI flag and unchanged encoder output.
## Summary The encoder now removes repeated values from the output. The decoder reads both the new and the old (expanded) form. ## Changes - New `aliases` option (default `True`), `--no-aliases` in the CLI and an `aliases` argument in `convert_json_file_to_lineage`. `aliases=False` gives the 0.2.0 text. - `REF` section and `@Rn` references for repeated `HP` / `ATTR` / `DSD` values and elements (also when attributes come in a different order). - `*` default entry in `HP` and `DSD`; an empty entry means "none". - `IKP` section and `ikp=` node token for shared `identity_key` prefixes. - `NDEF` section with default node tokens (`origin`, `path`, `flags`, ...); a node can override a token or remove it with an empty `key=`. - `E` grouped by source (`N1>N2,N3`); end edges stay bare (`N7`). - `EP` default lines (`*>* etype=direct`, `* etype=summary`); explicit entries only for exceptions. - A section is written only when it makes the text shorter. - Version bumped to 0.3.0; README, CHANGELOG and STRUCTURE updated. ## Behaviour change - The default output differs from 0.2.0 and needs a reader from 0.3.0. - The order of the edge list is not preserved. The set of edges and their types is. ## Results - `out.json`: 18172 -> 5474 chars (30%) - `big_query_lineage.json`: 22802 -> 9767 chars (43%) ## Tests - New encoder and real-file tests in `tests/test_aliases.py`. - `tests/test_exclude_fields.py` runs with `aliases=False` for exact-text checks, and every `big_query_lineage.json` case also checks the compressed form. - 363 tests pass; a one-off check on 3000 random graphs decoded identically in both modes.
itspatys
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The encoder now removes repeated values from the output. The decoder reads both
the new and the old (expanded) form.
Changes
aliasesoption (defaultTrue),--no-aliasesin the CLI and analiasesargument in
convert_json_file_to_lineage.aliases=Falsegives the 0.2.0 text.REFsection and@Rnreferences for repeatedHP/ATTR/DSDvalues andelements (also when attributes come in a different order).
*default entry inHPandDSD; an empty entry means "none".IKPsection andikp=node token for sharedidentity_keyprefixes.NDEFsection with default node tokens (origin,path,flags, ...); a node canoverride a token or remove it with an empty
key=.Egrouped by source (N1>N2,N3); end edges stay bare (N7).EPdefault lines (*>* etype=direct,* etype=summary); explicit entries only forexceptions.
Behaviour change
Results
out.json: 18172 -> 5474 chars (30%)big_query_lineage.json: 22802 -> 9767 chars (43%)Tests
tests/test_aliases.py.tests/test_exclude_fields.pyruns withaliases=Falsefor exact-text checks, andevery
big_query_lineage.jsoncase also checks the compressed form.