Db v4 schema layer#2658
Open
arienandalibi wants to merge 37 commits into
Open
Conversation
…global property id i corresponding to the segment. Persisted to disk. In-memory part grows incrementally.
…ow) of properties present in each (segment, layer) to efficiently retrieve the lists of relevant properties. We union over all segments in the storage. We also filter out props/metadata that aren't present now instead of all globally registered properties.
… for properties and one for metadata in Meta. Wired it in so that layers where we know the property isn't present are skipped.
…und in this layer. Added these for the node storage and edge storage. Skip layers in tprop_iter_layers, which is used in time semantics.
…es. These were used to track property presence per-(segment, layer) on disk and in-memory. Now, we only keep track of property presence per layer. We skip layers where we know the property was added, but that's it.
… previous layer property schemas (property presence bitsets) to avoid work. All of it was redundant and didn't save anything. Removed functions that pass these `LayerPropSchema`s around.
…pe, dst_node_type). Introduce limits for max number of edges/values in properties and edges.
# Conflicts: # raphtory-storage/src/graph/nodes/node_storage_ops.rs # raphtory/src/db/graph/views/filter/model/degree_filter.rs
…g all properties. As soon as a property key has more than ENUM_BOUNDARY values, we skip property value collection.
…ies and metadata. This is held by GraphWithVectors(Inner). It gets passed to GqlGraph for base/unfiltered (e.g. materialized) graphs, which passes them to EdgeSchema by Arc.
…String, String) type alias. This allows us to pass references to the key easily. Faster on cache lookup
# Conflicts: # raphtory-storage/src/graph/nodes/node_storage_ops.rs
# Conflicts: # raphtory-graphql/src/data.rs # raphtory-graphql/src/graph.rs # raphtory-storage/src/graph/nodes/node_storage_ops.rs
…es and metadata per layer. We use the maintained property presence bitsets to do this
… rid of NodeSchemaKey. Wire the cache through in the schemas.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: aee77a1 | Previous: 9823ef7 | Ratio |
|---|---|---|---|
lotr_graph/num_edges |
4 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
4 ns/iter (± 0) |
1 ns/iter (± 0) |
4 |
lotr_graph/graph_latest |
3 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
7800402 ns/iter (± 38373) |
1564816 ns/iter (± 35303) |
4.98 |
lotr_graph_window_100/num_nodes |
13 ns/iter (± 0) |
5 ns/iter (± 0) |
2.60 |
lotr_graph_window_100_materialise/materialize |
7870030 ns/iter (± 41252) |
1669150 ns/iter (± 10700) |
4.71 |
lotr_graph_window_10/has_node_existing |
141 ns/iter (± 9) |
62 ns/iter (± 11) |
2.27 |
lotr_graph_window_10_materialise/materialize |
3289025 ns/iter (± 8541) |
971980 ns/iter (± 4278) |
3.38 |
lotr_graph_subgraph_10pc_materialise/materialize |
2068910 ns/iter (± 15530) |
334634 ns/iter (± 1287) |
6.18 |
lotr_graph_subgraph_10pc_windowed/has_node_existing |
145 ns/iter (± 13) |
62 ns/iter (± 14) |
2.34 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1242704 ns/iter (± 13349) |
230399 ns/iter (± 2617) |
5.39 |
lotr_graph_window_50_layered/num_edges_temporal |
145693 ns/iter (± 2293) |
70121 ns/iter (± 7586) |
2.08 |
lotr_graph_window_50_layered/has_node_existing |
374 ns/iter (± 20) |
129 ns/iter (± 12) |
2.90 |
lotr_graph_window_50_layered/graph_latest |
86635 ns/iter (± 1638) |
36649 ns/iter (± 916) |
2.36 |
lotr_graph_window_50_layered_materialise/materialize |
28506457 ns/iter (± 75868) |
3488825 ns/iter (± 24948) |
8.17 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
585645 ns/iter (± 28120) |
192686 ns/iter (± 1569) |
3.04 |
lotr_graph_persistent_window_50_layered/has_node_existing |
417 ns/iter (± 363) |
174 ns/iter (± 83) |
2.40 |
lotr_graph_persistent_window_50_layered/graph_latest |
121050 ns/iter (± 1109) |
57549 ns/iter (± 4809) |
2.10 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
48760506 ns/iter (± 83599) |
5298035 ns/iter (± 147912) |
9.20 |
This comment was automatically generated by workflow using github-action-benchmark.
…e bitset to integration-tests (in pometry-storage) instead of raphtory-tests (in Raphtory). Fixed queries and responses to match new schemas.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adding a bitset (Vec) to keep track of which properties exist in which layers. Use this to optimize iterators (especially relevant when using disk storage). Also optimized the generation of schemas (node schema, edge schema, ...) by iterating edges only once and implementing a cache for already generated schemas.
Why are the changes needed?
Optimizations can be made to improve UI experience and disk accesses.
Does this PR introduce any user-facing change? If yes is this documented?
It shouldn't
How was this patch tested?
Will need to be tested using the UI
Are there any further changes required?
There shouldn't be