introduce data aggregation (#194)#234
Draft
stklcode wants to merge 3 commits into
Draft
Conversation
MatzeKitt
reviewed
Sep 8, 2022
575425b to
ac6cb56
Compare
This column will be used for aggregation and is filled with 1 initially. The number of hits for a specific date/target/referrer is thus no longer the number of entries, but the sum of all hits.
We now aggregate statistics data in the cleanup routine preserving only distinct entries for each date/referrer/target combination with the sum of all hits.
If Statify is extended by custom columns the aggregation routine will fail. To support such scenarios, we introduce a boolean hook, so the previous behavior can be preserved without breaking compatibility.
ac6cb56 to
fd58722
Compare
fd58722 to
fe5e993
Compare
fe5e993 to
cdf4f5a
Compare
cdf4f5a to
18b8a63
Compare
18b8a63 to
f8e356a
Compare
f8e356a to
fe3dd71
Compare
Member
|
These changes would require changes to keep the functionality of Statify - Extended Evaluation, but hopefully helps to improve the performance with large Statify tables (I've installs with 1,000,000+ entries): patrickrobrecht/extended-evaluation-for-statify#29. |
48543bb to
e789310
Compare
|
Kudos, SonarCloud Quality Gate passed!
|
Contributor
Author
|
As previously discussed in various places this feature might need some additional planning.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.








We add a new column named
hitsto the database table with default value of 1.The tracking routine is preserved as is, i.e. it will insert a single row per hit.
The cleanup routine (cron job) is extended with an aggregation function that walks through the database and groups distinct date/referrer/target combinations with the sum of all hits.
before:
after:
The output is generated from
SUM(hits)instead ofCOUNT(*)then.We also introduce a boolean book
statify__skip_aggregation(defaults tofalse) to keep the previous behavior and disable the new aggregation.This might be necessary if the inserted entry is extended with custom columns, e.g. in a
statify__visit_savedhook. If this is the case, aggregation will fail, because we don't know about additional columns. We could of course add another hook for custom aggregation columns and dynamically build up the statements, but for most users this won't be necessary and it can be extended in future releases as well.