feat(db): add image token counts to hourly stats#1674
feat(db): add image token counts to hourly stats#1674
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds imageInputTokens and imageOutputTokens to aggregation logic, hourly-stats DB schema, and migration so image token sums are produced and persisted for project and API-key hourly stats. Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Client
participant LogsDB as Logs DB
participant Aggregator as Worker Aggregator
participant HourlyDB as Hourly Stats DB
participant TestHelper as Test Helper
Client->>LogsDB: insert log rows with image_input_tokens / image_output_tokens
Aggregator->>LogsDB: query recent log rows
Aggregator->>Aggregator: compute COALESCE(SUM(CAST(... AS numeric)), 0) for imageInputTokens / imageOutputTokens
Aggregator->>HourlyDB: upsert hourly stats including imageInputTokens/imageOutputTokens
TestHelper->>HourlyDB: read hourly stats for assertions
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR adds support for tracking image token counts in the hourly aggregation statistics tables. It extends the existing hourly stats infrastructure to include imageInputTokens and imageOutputTokens fields, mirroring the structure already present in the log table.
Changes:
- Added
imageInputTokensandimageOutputTokenscolumns to all 4 hourly stats tables with default value "0" - Updated worker aggregation logic to sum image token counts from the log table
- Updated test helper aggregation function to include image token fields
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| packages/db/src/schema.ts | Added imageInputTokens and imageOutputTokens decimal columns to all 4 hourly stats tables |
| apps/worker/src/services/project-stats-aggregator.ts | Extended getCommonAggregationFields() to aggregate image token counts from log table |
| apps/api/src/testing.ts | Extended test helper's getCommonAggregationFields() to include image token aggregation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| imageInputTokens: decimal().notNull().default("0"), | ||
| imageOutputTokens: decimal().notNull().default("0"), |
There was a problem hiding this comment.
Missing database migration file for adding imageInputTokens and imageOutputTokens columns. Schema changes need corresponding migrations to be applied to the database. Run pnpm migrations in the packages/db directory to generate the migration file for these new columns across all four hourly stats tables.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@packages/db/src/schema.ts`:
- Around line 1229-1230: The schema defines imageInputTokens/imageOutputTokens
(imageInputTokens/imageOutputTokens in schema.ts) but the migration
1770740952_same_lake.sql only creates image_input_cost/image_output_cost; update
that migration by adding the two missing columns "image_input_tokens" numeric
DEFAULT '0' NOT NULL and "image_output_tokens" numeric DEFAULT '0' NOT NULL to
each of the four CREATE TABLE statements, placing them immediately after the
"cached_tokens" column so the migration matches the schema.
| imageInputTokens: decimal().notNull().default("0"), | ||
| imageOutputTokens: decimal().notNull().default("0"), |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's check the schema file at the specified lines
wc -l packages/db/src/schema.tsRepository: theopenco/llmgateway
Length of output: 94
🏁 Script executed:
# Read the schema file around the specified lines to verify the changes
sed -n '1225,1235p' packages/db/src/schema.ts
echo "---"
sed -n '1293,1303p' packages/db/src/schema.ts
echo "---"
sed -n '1372,1382p' packages/db/src/schema.ts
echo "---"
sed -n '1451,1461p' packages/db/src/schema.tsRepository: theopenco/llmgateway
Length of output: 2027
🏁 Script executed:
# Search for migration files that might reference these new columns
find . -type f \( -name "*.sql" -o -name "*.ts" -o -name "*.js" \) -path "*/migration*" 2>/dev/null | head -20Repository: theopenco/llmgateway
Length of output: 1223
🏁 Script executed:
# Search for any reference to imageInputTokens or imageOutputTokens in the codebase
rg 'imageInputTokens|imageOutputTokens' --type-list | head -5
echo "---"
rg 'imageInputTokens|imageOutputTokens' -g '*.sql' -g '*.ts' --max-count=50Repository: theopenco/llmgateway
Length of output: 7886
🏁 Script executed:
# Check the most recent migration files
cat packages/db/migrations/1749327353_white_vivisector.sql
echo "---"
cat packages/db/migrations/1749323043_panoramic_gorilla_man.sqlRepository: theopenco/llmgateway
Length of output: 1128
🏁 Script executed:
# Search more specifically for ALTER TABLE statements that might add these columns
rg 'ALTER TABLE.*image' --type sql
echo "---"
rg 'imageInputTokens|imageOutputTokens' -g '*.sql'Repository: theopenco/llmgateway
Length of output: 696
🏁 Script executed:
# Check if there's a drizzle config and if migrations are auto-generated
find . -name "drizzle.config.*" -o -name ".drizzlerc*" 2>/dev/null
echo "---"
# Check recent git changes to migration files
git log --oneline packages/db/migrations/ 2>/dev/null | head -10Repository: theopenco/llmgateway
Length of output: 157
🏁 Script executed:
# Check the migration file that was just found
cat packages/db/migrations/1770626225_clever_silhouette.sqlRepository: theopenco/llmgateway
Length of output: 363
🏁 Script executed:
# Look for any other recent migrations that might add columns to the hourly stats tables
rg 'projectHourlyStats|projectHourlyModelStats|apiKeyHourlyStats|apiKeyHourlyModelStats' --type sql
echo "---"
# Check for any ALTER TABLE statements on these tables in migrations
rg 'ALTER TABLE "project_hourly|ALTER TABLE "api_key_hourly' --type sqlRepository: theopenco/llmgateway
Length of output: 67
🏁 Script executed:
# Get the most recent migrations to see the overall timeline
ls -lt packages/db/migrations/ | head -20Repository: theopenco/llmgateway
Length of output: 1600
🏁 Script executed:
# Let's check if the migration file 1770626225 exists and what it contains entirely
cat packages/db/migrations/1770626225_clever_silhouette.sql 2>/dev/null || echo "File not found in 1749xxx range, checking full directory..."Repository: theopenco/llmgateway
Length of output: 363
🏁 Script executed:
# Let's be more thorough - check all migration files for any mention of hourly stats tables
ls packages/db/migrations/*.sql | while read f; do
if rg -l 'project_hourly|api_key_hourly' "$f" 2>/dev/null; then
echo "Found in: $f"
cat "$f"
fi
doneRepository: theopenco/llmgateway
Length of output: 10110
🏁 Script executed:
# Also check git history for the schema changes
git log --oneline --all packages/db/src/schema.ts 2>/dev/null | head -5
echo "---"
git log --oneline --all packages/db/migrations/ 2>/dev/null | head -5Repository: theopenco/llmgateway
Length of output: 281
🏁 Script executed:
# Let's check the schema file to see all the columns defined for one of these tables
sed -n '1200,1240p' packages/db/src/schema.tsRepository: theopenco/llmgateway
Length of output: 1849
🏁 Script executed:
# Let's get the entire definition of projectHourlyStats table from schema
rg -A 50 'export const projectHourlyStats = pgTable' packages/db/src/schema.ts | head -70Repository: theopenco/llmgateway
Length of output: 2253
🏁 Script executed:
# Let me check the migration file more carefully - search for all columns in the CREATE TABLE statements
cat packages/db/migrations/1770740952_same_lake.sql | grep -E 'image_|tokens'Repository: theopenco/llmgateway
Length of output: 1371
🏁 Script executed:
# Also check if there's an ALTER TABLE to add these columns to existing tables
rg 'ALTER TABLE.*project_hourly.*image_.*_tokens|ALTER TABLE.*api_key_hourly.*image_.*_tokens' --type sqlRepository: theopenco/llmgateway
Length of output: 46
🏁 Script executed:
# Let me see the full content of the migration to understand what it does
wc -l packages/db/migrations/1770740952_same_lake.sqlRepository: theopenco/llmgateway
Length of output: 115
🏁 Script executed:
# Let me view the entire migration file
cat packages/db/migrations/1770740952_same_lake.sqlRepository: theopenco/llmgateway
Length of output: 10004
Schema and migration are misaligned — the imageInputTokens and imageOutputTokens columns are defined in schema.ts but missing from the corresponding migration.
The migration file 1770740952_same_lake.sql creates the four hourly stats tables but excludes image_input_tokens and image_output_tokens columns. The migration only includes image_input_cost and image_output_cost (cost columns, not token columns). This mismatch will cause runtime failures when the code attempts to read or write these token columns to the database.
Add "image_input_tokens" numeric DEFAULT '0' NOT NULL and "image_output_tokens" numeric DEFAULT '0' NOT NULL to all four CREATE TABLE statements in the migration, positioned after "cached_tokens" to match the schema definition.
🤖 Prompt for AI Agents
In `@packages/db/src/schema.ts` around lines 1229 - 1230, The schema defines
imageInputTokens/imageOutputTokens (imageInputTokens/imageOutputTokens in
schema.ts) but the migration 1770740952_same_lake.sql only creates
image_input_cost/image_output_cost; update that migration by adding the two
missing columns "image_input_tokens" numeric DEFAULT '0' NOT NULL and
"image_output_tokens" numeric DEFAULT '0' NOT NULL to each of the four CREATE
TABLE statements, placing them immediately after the "cached_tokens" column so
the migration matches the schema.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
steebchen
force-pushed
the
feat/add-image-token-counts-to-hourly-stats
branch
from
39ca807 to
013bb50
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/db/migrations/meta/_journal.json (1)
727-741:⚠️ Potential issue | 🟡 MinorDuplicate
idx: 103entries in migration journal.There are two consecutive entries with
idx: 103(lines 727-732 and 733-739). This appears to be a pre-existing issue in the journal, but it should be corrected to maintain journal integrity. The second entry should likely beidx: 104, with subsequent entries renumbered accordingly.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/db/migrations/meta/_journal.json` around lines 727 - 741, Two consecutive migration journal entries share the same idx (idx: 103) for the entries with tag "1773841690_keen_toad_men" and "1773841805_dashing_sentinels"; update the second duplicate entry to idx: 104 and then renumber all subsequent entries' idx values incrementally to restore a unique, sequential idx sequence (ensure you update any following entry that currently starts with idx: 104 to 105, etc., so the journal's idx values remain continuous).
🧹 Nitpick comments (1)
apps/api/src/testing.ts (1)
50-198: Consider extracting shared aggregation fields to reduce duplication.
getCommonAggregationFields()is duplicated between this file andapps/worker/src/services/project-stats-aggregator.ts. Both implementations are nearly identical (~45 fields). This duplication increases maintenance burden—any future field additions must be synchronized across both files.Consider extracting the shared aggregation logic to
@llmgateway/dbor a shared utility module.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@apps/api/src/testing.ts` around lines 50 - 198, getCommonAggregationFields() is duplicated here and in apps/worker/src/services/project-stats-aggregator.ts; extract it into a single shared module (e.g., export a function from `@llmgateway/db` or a new shared util) and have both places import it. Move the full implementation (all aggregation fields) into the new module as an exported function (retain the name getCommonAggregationFields or a clear alias), update both callers to import and use that exported function, and remove the local duplicate definitions; ensure the module exports the same typed SQL fragments so existing callers (in testing.ts and project-stats-aggregator.ts) compile without other changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@packages/db/migrations/meta/_journal.json`:
- Around line 727-741: Two consecutive migration journal entries share the same
idx (idx: 103) for the entries with tag "1773841690_keen_toad_men" and
"1773841805_dashing_sentinels"; update the second duplicate entry to idx: 104
and then renumber all subsequent entries' idx values incrementally to restore a
unique, sequential idx sequence (ensure you update any following entry that
currently starts with idx: 104 to 105, etc., so the journal's idx values remain
continuous).
---
Nitpick comments:
In `@apps/api/src/testing.ts`:
- Around line 50-198: getCommonAggregationFields() is duplicated here and in
apps/worker/src/services/project-stats-aggregator.ts; extract it into a single
shared module (e.g., export a function from `@llmgateway/db` or a new shared util)
and have both places import it. Move the full implementation (all aggregation
fields) into the new module as an exported function (retain the name
getCommonAggregationFields or a clear alias), update both callers to import and
use that exported function, and remove the local duplicate definitions; ensure
the module exports the same typed SQL fragments so existing callers (in
testing.ts and project-stats-aggregator.ts) compile without other changes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: a5adc6a1-e331-4fc3-a363-9b0440803e21
📒 Files selected for processing (4)
apps/api/src/testing.tsapps/worker/src/services/project-stats-aggregator.tspackages/db/migrations/meta/_journal.jsonpackages/db/src/schema.ts
Summary
imageInputTokensandimageOutputTokenscolumns to all 4 hourly aggregation tables (project_hourly_stats,project_hourly_model_stats,api_key_hourly_stats,api_key_hourly_model_stats)logtableTest plan
pnpm buildpassespnpm formatpasses🤖 Generated with Claude Code
Summary by CodeRabbit