Cesto

Ask your grocery receipts anything.

Cesto is a Kotlin Multiplatform app (Android + iOS) that finds your Continente grocery receipts in Gmail and lets you ask questions about them in plain language:

"How much did I spend in August?" "How much sugar did I buy last month?" "Did I buy anything with lactose?"

Status: early development. Done so far: the modular KMP architecture, the light/dark theme, onboarding with the Gmail permission on Android and iOS, the receipt sync (Cartão Continente receipt PDFs from Gmail, their text read on the phone), the Gemini key in Settings, LLM extraction of the items, and the Ask tab with an agent loop over SQL tools, and RAG: products found by meaning with Gemini embeddings (see Roadmap).

Screenshots

Onboarding · Ask · Receipts · Settings (adding a Gemini key) · Settings (key saved)

How it will work

1. Connect Gmail. Google's own dialog asks for read-only access (gmail.readonly). ✅
2. Find receipts. Search Cartão Continente emails (noreply@cartaocontinente.pt) from the last 3 months, up to 4 emails in parallel, and download the receipt PDFs. ✅
3. Extract once. PDF → text on the phone (date, total, receipt number, ATCUD) ✅ → Gemini turns only the item section into structured lines (products, quantities, prices), stored in the local database. ✅
4. Ask. Gemini answers by calling a few tools (findProducts, semanticSearch, sumQuantity, sumSpending, topProducts, listReceipts); the app runs them as SQL on the phone and only sends the small results back. Numbers come from SQL, not from the LLM's arithmetic. ✅

The agent loop

It's a loop, not one call (AskQuestionUseCase): the LLM picks a tool, the app runs it on the phone and sends back only the small result, and the LLM decides the next step, until it answers with text instead of a tool call (at most 30 rounds). The receipt text is used once, to fill the product and receipt_item tables; questions only work on those tables. Each answer shows how many receipts it came from; errors (no key, quota, Gemini busy, offline) are shown in the conversation with a way to fix them.

RAG: finding products by meaning

Receipts never say "dairy" or "snacks", so keyword search can't find those products. Semantic search can:

1. Making vectors. After each sync, only new products (name + category, never prices or receipt data) go to Gemini's embedding API (gemini-embedding-2, 768 numbers), up to 100 per request. Each product's vector is stored on the phone in a separate product_embedding table (one row per product and model). Products that already have a vector are skipped, so vectors never change when new receipts arrive.
2. Searching. For a question like "How much did I spend on dairy?", the agent calls semanticSearch("dairy"). The phone embeds the search words, compares them with every product vector (cosine similarity in plain Kotlin, no vector database), and returns the closest products' ids and names.
3. Deciding and counting. The result goes back to the agent, which keeps only the products that really match and calls the SQL tools (sumSpending, sumQuantity) to count them.

Vectors find which products, SQL counts how much, and Gemini only reads names and totals, never vectors. Gemini embeddings run in Google's cloud with your own key; an on-device model (EmbeddingGemma) can replace them later behind the same EmbeddingProvider interface.

Daily spending notification (planned, Android)

Every morning around 09:00: "Yesterday you spent 23,45 € (3 receipts)".

1. Scheduling. Koin only creates DailySpendingScheduler; CestoApp.onCreate() calls schedule(), a unique 24 h periodic job with KEEP, so starting the app again changes nothing. WorkManager keeps the job across reboots and updates; after a force-stop, the next app start schedules it again.
2. The worker. DailySpendingWorker syncs new receipts from Gmail (silent token; if that fails, it uses the receipts already stored), sums yesterday's total_cents with SQL, and posts a notification unless there were no receipts.
3. Staying at 09:00. A periodic job counts the next 24 h from when the last run finished, so delays (Doze) and clock changes would make it drift. Right before Result.success(), the worker calls pinNextRun(id) (setNextScheduleTimeOverride) to move the next run back to tomorrow 09:00. It never enqueues itself again.

The sum is a shared use case, so iOS can reuse it later (BGAppRefreshTask + a local notification).

Accounts and sign-in (planned)

Sign in with Google through Google Cloud Identity Platform, so one phone can hold several accounts and, later, buy Gemini credit.

Sign in ─► Google ID token (Android Credential Manager · iOS GoogleSignIn, audience = Web client ID)
             └─► accounts:signInWithIdp ─► Identity Platform user (localId) + idToken (1 h) + refreshToken
Connect Gmail ─► GmailAuthorizer for that account (as today) ─► Gmail token stays on the phone
API calls     ─► validIdToken(): refreshed through securetoken.googleapis.com when it expires

1. :account data layer. IdentityPlatformApi (Ktor: signInWithIdp, token refresh), a SessionStore with every signed-in account and the active one (refresh tokens encrypted: Android Keystore / iOS Keychain), and AccountRepository (signIn, accounts, activeAccount, switchTo, validIdToken, signOut).
2. Platform sign-in. ✅ Onboarding's Continue with Google signs in, then asks for Gmail. Android Credential Manager ("Sign in with Google") and iOS GoogleSignIn return the Google ID token; Gmail permission stays a second step (Google skips the account picker the second time).
3. One database per account. cesto-<localId>.db, so receipts never mix; an account switcher in Settings; background work runs for the active account.
4. Gemini credit (later, separate). A backend (Cloud Run) verifies the idToken, checks Play Billing / StoreKit purchases, keeps a credit ledger per localId and calls Gemini with its own key. App Check (Play Integrity / App Attest) protects it. Your own key stays as a free option.

The Identity Platform API key isn't a secret (it ships in every app); it's restricted to the Identity Toolkit and Token Service APIs, and real protection comes from Google ID tokens accepted only from our own OAuth clients, and later App Check.

Privacy first

• Receipts, PDFs and the database stay on the phone. The account (Identity Platform) only knows your Google identity, never receipts or Gmail tokens.
• The Gmail token never leaves the device; the app never sends, deletes or changes emails.
• The LLM never sees the database or raw emails, only small tool results.
• Gemini runs with your own API key (Settings): it's stored encrypted on the phone (Android Keystore) and only sent to Google. No key ships in the app, and there's no backend in between.

Architecture

Feature modules with Clean Architecture inside each feature. One composition root (:app) knows every module; everything else only knows what it needs.

Module	Contains
androidApp	Android entry point: CestoApp starts Koin, MainActivity, AndroidGmailAuthorizer (Google Identity AuthorizationClient) and AndroidGoogleIdTokenProvider (Credential Manager)
iosApp	iOS entry point (SwiftUI), GoogleGmailAuthorizer and GoogleIdTokenSignIn (GoogleSignIn, Swift Package)
:app	Composition root: App() with CestoTheme, the bottom bar (Ask · Receipts · Settings) and the NavHost, initKoin() with all Koin modules; builds the iOS framework Shared
:feature:onboarding	First screen: one card with the logo, the app name and Continue with Google (sign in, then Gmail permission)
:feature:receipts	Receipt sync (Gmail REST with Ktor → database → PDF files), item extraction with Gemini, and the receipt list with pull to refresh
:feature:chat	Ask tab: the agent loop (AskQuestionUseCase), Gemini function calling, the SQL tools and the chat screen
:feature:settings	Settings tab: the signed-in account with Sign out (back to onboarding), your own Gemini key in two steps, tested before saving, and an Enable billing link; later Gmail and privacy
:gmail-auth	The Google SDK interfaces the platform apps implement: GmailAuthorizer (Gmail permission) and GoogleIdTokenProvider (Sign in with Google)
:systemdesign	CestoTheme: light and dark colour schemes; components used by 2+ features (CestoScreenTitle)
:core	Non-UI shared code: the Ktor HttpClient (OkHttp / Darwin engine) and its Koin module
:llm	Gemini with the user's own key: encrypted key storage (Android Keystore), the Gemini REST client (key check, structured output, function calling) and EmbeddingProvider (Gemini embeddings)
:account	Cesto accounts (Identity Platform): the REST client (signInWithIdp, token refresh), every signed-in account with encrypted refresh tokens (Android Keystore), AccountRepository; in Koin, not used by a screen yet
:database	Local SQLite database (SQLDelight): receipts, products, receipt lines, product vectors and the Gmail messages already checked; receipts fills it, the chat's read-only queries (Insights.sq) use it

Rules

• Features never depend on each other. :app connects them with navigation lambdas (onConnect, onSignedOut); each feature owns its route.
• Floating pill bottom bar in :app (TopLevelDestination): tabs draw behind it and get a contentPadding so their last item ends above it. The selected tab is derived from the back stack, each tab keeps its state when you switch, and back goes to Ask, then closes the app. Onboarding and demo mode are outside the bar.
• Shared modules never depend upward, and :systemdesign and :core don't know each other.
• Inside a feature: presentation → domain ← data. The domain is plain Kotlin; the data layer implements the domain's repository interfaces; ViewModels only call use cases.
• Screen state lives in the ViewModel (StateFlow), composables are stateless (Route + Screen).
• Platform SDKs stay at the edge. Where the iOS SDK is only practical from Swift (GoogleSignIn), the shared code defines an interface and the platform apps implement it and pass it to initKoin().

Gmail permission

Gmail access is an OAuth scope, not an OS permission: Google's SDK shows the consent dialog and returns an access token, which Ktor sends with each Gmail REST call.

Connect Gmail ─► GmailAuthorizer (androidApp: AuthorizationClient · iosApp: GoogleSignIn)
                     └─► Google's account picker + consent ─► access token (stays on the phone)
App start     ─► same call without UI: already granted? ─► skip onboarding
Gmail request ─► same call without UI ─► "Authorization: Bearer <token>" (never asks the user)

Receipt pipeline: PDF → text → items

The sync stays the single entry point (SyncReceiptsUseCase). ReceiptRepositoryImpl coordinates five steps per receipt: download the PDF from Gmail, save the file, extract its text on the phone (PdfBox-Android / PDFKit), parse the fixed fields of a Continente receipt (date, total, receipt number, ATCUD), and save them. Yellow boxes were added with this step. Receipts downloaded by an earlier app version are read from their saved file, without asking Gmail again.

Then item extraction: only the item section (from the DESCRICAO header to SUBTOTAL / TOTAL A PAGAR, long numbers masked) goes to Gemini with a fixed JSON format. Each line becomes a receipt_item, and each unique printed name a product (full Portuguese name, category, units per pack). The lines' sum is checked against the receipt's subtotal. Gemini overloaded → retry, then a lighter model; key or quota errors pause extraction until the next sync.

Sync: Gmail loop and Gemini loop

Each sync runs two loops. The Gmail loop searches for Continente receipt emails page by page (ids only) and keeps the emails that are new or still have a FOUND receipt. They're handled 4 at a time: each email is read once, its PDF parts are saved as FOUND receipts, and they're downloaded with the attachment ids from that same response, stored and read on the phone. The next page is fetched when every batch is done. Then the Gemini loop extracts the items of the receipts with text, 4 at a time from the database. Every Gmail request goes through KtorGmailApi.

Tech stack

Area	Library
UI	Compose Multiplatform 1.12 (Material 3), Navigation Compose 2.9
DI	Koin 4.2 (koin-compose-viewmodel)
Network	Ktor 3.6 (OkHttp on Android, Darwin on iOS), kotlinx.serialization
Gmail permission	Google Identity AuthorizationClient (play-services-auth 22) · GoogleSignIn-iOS 10
Sign-in	Credential Manager 1.6 + googleid 1.2 (Sign in with Google) · GoogleSignIn-iOS · Identity Platform REST via Ktor
Database	SQLDelight 2.4 (Android driver, iOS native driver with the system SQLite), migrations in .sqm files
PDF text	PdfBox-Android 2.0.27 · PDFKit (iOS, called from Kotlin/Native)
Dates	kotlinx-datetime 0.8
Tests	kotlin.test, kotlinx-coroutines-test, Ktor MockEngine, in-memory SQLite; shared tests run on Android and iOS
LLM	Gemini REST API via Ktor (structured output, function calling), with the user's own key (Android Keystore: AES-GCM)
RAG	Gemini embeddings (gemini-embedding-2), vectors as BLOBs in SQLite, cosine similarity in plain Kotlin
Planned	Open Food Facts, iOS Keychain for the key, on-device embeddings (EmbeddingGemma)

Kotlin 2.4 · Android minSdk 24, targetSdk 37 · iOS 18.2+

Getting started

Requirements

• Android Studio with the Kotlin Multiplatform plugin; JDK 21 (Gradle downloads it if missing)
• Xcode 16+ for iOS

Google Cloud setup (Gmail permission)

The app needs OAuth clients in a Google Cloud project:

1. Enable the Gmail API.

2. Google Auth Platform: consent screen External, publishing status Testing, add your Google account as a test user, and add the scope .../auth/gmail.readonly. gmail.readonly is a restricted scope: Testing mode (up to 100 test users) avoids Google's verification.

3. Clients:

   • Android: package com.majidbahmani.cesto + your debug keystore's SHA-1 (./gradlew :androidApp:signingReport). Nothing to copy into the project.
   • iOS: bundle ID com.majidbahmani.cesto.Cesto. Copy iosApp/Configuration/Secrets.xcconfig.example to Secrets.xcconfig (gitignored) and fill in the client ID and its reversed form.

4. Sign-in (Identity Platform, planned): enable Identity Platform, add the Google provider with a Web application OAuth client's ID and secret (no redirect URI needed), and add the Android and iOS client IDs under Allowed client IDs. Check that the auto-created Browser key allows the Identity Toolkit API and Token Service API. Then fill in, both gitignored:

   • local.properties: IDENTITY_PLATFORM_API_KEY=... and GOOGLE_WEB_CLIENT_ID=... (the Web client's ID, the audience of Android's Google ID token). CI reads the same names from environment variables.
   • Secrets.xcconfig: IDENTITY_PLATFORM_API_KEY = ... (iOS uses its own iOS client ID). Before a Play release: add the Play App Signing SHA-1 to the Android client and restrict the key.

Without these the app still builds and runs; "Continue with Google" then shows an error.

Run

• Android: run androidApp from Android Studio, or ./gradlew :androidApp:installDebug
• iOS: open iosApp/iosApp.xcodeproj in Xcode and run (Xcode builds the Kotlin framework with :app:embedAndSignAppleFrameworkForXcode)

Test

./gradlew testAndroidHostTest
./gradlew iosSimulatorArm64Test

The same commonTest tests run on the JVM and on the iOS simulator: the Gmail access wrapper, the onboarding repository and ViewModel, the Gmail REST client (MockEngine) and message parsing, the database schema, migrations and queries (in-memory SQLite), item extraction, the agent loop (scripted fake model), the SQL tools, product vectors and semantic search, the Gemini request/response mapping, the ViewModels, the HTTP client, and the Koin graph (AppModulesTest, because Koin only reports missing bindings at runtime).

CI (.github/workflows/ci.yml), on every PR and every push to main: Changes → Build (debug APK) → Unit tests, Android Lint and Code style in parallel → CI result, the single check for branch protection. No secrets are needed: CI never calls Gmail or Gemini. Changes to Markdown, art/ or the license skip the build. The iOS tests run locally for now.

Code style is ktlint (Android Studio style) with the Compose rules, through Spotless:

./gradlew spotlessApply    # format before committing
./gradlew spotlessCheck    # what CI runs
./gradlew :androidApp:lintDebug

Release (GitHub only)

Pushing a version tag builds a signed release APK and publishes it as a GitHub Release (.github/workflows/release.yml); there's no Google Play release.

git tag v1.0.0
git push origin v1.0.0

• Build signed APK: v1.2.3 → versionName 1.2.3, versionCode 10203; the signing key comes from encrypted secrets, only in this job, and the signature is verified with apksigner.
• Publish GitHub Release: cesto-1.2.3.apk, SHA256SUMS.txt and notes from the merged PRs. The only job that can write, and it never sees the key.
• Without the signing secrets, a local ./gradlew :androidApp:assembleRelease just builds an unsigned APK.

Design decisions

Decision	Why
Data on the device, not in the cloud	Receipts contain the NIF and shopping habits; Gmail is the source of truth, so nothing needs a backup; keeping restricted-scope Gmail data off servers avoids Google's security assessment
No Cognito / app accounts	The Gmail token would end up on a server, and Cognito can't refresh Google tokens. One Google dialog is all the user sees
Identity Platform accounts (planned)	Several accounts on one phone and paid Gemini credit need a stable user id a backend can verify; Google sign-in only, and the Gmail token still never leaves the phone
Your own Gemini key, no backend	A key inside the app could be extracted and used on someone else's bill; with each user's own key (set up in two steps in Settings, tested before saving) no proxy or server is needed, and costs stay on the user's account. Billing is recommended: on the free tier Google may use prompts to improve its products
Extract once, then query with tools	Sending all receipt text per question is expensive and LLMs make arithmetic mistakes; SQL computes exact sums and scales to 10,000+ items
:gmail-auth as its own leaf module	Onboarding and receipts both need it, and features don't depend on each other
Platform implementations in the apps	GoogleSignIn is a Swift Package only practical from Swift; AuthorizationClient needs an Activity for the consent screen
Feature modules from the start	Onboarding, receipts and chat are separate features; the compiler enforces the dependency rules
Koin + Ktor	KMP-ready DI and HTTP; the engine is injected so tests use MockEngine
:database as a shared module	Receipts fill the database and the chat's SQL tools will query it; features can't depend on each other
Receipts keyed by Gmail messageId + partId	Gmail returns a different attachmentId on every request, so it can't identify a PDF; it's fetched fresh for each download
Sync: parallel, each email on its own	Up to 4 emails at once (a slow PDF doesn't block the others); each email is read once and its PDFs downloaded right away. An email is marked checked in the same transaction as its receipts; anything that failed is retried by the next sync, and only a failed search fails the whole sync
PDF → text on the phone, fixed fields without an LLM	The PDFs have a text layer; date, total, receipt number and ATCUD are on fixed lines, so they're read exactly and for free. Only the items (their order differs between PdfBox and PDFKit) need the LLM
Money in cents, receipt times in Lisbon time	Exact sums without floating point; the receipt's printed time is Portuguese local time wherever the phone is
Look-back period in the domain	"3 months" is a product rule (SyncReceiptsUseCase); the data layer turns it into Gmail syntax (newer_than:3m)
The list reads only the database	The sync writes, the screen observes (single source of truth); saved receipts stay visible offline or when Gmail fails
Relative PDF paths	iOS changes the app container's absolute path between installs and updates; PDFs are also excluded from iCloud backup
No Android backup (allowBackup=false, data extraction rules)	Receipt data never leaves the phone, not even to Google's backup or a new device; a new install syncs from Gmail again

Roadmap

• v1: end to end. Gmail search + PDF download + receipt list ✅ · PDF → text (date, total) ✅ · PDF viewer · LLM extraction of the items ✅ · Gemini key in Settings ✅ · chat with tool calling and an agent loop ✅ · Open Food Facts (sugar, ingredients, category) · demo mode with sample receipts · daily notification with yesterday's spending (WorkManager)
• v2: RAG. Gemini embeddings + semanticSearch tool ("dairy" finds IOG GREGO NAT) ✅ · nutrition guidelines
• Accounts. Identity Platform sign-in (:account) · one database per account + switcher · Gemini credit (backend, Play Billing, App Check)
• v3: on device. On-device embedding model (EmbeddingGemma), evaluation set, optional offline LLM, more stores

Receipts show what you bought, not what you ate: the app talks about "sugar in groceries purchased".

License

MIT © 2026 Majid Bahmani