💳 CardWise

A retrieval-augmented generation (RAG) assistant that answers questions about credit cards grounded in the official card documents — with mandatory source citations and a "don't answer if the docs don't say so" guardrail against hallucination.

Ask things like "What is the cashback rate for Axis ACE, and any exclusions?" and get an answer drawn only from the ingested PDFs, with the card name cited.

---

Contents

• Features
• Demo
• How it works
• Agentic mode
• Tech stack
• Project structure
• Cards currently ingested
• Setup
• Configuration
• Usage
• Evaluation
• Guardrails
• Design decisions
• Limitations & roadmap
• Tests
• Contributing
• License

---

Features

• Hybrid retrieval — dense (semantic) + BM25 (keyword) fused with Reciprocal Rank Fusion, so exact-value queries ("what's the fee cap?") and conceptual ones both work.
• Grounded, cited answers — every response is drawn only from the ingested PDFs and cites the card it came from; the model declines when the documents are silent.
• Hallucination guardrails in depth — an input quality gate, a citation-enforcing prompt, and a numeric-claim validator (see Guardrails).
• Measured, not assumed — a 20-query gold set scored with RAGAS plus a deterministic numeric-exact guard; retrieval config chosen from the numbers (see Evaluation).
• Agentic mode — a LangGraph ReAct agent that routes between a document-search tool and a deterministic rewards calculator, with a step cap + RAG fallback encoded in the graph and a collapsible tool-call trace in the UI (see Agentic mode).
• Interactive demo — a Streamlit chat UI with live retrieval-mode switching and a per-card filter.
• CI-friendly tests — fast, API-free unit tests that run on every push.

Results at a glance (20-query gold set, hybrid @ k=10): context recall 0.883, numeric-exact retrieval 1.000, 44 retrieval/eval + 29 agent unit tests (all API-free).

---

Demo

▶ Try it live: cardwise-ai.streamlit.app

How it works

1. Ingest — extract text from card PDFs, split into overlapping chunks, embed with gemini-embedding-001, and store in a persistent Chroma collection with per-card metadata.
2. Retrieve — fetch relevant chunks via MMR (semantic + diversity) or Hybrid (BM25 + semantic, fused with Reciprocal Rank Fusion); switch in the sidebar. Hybrid is the benchmarked, recommended mode for exact-value and comparison queries.
3. Generate — pass the chunks to gemini-2.5-flash through a citation-enforcing prompt and stream the answer.
4. Evaluate — score with RAGAS + a deterministic numeric-exact guard. See Evaluation.

Agentic mode

For questions that need computation or a decision rather than a lookup — "I spend ₹10,000/month on Swiggy, which card nets me more?" — CardWise can run as a LangGraph ReAct agent that chooses between two tools instead of doing a single retrieval pass. Select Agent in the sidebar to try it.

graph TD;
    __start__([start]) --> agent(agent);
    agent -. end .-> __end__([end]);
    agent -.-> fallback(rag_fallback);
    agent -.-> tools(tools);
    tools --> agent;
    fallback --> __end__;

Loading

Two tools, routed by intent (the LLM picks by reading each tool's docstring — routing is a documentation problem first):

Tool	Use for	Why
card_search	Factual lookups (rates, caps, fees, exclusions)	Wraps the hybrid retriever at k=10; returns cited chunks
rewards_value	"Which card earns/saves more" over a spend profile	Pure, deterministic math — LLMs are unreliable at arithmetic, so a wrong fee is delegated to code and unit-tested

Safety lives in the graph topology, not the prompt:

• Step cap (MAX_STEPS) on the conditional edge — the loop cannot be talked past stopping.
• RAG fallback node — if the loop is cut off, the user still gets a plain-retrieval answer, never a stack trace.
• Tool errors become messages the model can react to (handle_tool_errors=True); and all reward math must come from rewards_value — the model is forbidden from computing in prose.

Honest advice, computed deterministically. rewards_value returns a summary that flags when a technically-correct ranking would still mislead — every card net-negative, all cards earning the same rate (a fee-only comparison), or a tie — and the agent must surface it.

Print a full agent trace from the CLI (spends API quota, so run it yourself):

PYTHONPATH=. .venv/bin/python -m src.scripts.agent_trace

Tech stack

• LangChain (core, text-splitters, community, experimental) — pipeline orchestration
• LangGraph — the agent's ReAct state graph (tools, step cap, fallback)
• Google Gemini — gemini-embedding-001 (embeddings) + gemini-2.5-flash (generation & eval judge)
• Chroma — local persistent vector store
• PyMuPDF — PDF text extraction
• RAGAS — RAG quality evaluation
• rank-bm25 — keyword scoring for the hybrid retriever's sparse leg
• Streamlit — interactive demo frontend

Project structure

cardwise-ai/
├── app.py                       # Streamlit chat UI (demo frontend)
├── data/raw/                    # source card PDFs
├── src/
│   ├── ingestion/
│   │   ├── loader.py            # PDF → text + quality_check
│   │   └── chunker.py           # chunk_document() + benchmark_chunking()
│   ├── vectorstore/
│   │   └── embedder.py          # build_vectorstore() / load_vectorstore()
│   ├── rag/
│   │   ├── chain.py             # build_card_rag_chain() / build_answer_chain()
│   │   └── prompts.py           # citation-enforcing advisor prompt
│   ├── retrieval/
│   │   └── retriever.py         # hybrid BM25 + semantic retriever, RRF fusion
│   ├── agent/                   # agentic mode (LangGraph ReAct)
│   │   ├── graph.py             # agent<->tools loop, step cap, RAG fallback
│   │   └── tools/
│   │       ├── retriever_tool.py  # card_search (hybrid lookup)
│   │       └── rewards_calc.py     # rewards_value (deterministic calculator)
│   ├── evaluation/
│   │   ├── eval.py              # RAGAS eval harness (Gemini judge) + gold-set runner
│   │   ├── gold_set.json        # 20-query evaluation gold set
│   │   └── numeric_validator.py # numeric-claim extraction + numeric-hit guard
│   └── scripts/
│       ├── ingest_all.py        # build the vector store from all cards
│       ├── run_queries.py       # run sample queries from the CLI
│       ├── run_hybrid.py        # try the hybrid / stratified retrieval
│       └── agent_trace.py       # print a README-ready agent tool-call trace
└── requirements.txt

Cards currently ingested

Card	Issuer	Type	Annual fee
SBI SimplyCLICK	SBI Card	online-shopping	₹499
Axis ACE	Axis Bank	cashback	₹499
HDFC Millennia	HDFC Bank	cashback	₹1000

Setup

# 1. Create a virtual environment and install dependencies
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

# 2. Add your Gemini API key (never commit this file)
echo "GOOGLE_API_KEY=your_key_here" > .env

.env is gitignored — the key is read from the environment at runtime and is never hardcoded or sent to the browser.

Configuration

Sensible defaults are baked in; the knobs worth knowing:

Setting	Default	Where
GOOGLE_API_KEY	— (required)	.env / environment
Embedding model	gemini-embedding-001	EMBEDDING_MODEL in embedder.py
Generation & judge model	gemini-2.5-flash	chain.py / eval.py
Retrieval budget k	10	as_runnable in retriever.py
RRF constant rrf_k	60	reciprocal_rank_fusion in retriever.py
Chroma persist dir	chroma_db/	persist_dir in embedder.py

Note: the store must be built and queried with the same embedding model — changing EMBEDDING_MODEL requires re-running the ingest.

Usage

# Build the vector store (run once, or after changing the card set)
PYTHONPATH=. .venv/bin/python -m src.scripts.ingest_all

# Launch the interactive frontend  →  http://localhost:8501
PYTHONPATH=. .venv/bin/streamlit run app.py

# Run sample queries from the terminal
PYTHONPATH=. .venv/bin/python -m src.scripts.run_queries

# Try the experimental hybrid + stratified retrieval
PYTHONPATH=. .venv/bin/python -m src.scripts.run_hybrid

Evaluation

The RAG pipeline is scored with RAGAS using gemini-2.5-flash as the judge and gemini-embedding-001 for the embedding-based metric. Each question is scored across four metrics, all ranging 0 → 1 (higher is better):

Metric	What it measures	Inputs used
Faithfulness	Is every claim in the answer supported by the retrieved context? (anti-hallucination)	answer + contexts
Answer relevancy	Does the answer actually address the question asked?	question + answer
Context precision	Of the chunks retrieved, how many are actually relevant? (retrieval signal-to-noise)	question + contexts + reference
Context recall	Did retrieval surface all the context needed to cover the reference answer?	question + contexts + reference

# Run the baseline eval (scores all questions; pass a smaller limit for a quick run)
PYTHONPATH=. .venv/bin/python -m src.evaluation.eval             # MMR,    week1 (5 Q)
PYTHONPATH=. .venv/bin/python -m src.evaluation.eval hybrid      # hybrid, week1 (5 Q)
PYTHONPATH=. .venv/bin/python -m src.evaluation.eval hybrid gold # hybrid, 20-query gold set

Gold set (20 queries)

Retrieval is measured against a fixed 20-query gold set (gold_set.json) — a test matrix of three cards × three failure modes, scored by the same run_baseline_eval:

Type	Count	What it stress-tests	Retrieval leg exercised
numeric	7	Exact values (4%, Rs. 500, 10,000)	BM25 / keyword — where pure-dense fails
comparison	7	"Which card for X" across cards	Card stratification — surface all relevant cards
conceptual	6	How / eligibility / redemption	Dense / semantic — meaning, not tokens

• Numeric is over-weighted — pure-semantic can't tell "5%" from "1%", so those queries are where hybrid must prove itself.
• Every answer is grounded in the PDF text — annual-fee/FX queries were excluded (those live in metadata, not chunks), so the gold answer measures the retriever, not the corpus.
• expected_card_ids + expected_numeric drive deterministic, no-LLM checks (ID-precision and the numeric_hit "zero fabricated fees" guard).
• Comparisons span 2–3 cards (c7 all three), doubling as stratification checks.

Before / after: dense-only vs hybrid (gold set, k=10)

Same 20 queries, same retrieval budget (k=10) — only the fusion differs. RAGAS judge, higher is better:

Retriever	Ctx Precision	Ctx Recall	Numeric-exact
Dense only (semantic)	0.630	0.829	1.000
Hybrid (RRF)	0.619	0.883	1.000

Takeaway: hybrid lifts context recall 0.829 → 0.883 (clears the ≥ 0.85 target) at equal precision. Numeric-exact is 1.000 for both — at k=10 a wide net over three cards always contains the number, so the metric saturates; BM25's edge is a ranking effect visible at smaller k. k=10 is the smallest budget that clears recall (up from 0.792 at k=6); stratification was tried and rejected — it starved single-card queries.

Week-1 smoke scores (5 questions)

An earlier, smaller sanity check across retrieval methods:

Metric	MMR	Hybrid
Faithfulness	0.903	0.897
Answer relevancy	0.940	0.958
Context recall	1.000	1.000
Context precision	0.618	0.783

Overall: both methods perform comparably — full context recall and strong, well-grounded answers — with hybrid edging out MMR (keyword matching locks onto the exact terms asked).

Agent-level evaluation (tool routing + answer)

The gold set above scores the retriever; this scores the agent's decision layer. 10 cases (4 factual, 3 compute, 3 eligibility) check two things per query: whether the agent routes to the expected tool, and whether the final answer contains the expected fact (src/evaluation/agent_eval.py):

Metric	Score
Tool-routing accuracy	100% (10/10)
Answer check	90% (9/10)

Every factual query routes to card_search, every value/comparison query to rewards_value, and every approval query to eligibility_check. The one answer-check miss is a substring-match artifact — the likely verdict was correct but the writer node paraphrased the word — not a routing or correctness failure.

Run it (spends API quota):

PYTHONPATH=. .venv/bin/python -m src.evaluation.agent_eval

Guardrails

The domain is financial — a wrong fee is a liability — so hallucination is defended at three points, not one:

1. Input gate — quality_check() (loader.py) requires a concrete earning rate (regex: 5%, 10X, ₹500) before a PDF is embedded; in strict mode it fails closed. Runs with zero API cost:

   PYTHONPATH=. .venv/bin/python -m src.scripts.ingest_all --check-only

2. Generation — a citation-enforcing prompt (prompts.py) makes the model answer only from context, cite the card, and decline when the docs are silent.

3. Output — in numeric_validator.py, extract_numeric_claims() pulls numeric claims and numeric_hit() deterministically checks the needed figure was retrieved (it powers the eval's numeric-exact metric; wiring it into the live answer path is next).

Regex layers are linear and length-capped to avoid catastrophic backtracking (ReDoS) on untrusted PDF/LLM text.

---

Design decisions

Deliberate choices, and the failure mode each avoids:

• Hybrid, not pure-semantic. Embedding search ranks "$395" and "$350" as near-equal, silently returning wrong amounts on exact-value queries; keyword scoring fixes it — measured (recall 0.829 → 0.883 vs dense-only, numeric-exact 1.000), not assumed.
• Card-stratified retrieval. stratified_retrieve() pulls top chunks per card so "which card for X?" isn't biased toward the longest PDF.
• One embedding model as source of truth. EMBEDDING_MODEL is defined once; build and query must match, or vector distances are meaningless with no error to warn you.
• Indian-card corpus. US card PDFs omit reward rates; the Indian T&Cs contain them — so the quality gate and the assistant have concrete facts to work with.
• RRF, not weighted-sum. BM25 and cosine scores aren't on comparable scales, so blending needs unstable normalization. RRF fuses on rank only (score = Σ 1/(rrf_k + rank)) — no normalization, one knob (rrf_k). Weighted-sum stays behind fusion="weighted".

Vector store: Chroma, for now

Chroma fits this stage: zero-ops, local, <100k chunks — retrieval quality is the bottleneck, not indexing. Migrate to Pinecone/Weaviate at ~500k+ chunks, multi-tenant needs, or managed HNSW for uptime — an operational win, not better recall. Low-risk: retrieval sits behind the single HybridRetriever interface.

---

Limitations & roadmap

Known limits, stated honestly:

• Demo-scale corpus. Three cards are ingested. The architecture scales, but retrieval numbers are reported on a small corpus — e.g. numeric-exact retrieval saturates at high k because a wide net over three cards always contains the answer.
• Hybrid is benchmarked but not the UI default. The sidebar still defaults to MMR; making the benchmarked hybrid retriever the default is a one-line change under evaluation.
• Numeric guard is eval-time only. numeric_hit() runs in the evaluation harness; cross-checking numeric claims on the live answer path is not wired yet.
• Agent calculator uses illustrative rates. rewards_value hardcodes demo cashback-equivalent rates and simplified monthly caps (the mechanism and its determinism are the point); wiring the rates from the card metadata/documents is next.
• rrf_k not swept. The RRF constant is fixed at 60; a sweep (10 / 60 / 100) against the gold set could squeeze out more recall.
• LLM-judge variance. RAGAS metrics occasionally return IncompleteOutputException (judge truncation), so a metric may score on n < 20 for a given run.

---

Tests

Fast, API-free unit tests run in CI with no key or quota. They pin what regresses silently: the quality gate's fail-closed contract, numeric extraction/guard, hybrid (RRF) ranking, and the agent layer — the deterministic rewards calculator (including its input validation and net-negative/fee-only/tie caveats), card_search wiring, and the graph's routing, step cap, and error recovery.

# Install dev dependencies (runtime + pytest) and run the suite
.venv/bin/pip install -r requirements-dev.txt
PYTHONPATH=. .venv/bin/pytest -q

CI runs this suite on every push and pull request to main (see the CI badge above), and a ruff lint + format check gates merges to main.

---

Contributing

1. Fork and create a feature branch off main.
2. Install dev dependencies: .venv/bin/pip install -r requirements-dev.txt.
3. Before opening a PR, run the checks that CI runs:

   PYTHONPATH=. .venv/bin/pytest -q
   ruff check . && ruff format --check .

4. Keep new logic covered by an API-free unit test where possible.

---

License

Released under the MIT License.