🐋 OrcaID

A browser-only TypeScript app that identifies orcas from a photo. Both deep learning models run client-side via onnxruntime-web — no server, no upload of your images anywhere.

Pipeline

1. Upload an image (src/main.ts).
2. YOLOv8-small detects orca parts and returns box coordinates (src/yolo.ts).
3. Crop each box out of the image, in memory, on a canvas (src/imageUtils.ts).
4. EfficientNet-B0 turns each crop into an L2-normalized embedding vector (src/embedder.ts).

src/pipeline.ts wires the four steps together.

Run it

npm install
# add the two .onnx files — see public/models/README.md
npm run dev

Then open the printed localhost URL, choose an image, and click Run pipeline.

How to query the embedding vectors

The embedder outputs a 1280-d L2-normalized vector per crop, so cosine similarity == dot product. To answer "which known whale is this?" you do a nearest-neighbour search over a catalogue of known embeddings. Pick by scale:

Catalogue size	Recommendation	Why
< ~50k vectors (typical orca catalogues)	SQLite + sqlite-vec	One file, zero infra, brute-force cosine is exact and plenty fast at this size. Trivial to ship next to this app.
Postgres already in your stack	pgvector	Keep whale metadata (sightings, dates, IDs) and vectors in one DB; HNSW index for speed.
Millions of vectors / managed	Qdrant, Weaviate, or Pinecone	Purpose-built ANN indexes, filtering, horizontal scale.
Pure in-browser demo	hnswlib-wasm or a flat array	No backend at all; brute-force dot product in JS is fine for a few thousand.

sqlite-vec — it's already wired up in server/. Orca photo-ID catalogues are small (hundreds to a few thousand individuals), the search is exact, and there's nothing to operate. Migrate to pgvector only when you want to join embeddings against rich sighting metadata, and to a dedicated vector DB only if you ever reach millions of vectors.

The catalogue server (server/)

Since sqlite-vec is a native SQLite extension it runs in Node, not the browser. The flow is: browser computes a vector → POSTs it to a tiny API → sqlite-vec does the nearest-neighbour search → returns the matching whale.

• server/db.ts — schema, enroll(), and identify() (cosine KNN aggregated per whale)
• server/server.ts — POST /enroll, POST /identify, GET /stats
• server/seed.ts — bulk ingestion script

# Try the whole loop with fake data first (no models needed):
npm run seed -- --demo      # seeds 4 whales × 3 crops of random vectors
npm run server              # API on http://localhost:8787

# In another terminal: npm run dev, upload an image, then use the
# "Enroll" / "Identify" buttons under each detected crop.

Real ingestion reads a JSONL file, one crop per line:

{"whaleId":"T046","part":"dorsal-fin","embedding":[0.013, ...1280 floats...],"source":"img1.jpg"}

npm run seed -- catalogue.jsonl

identify() pulls the k nearest crops, groups them by whale_id, and returns each candidate's best/mean cosine similarity and vote count. Threshold the best similarity, and you have your ID (or "new individual" if nothing is close enough). The vectors are L2-normalized, so the vec0 table uses distance_metric=cosine and similarity = 1 - distance.

Notes

• CLASS_LABELS in src/main.ts must match the classes your YOLOv8 model was trained on.
• onnxruntime-web's wasm binaries are loaded from a CDN (src/onnx.ts); switch ort.env.wasm.wasmPaths to a local copy for fully offline use.