Building An Agentic Pipeline for Evaluating Foundation Models under Dataset Shift - Brainhack Vanderbilt 2026

 ·  short name: Agentic Transfer Bench

Status: 0_concept_no_content. This repo currently holds documentation only. The code arrives in Phase 2 and exists to produce evidence.

Can an AI agent make machine-learning workflows easier to build and maintain?

This project explores the potential of agentic pipelines to automate complex, non-trivial decisions in ML workflows. We use one concrete task as a testbed: adapting an open PPG (photoplethysmography) foundation model to a new dataset.

Key questions:

• How can we quantify dataset shift?
• How should the degree of shift inform the choice of adaptation strategy, from linear probing to full fine-tuning?
• Can an LLM agent make these decisions reliably?
• Can it reduce human effort without introducing unacceptable errors? How do we define unacceptable errors?

At each decision point we build the step two ways — an LLM agent and a plain deterministic rule — and measure which wins, and at what cost. The outcome is a practical understanding of agentic pipelines, backed by our experiments and documented for future use.

PPG is just the example. The workflow — profile a dataset → quantify shift → choose an adaptation strategy → verify — is domain-general.

Proposed project structure

Phase	Output	Where
1 — Understand & document (priority)	A primer on agentic pipelines in our own words; a decision map for the PPG pipeline	docs/agentic-pipelines.md, docs/case-study-ppg.md
2 — Validate with a code prototype	1–2 decision points built agent-vs-rule, a decision log, a small eval harness, checked against a verification checkpoint before write-up	TASKS.md (findings write-up format TBD)

Read in order: TASKS.md (the phased plan — start here to see what to do and when) → docs/agentic-pipelines.md → docs/case-study-ppg.md → docs/ONBOARDING.md (accounts, data access, environment, needed for Phase 2 only). The Phase-2 code contract isn't written yet — Phase 1 decides the prototype's scope first.

git clone https://github.com/rgbayrak/Agentic-Transfer-Bench
cd Agentic-Transfer-Bench

Repo structure=

├── README.md                   <- you are here
├── TASKS.md                    <- 2-day plan, check-boxed
├── CONTRIBUTORS.md             <- who worked on it (add yourself with your first PR)
├── docs/
│   ├── agentic-pipelines.md      <- the concept, in our own words (primer + reading list)
│   ├── case-study-ppg.md         <- decision map: where does agency belong here?
│   ├── ONBOARDING.md             <- accounts, data access, git workflow, environment
│   └── public-data-download.md   <- DataLad walk-through for the 
├── data/                       <- gitignored; datasets land here in Phase 2
└── outputs/                    <- gitignored; metrics, plots, reports in Phase 2

Data

Two open PPG sources with different amounts of domain shift from wearable-PPG pretraining. Which is the lower-shift reference and which the harder target is a Phase-1 decision (docs/case-study-ppg.md §1).

• WildPPG (ETH Zurich, open, manual download) — wrist PPG collected in the wild with an ECG HR reference.
• OpenNeuro ds006644 (open, via DataLad) — scanner-session physiological recordings; BIDS *_physio.tsv.gz + .json with a cardiac / pulse-ox channel. Fetch only the *physio* files — see docs/ONBOARDING.md §2.1 and docs/public-data-download.md.

Foundation model: PaPaGei-S (Nokia Bell Labs), optionally Pulse-PPG.

See docs/ONBOARDING.md for download steps and the Colab option.

Contributors

Everyone who contributes documentation, code, or analysis is included as a co-author on any resulting write-up, preprint, or poster, and acknowledged via their commit history and a CONTRIBUTORS file.

License

GPLv3 for this repo's content (see LICENSE). Datasets and model weights retain their own upstream licenses — check before redistributing (see docs/ONBOARDING.md).