Project: Building An Agentic Pipeline for Evaluating Foundation Models under Dataset Shift #114

Title

Agentic Transfer Bench

Leaders

Roza G. Bayrak (@rgbayrak)

Collaborators

No response

Project description

Can an AI agent make machine-learning workflows easier to build and maintain?

This project will explore the potential of agentic pipelines to automate complex, non-trivial decisions in machine-learning workflows. We’ll use a concrete task as a testbed: adapting an open PPG foundation model to a new dataset.

Key questions include: How can we quantify dataset shift? How should the degree of shift inform the choice of adaptation strategy, from linear probing to full fine-tuning? Can an LLM agent make these decisions reliably? And can it reduce human effort without introducing unacceptable errors?

The final outcome will be a practical understanding of agentic pipelines, supported by our experiments and documented for future use.

Link to project repository/sources

https://github.com/rgbayrak/Agentic-Transfer-Bench

Concerete goals with specific tasks for Brainhack Vanderbilt 2026

We research and plan first; code last. We will write code to test what we learnt as a team.
Document → Implement → Test → Report

Phase 1 — Understand (research & writing track; everyone)

• Read the primary literature on agentic systems; write docs/agentic-pipelines.md, a plain-language primer covering the script→agent spectrum, agent anatomy, core vocabulary, when agency helps vs. hurts, patterns, anti-patterns, evaluation criteria, and an annotated reading list.
• Map the real pipeline in docs/case-study-ppg.md: for each step (fetch → parse → preprocess → embed → decide → adapt → verify), mark it deterministic or agentic, and write a testable hypothesis for why.

Deliverable: primer + decision map, reviewed as a team before any code is written.

Phase 2 — Build

Deterministic backbone (scaffolding):

• fetch_dataset — WildPPG and OpenNeuro ds006644
• parse_dataset, preprocess (resample / window / normalize)
• load_fm + embed (PaPaGei-S, optionally Pulse-PPG)
• train_linear_probe with subject-wise cross-validation
• compute_shift

Build a minimal eval harness that runs both variants (agent and rule) on ≥2 datasets and records outcome, decision quality, and cost (tokens / $ / wall-clock / step count).

Phase 3 — Report

• Write docs/findings.md: one section per decision point, structured as what we expected → what we saw → recommendation, ending with a revised, one-sentence definition of "agentic" based on what you actually found.

Deliverable: findings write-up; this is the artifact you'd present or submit.

Good first issues

1. Explore and document the vocabulary.

• Orchestrator / lead agent: What is an orchestrator? What role does it play in an agentic pipeline? How does it coordinate subtasks, tools, and other agents?
• Specialized sub-agents / workers: What makes a sub-agent “specialized”? How do sub-agents differ from ordinary functions or tools? When is using multiple agents useful?
• Deterministic tools & APIs: What distinguishes a tool or API from an agent? How do agents interact with deterministic tools? What role do tools such as APIs, scripts, and MCP play in an agentic workflow?
• Agentic decision-making: What makes a decision or workflow step “agentic”? How does this differ from a fixed, deterministic pipeline?
• Human-in-the-loop: What does it mean for an agent to remain in the loop without becoming autonomous? Where should humans review, approve, override, or redirect agent decisions?
• Agentic vs. autonomous: Are these actually different concepts? What level of autonomy is necessary for something to be considered agentic?

2. Identify and document the tools needed for the adaptation of a PPG time-series foundation model to a new, unseen dataset.

What tools are needed to:

• inspect and understand the new dataset?
• run experiments and evaluate models?
• measure and diagnose domain shift?
• evaluate possible adaptation strategies?

Which decisions should remain human-in-the-loop?

Skills

No prior agents or PPG experience required, the research-and-writing tasks are open to everyone!

"Working" = you've used it on a real task and can debug it. Only a subset of the team needs specialized skills.
Phase 1:

• Technical research and writing — working, everyone on Phase 1. Explain a concept plainly, cite sources, state a claim and its evidence.

Phase 2:

• Python — working, all Phase-2 contributors. Functions, dataclasses / type hints, pytest, reading someone else's module.
• Command line / dev environment — working, git (branch / rebase / PR), venv or conda, running scripts, ssh.
• Stats for dataset shift — working, 1–2 people.
• Working ML knowledge — working, 2–3 people.
• Deep-learning tooling — working, 1 person. Load a checkpoint, run a forward pass, batching, CPU/GPU device handling.
• Claude Code specifically — nice to have, lead + 1. Only the orchestration authors need the agent runtime; everyone else uses Copilot / Codex / Cursor / none.

Onboarding documentation

https://github.com/rgbayrak/Agentic-Transfer-Bench see README.md

What will participants learn?

By the end of BrainHack, you should be able to:

1. Define "agentic" precisely enough to argue about it, distinguish a script, a DAG, a tool-using LLM call, and an orchestrator, using consistent vocabulary (tool / skill / agent / orchestrator).
2. Communicate a technical finding in plain language, with citations and a clear claim/evidence structure.
3. Practice real collaborative engineering: forking, branching, pull requests, issues, and pulling only the data you need from a BIDS dataset via DataLad/git-annex.
4. Quantify dataset shift (covariate, label, and concept shift) using at least two of: MMD, proxy-A-distance, Population Stability Index, KL divergence.
5. Use frozen-embedding linear probing and light fine-tuning (LoRA) to adapt a foundation model to a new dataset.
6. Understand how to evaluate an agentic system on five axes: outcome, decision quality, cost, robustness, transparency (decision logging + deterministic fallback).

Public data to use

• WildPPG (https://siplab.org/projects/WildPPG)
• OpenNeuro ds006644 (https://openneuro.org/datasets/ds006644/versions/1.0.1)

Number of collaborators

4+

Credit to collaborators

Project collaborators will be included as co-authors on any resulting write-up, preprint, or poster, and acknowledged in the GitHub repository via their commit history and a CONTRIBUTORS file. Anyone who contributes documentation, code, or analysis counts.

Image

Project Summary

This project will explore when and how to leverage agentic AI in ML pipelines, using PPG foundation-model adaptation to a new dataset as a real-world testbed.

Type

method_development, pipeline_development, documentation

Development status

0_concept_no_content

Topic

machine_learning, deep_learning, physiology

Tools

Pytorch, Datalad, other

Programming language

Python

Modalities

Peripheral physiology

Git skills

0_no_git_skills, 1_commit_push, 2_branches_PRs, 3_continuous_integration

Anything else?

No response

Things to do after the project is submitted and ready to review.

• Add a comment below the main post of your issue saying: Hi @brainhack-vandy/project-monitors my project is ready!