Skip to content
Open

PFMs #378

Show file tree
Hide file tree
Changes from 58 commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
f92be17
PFMs pipeline, modified from Andrea's code, tested using Mac25Rhesus_…
Jan 7, 2026
dc7db84
forgot the actual pipeline script
Jan 7, 2026
3041214
add random seed
Jan 7, 2026
7346fbe
moved example script from pipeline dir to Examples/Scripts
Jan 20, 2026
98ee4c2
Added random seed and warm start optional parameters
Jan 20, 2026
359d752
remove script at old location
Jan 20, 2026
427e75a
Update PFM/PFMPipeline.sh
bqrosen Jan 20, 2026
bcef6f5
removed unnecessary unix call and cifti magic numbers
Jan 20, 2026
23260a9
fixed mandatory newopts syntax
Jan 20, 2026
e45ea93
nStarts 1000 in example script
Jan 30, 2026
2868e12
remove global DEFAULTS from batch script. Rename functions.
Feb 5, 2026
42f9c98
remove single regression option
Feb 5, 2026
da38924
remove single regression option
Feb 5, 2026
61dd140
improve commments
Feb 5, 2026
48f97b8
remove unnecessary extra script copies
Feb 5, 2026
4375d35
improve commenting
Feb 5, 2026
e3d1986
improve comments
Feb 5, 2026
47e2c86
fix step name
Feb 5, 2026
1c17127
minor fixes
Feb 5, 2026
586b2e7
move output string definition to batch script
Feb 5, 2026
f0573be
Consolidate output-string and output-prefix
Feb 5, 2026
5ce1478
remove low ica-dims arguments, because weighted regression no longer …
Feb 10, 2026
3d5eee2
clear PFM folder in RunPROFUMO step to prevent + file creation on reruns
Feb 16, 2026
08533a1
revise PFMdir clearing to leave in the dataLocations.json created by …
Feb 16, 2026
1d3a04e
add --loadSequentially argument to PROFUMO call by default, convert P…
Feb 24, 2026
ed0e699
change the cifti version conversion log message to warning
Feb 24, 2026
e354d0f
update PFM usages and refactor usages for newopts.
Feb 25, 2026
d1f57f9
re-write PROFUMO inputs (as cifti 2) with wb_command, unconditionally…
Feb 25, 2026
85cceed
Update PFM pipeline scripts
arkky68 Apr 10, 2026
9922f37
Update subject list and environment script path
arkky68 Apr 10, 2026
f7d1d3d
Update Examples/Scripts/RunPFMpipelineFullRun.sh
arkky68 Apr 15, 2026
5e26211
Update Examples/Scripts/RunPFMpipelineFullRun.sh
arkky68 Apr 28, 2026
8cf3c8a
Ignore local sif file
arkky68 Apr 28, 2026
1764c3f
Add Wishart filter prefiltering step to PFM pipeline
arkky68 May 8, 2026
78a12a9
Add NumWishart parameter to PFM launcher
arkky68 May 8, 2026
9b9618f
Fix matrix dimension issue in TCS calculation
arkky68 May 8, 2026
bb787d6
Update VolumeTemplateCIFTI path to use HOME directory
arkky68 May 8, 2026
73efef6
Remove LowDims variable from script
arkky68 May 8, 2026
1d5ad7c
Cleaning up and add comments
arkky68 May 8, 2026
cead749
Clean temp WF files before PROFUMO postprocessing
arkky68 May 8, 2026
4f8df64
Address review( move postprocessing to PostPROFUMO, fix descriptions,…
arkky68 May 11, 2026
64b0d78
Update with ApplyWFProfumo scripts
arkky68 May 24, 2026
63634d1
Address review: use concat file for WF, add per-run demeaning, fix Re…
arkky68 Jun 8, 2026
c377ddb
Correct Typo
arkky68 Jun 8, 2026
1c07db5
Make PRUFUMO container ignore the local python environmnet. Clean up…
Jul 7, 2026
ce5480e
Pfm Update with human subjects (#392)
arkky68 Jul 9, 2026
aad74dc
Merge remote-tracking branch 'origin/pr-392' into PFMs
Jul 9, 2026
774d7d4
Update tICA/scripts/tICACleanData.sh
bqrosen Jul 17, 2026
a888a26
Update PFM/PFMPipeline.sh
bqrosen Jul 17, 2026
d5edd76
Update PFM/PFMPipeline.sh
bqrosen Jul 17, 2026
1f7bcd9
default profumo-threads to # physical cores. add url to download .sif
Jul 17, 2026
0a7af01
update HCP-YA example parameters and filenames. fix function name whe…
Jul 20, 2026
acdca96
defuctionalize
Jul 20, 2026
ddb9f13
remove only "Results.ppp" dir
Jul 20, 2026
bb4826b
un-variance normalize and un-demean deconcatenated post-WF data
Jul 20, 2026
47a38db
moving the concatenation and deconcatenation to PFMPipeline.sh so App…
Jul 20, 2026
271f419
for Wishart filtering, Add un-variance normalization and un-demeaning…
Jul 22, 2026
5b48d6f
fix multi-run deconcatenation, and comment out single run for now
Jul 22, 2026
aa474a0
Update highpass filter and reference image in script
arkky68 Aug 4, 2026
9046688
Only re-use wishart filtered data if KeepWishart
Aug 4, 2026
4a109c7
fix VN file path
Aug 5, 2026
b1fd5f9
remove post-deconcatenation variance restoration operation (as coppie…
Aug 12, 2026
9863892
Update PFM/PFMPipeline.sh
bqrosen Aug 13, 2026
111dec7
Added RegString. Moved VAgray creation out into subject loop.
Aug 13, 2026
9c3d9b0
Update PFM/PFMPipeline.sh
bqrosen Aug 13, 2026
bd78fe2
Restore variance to maps after PROFUMO, bugfixes with VAweightBool ha…
Aug 18, 2026
3ba3bf5
standardize indentation
Aug 18, 2026
90f5b20
va_norm lives in /T1w
Aug 18, 2026
4d19505
Feed PostPROFUMO a VarNormBool, instead of a file becuase clean_VN is…
Aug 18, 2026
ae4f2af
Set Variance Normalization and Vertex area weighting defaults to YES.
Aug 19, 2026
9e66de1
refactor to fix indentation and rearrange code blocks for readability
Aug 19, 2026
d419332
use va_norm from MNINonLinear
Aug 31, 2026
9a9962a
Make va_norm with grayordinates a temporary file, like in RSNRegression.
Aug 31, 2026
e233cff
fix tempfile extension
Sep 1, 2026
9b88271
fix cifti template for creating VAgray
Sep 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
248 changes: 248 additions & 0 deletions Examples/Scripts/RunPFMpipelineFullRun.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
#!/bin/bash

set -eu

# This is an example script to run the full PFM postprocessing pipeline
Comment thread
bqrosen marked this conversation as resolved.
# Steps involved:
# 1. RunPROFUMO - Run PROFUMO analysis
# 2. PostPROFUMO - Create time courses, spectra, and maps from PFM results
# 3. RSNRegression - Run RSN regression on PFM data for dual regression
# 4. GroupPFMs - Generate group-level statistics and averages
#
# Please make sure that PROFUMO, ICA-FIX, MSMAll and MakeAverageDataset are done properly
# matching the input arguments before running this PFM pipeline

get_options() {
local scriptName=$(basename "$0")
local arguments=("$@")

# initialize variables
StudyFolder="${HOME}/data/HCPpipelines_ExampleData"
Subjlist="100610@102311"
EnvironmentScript="${HOME}/projects/HCPpipelines/Examples/Scripts/SetUpHCPPipeline.sh"
GroupAverageName="S1200_MSMAll7T175"

RegName="MSMAll"
MatlabMode=1
RunLocal=0
QUEUE="matlabparallelhigh.q"

# parse arguments
local index argument

for ((index = 0; index < ${#arguments[@]}; ++index))
do
argument="${arguments[index]}"

case "$argument" in
--StudyFolder=*)
StudyFolder="${argument#*=}"
;;
--Subject=*)
Subjlist="${argument#*=}"
;;
--EnvironmentScript=*)
EnvironmentScript="${argument#*=}"
;;
--GroupAverageName=*)
GroupAverageName="${argument#*=}"
;;
--RegName=*)
RegName="${argument#*=}"
;;
--MatlabMode=*)
MatlabMode="${argument#*=}"
;;
*)
echo "ERROR: Unrecognized Option: ${argument}"
exit 1
;;
esac
done

# check required parameters
if [[ "$StudyFolder" == "" ]]
then
echo "ERROR: StudyFolder not specified"
exit 1
fi

if [[ "$Subjlist" == "" ]]
then
echo "ERROR: Subjlist not specified"
exit 1
fi

if [[ "$EnvironmentScript" == "" ]]
then
echo "ERROR: EnvironmentScript not specified"
exit 1
fi

if [[ "$GroupAverageName" == "" ]]
then
echo "ERROR: GroupAverageName not specified"
exit 1
fi

# if [[ "$RegName" == "" ]]
# then
# echo "ERROR: RegName not specified"
# exit 1
# fi

if [[ "$MatlabMode" == "" ]]
then
echo "ERROR: MatlabMode not specified"
exit 1
fi

# report options
echo "-- ${scriptName}: Specified Command-Line Options: -- Start --"
echo " StudyFolder: ${StudyFolder}"
echo " Subjlist: ${Subjlist}"
echo " EnvironmentScript: ${EnvironmentScript}"
echo " GroupAverageName: ${GroupAverageName}"
echo " RegName: ${RegName}"
echo " MatlabMode: ${MatlabMode}"
echo "-- ${scriptName}: Specified Command-Line Options: -- End --"
}

# get command line options
get_options "$@"

# set up pipeline environment variables and software
source "${EnvironmentScript}"

if ((RunLocal)) || [[ "$QUEUE" == "" ]]; then
echo "running locally"
queuing_command=("$HCPPIPEDIR"/global/scripts/captureoutput.sh)
else
echo "queueing with fsl_sub to $QUEUE"
queuing_command=("$FSLDIR/bin/fsl_sub" -q "$QUEUE")
fi

# Download the PROFUMO Singularity image from the following link and place it in the PFM folder,
# or change the path below to point to your own copy of the image
# https://balsa.wustl.edu/myelin/download?dirName=public&filepath=profumo_v2.sif
ProfumoSingularity="$HCPPIPEDIR/PFM/profumo_v2.sif"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually stuff like this is handled in the HCP Pipelines setup script.

@coalsont coalsont Aug 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I commented earlier, unless we are expecting every user to be required to run PFMs, then downloading and specifying the location of the profumo container seems specific to only this pipeline, unlike FSL or wb_command, which are used by nearly all pipelines. Even for freesurfer, bbregister is used in multiple pipelines, which is why it makes sense to have in the global setup.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I m thinking if we can make it overridable rather than required in the global setup:

ProfumoSingularity="${ProfumoSingularity:-$HCPPIPEDIR/PFM/profumo_v2.sif}"

That way users who don't run PFMs never have to set anything, but anyone running this regularly can export it from their setup script if they prefer.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't expect all end users to feel comfortable editing a line that has ${something:- in it, but I do expect some of them to download it to somewhere other than the pipelines folder. Someone who is comfortable with bash syntax can easily set things up some other way anyway.

This particular container appears to be single-purpose for a single pipeline. As long as that is the case, I don't think it belongs in a centralized configuration location (where people who aren't running PFMs would have to read it and understand that the setting isn't relevant to them).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With regard to putting it in the setup script, if we make it optional in there, isn't that fine? I think it might be best to keep all external dependencies referenced in there instead of scattered across the pipelines. I don't think we currently have any external dependencies that are not in the setup script.

@coalsont coalsont Aug 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we expect nearly all users to run PFMs, it might make sense. For any user that isn't going to run PFMs, having it in the global setup would just be clutter that they have to read, understand, and then skip over. We can move it to a global location later if it becomes something that an overwhelming majority of users will need to do. Keeping the global setup short by only containing things that nearly all users need to do is a benefit to the user experience and gets them started faster before asking them to make judgement calls.

"External" isn't the reason I put things in a global location, it is whether they are used in more than one pipeline (or aren't actually associated with any specific pipeline, like VolumeToCIFTI.sh). MSM, while also being a single file, is used in both PostFS and MSMAll, hence it would not be a good fit for being in a single pipeline's batch, as then users would need to configure it twice. That does not currently apply to profumo, and I suspect it may never apply. The hippunfold container likewise will probably only be used in one pipeline, and it isn't clear what percentage of users will want to use that, either, so it would also be a candidate for batch script configuration, regardless of being external.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the long run, I think HippUnfold will be run by most/all users (as the future CIFTI grayordinates space will use hippocampal surfaces rather than hippocampal volumes). Also, in the long run, I think PFMs will be the primary functional network analysis tool. I am leaning strongly towards putting them in the setup script for that reason.


# general settings
# set the start step beginning from RunPROFUMO which is by default the first step
StartStep="RunPROFUMO"
StopStep="GroupPFMs"
NumWishart="6"
KeepWishartFiles="NO"

# set how many subjects to do in parallel (local, not cluster-distributed) during RSN regression, defaults to all detected physical cores, '-1'
parLimit=-1

# general inputs
fMRINames="rfMRI_REST1_LR@rfMRI_REST1_RL@rfMRI_REST2_LR@rfMRI_REST2_RL"

randSeed=123 # random seed for PROFUMO

OutputfMRIName="rfMRI_REST"
# set the MR concat fMRI name, if multi-run FIX was used, leave empty for single runs
ConcatName=""

# set the output spectra size for individual projection, RunsXNumTimePoints #subjectExpectedTimepoints="3655"
subjectExpectedTimepoints="4800"

# set temporal highpass full-width (2*sigma) used in preprocessing
HighPass="0"
Comment thread
glasserm marked this conversation as resolved.
Outdated

#set fMRIResolution of data, like '2','1.60' or '2.40'
fMRIResolution="2.0"

# PFM settings for REST data
# set the PFM dimensionality
PFMdim="76"

PFMFolder=${StudyFolder}/$GroupAverageName/MNINonLinear/Results/${OutputfMRIName}_PFM_d${PFMdim}
# Reference image for PROFUMO
RefImage="${StudyFolder}/$GroupAverageName/MNINonLinear/Results/${OutputfMRIName}/${OutputfMRIName}_Atlas_MSMAll_hp${HighPass}_clean_tclean_meanvn.dscalar.nii"
Comment thread
glasserm marked this conversation as resolved.
Outdated

# set the file name component representing the preprocessing already done
fMRIProcSTRING="hp${HighPass}_clean_rclean_tclean"

# set the mesh resolution, like '32' for 32k_fs_LR
LowResMesh="32"

# Define OutputSTRING with seed designation
OutputSTRING="${OutputfMRIName}_d${PFMdim}_${GroupAverageName}_seed${randSeed}_PFMs"

# RSN regression settings
FixLegacyBiasString="NO"
ScaleFactor="0.01"

# Volume template file
VolumeTemplateCIFTI="${HOME}/data/HCPpipelines_ExampleData/${GroupAverageName}/MNINonLinear/${GroupAverageName}_CIFTIVolumeTemplate_${OutputfMRIName}.${fMRIResolution}.dscalar.nii"

## PROFUMO settings
ProfumoConfig="${PFMFolder}/dataLocations.json"
TR="0.72"
ProfumoThreads="-1" # number of threads for PROFUMO, -1 means auto-detect physical cores
DOFCorrection="0.5"
CovModel="Subject"
nStarts="5" # number of multi-start iterations for PROFUMO
RandomSeed="$randSeed" # random seed for PROFUMO reproducibility
# RefImage will be auto-set based on data type below

# build Profumo data location json
mkdir -p $PFMFolder
echo '{' > $ProfumoConfig
for Subject in $(echo $Subjlist | tr "@" "\n"); do
echo -e "\t\"$Subject\": {" >> $ProfumoConfig
for fMRIName in $(echo $fMRINames | tr "@" "\n"); do
runFile="${StudyFolder}/${Subject}/MNINonLinear/Results/${fMRIName}/${fMRIName}_Atlas${RegString}_${fMRIProcSTRING}.dtseries.nii"
if [[ -e $runFile ]]; then
echo -e "\t\t\"$fMRIName\": \"$runFile\"," >> $ProfumoConfig
fi
done
perl -pi -e 'if (eof) { s/,$// }' $ProfumoConfig # remove trailing comma
echo -e "\t}," >> $ProfumoConfig
done
perl -pi -e 'if (eof) { s/,$// }' $ProfumoConfig # remove trailing comma
echo "}" >> $ProfumoConfig

# PFM pipeline execution
echo "Starting PFM postprocessing pipeline"
echo "Data type: ${OutputfMRIName}"
echo "PFM dimension: ${PFMdim}"

"${queuing_command[@]}" "$HCPPIPEDIR"/PFM/PFMPipeline.sh \
--study-folder="$StudyFolder" \
--subject-list="$Subjlist" \
--fmri-names="$fMRINames" \
--output-fmri-name="$OutputfMRIName" \
--output-string="$OutputSTRING" \
--proc-string="$fMRIProcSTRING" \
--group-average-name="$GroupAverageName" \
--pfm-dimension="$PFMdim" \
--pfm-folder="$PFMFolder" \
--surf-reg-name="$RegName" \
--concat-name="$ConcatName" \
--low-res-mesh="$LowResMesh" \
--runs-timepoints="$subjectExpectedTimepoints" \
--fix-legacy-bias="$FixLegacyBiasString" \
--num-wishart="$NumWishart"\
--scale-factor="$ScaleFactor" \
--starting-step="$StartStep" \
--stop-after-step="$StopStep" \
--parallel-limit="$parLimit" \
--matlab-run-mode="$MatlabMode" \
--profumo-config="$ProfumoConfig" \
--profumo-singularity="$ProfumoSingularity" \
--profumo-tr="$TR" \
--keep-wishart-files="$KeepWishartFiles" \
--profumo-threads="$ProfumoThreads" \
--profumo-dof-correction="$DOFCorrection" \
--profumo-cov-model="$CovModel" \
--profumo-multi-start-iterations="$nStarts" \
--profumo-random-seed="$RandomSeed" \
--ref-image="$RefImage" \
--volume-template-file="$VolumeTemplateCIFTI"

echo "PFM pipeline submitted successfully!"

1 change: 1 addition & 0 deletions PFM/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.sif
Loading