Skip to content

AF2 output publishDir separate between base program files and plaintext metrics#453

Draft
keiran-rowell-unsw wants to merge 3 commits into
nf-core:devfrom
Australian-Structural-Biology-Computing:AF2_publishdir
Draft

AF2 output publishDir separate between base program files and plaintext metrics#453
keiran-rowell-unsw wants to merge 3 commits into
nf-core:devfrom
Australian-Structural-Biology-Computing:AF2_publishdir

Conversation

@keiran-rowell-unsw
Copy link
Copy Markdown
Contributor

@keiran-rowell-unsw keiran-rowell-unsw commented Jan 29, 2026

Just to demonstrate proposed publishDir layout

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jan 29, 2026

This PR is against the master branch ❌

  • Do not close this PR
  • Click Edit and change the base to dev
  • This CI test will remain failed until you push a new commit

Hi @keiran-rowell-unsw,

It looks like this pull-request is has been made against the Australian-Structural-Biology-Computing/proteinfold master branch.
The master branch on nf-core repositories should always contain code from the latest release.
Because of this, PRs to master are only allowed if they come from the Australian-Structural-Biology-Computing/proteinfold dev branch.

You do not need to close this PR, you can change the target branch to dev by clicking the "Edit" button at the top of this page.
Note that even after this, the test will continue to show as failing until you push a new commit.

Thanks again for your contribution!

@keiran-rowell-unsw keiran-rowell-unsw deleted the AF2_publishdir branch January 29, 2026 05:19
@keiran-rowell-unsw keiran-rowell-unsw restored the AF2_publishdir branch January 29, 2026 05:19
@keiran-rowell-unsw keiran-rowell-unsw changed the base branch from master to dev January 29, 2026 05:20
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jan 29, 2026 •
edited
Loading

nf-core pipelines lint overall result: Failed ❌

Posted for pipeline commit ff2cc2a

+| ✅ 327 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗  33 tests had warnings |!
-| ❌   1 tests failed       |-
Details

❌ Test failures:

  • schema_params - Param save_base_output from nextflow config not found in nextflow_schema.json

❗ Test warnings:

  • files_exist - File not found: conf/igenomes.config
  • files_exist - File not found: conf/igenomes_ignored.config
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • schema_description - No description provided in schema for parameter: rosettafold2na_uniref30_link
  • schema_description - No description provided in schema for parameter: rosettafold2na_bfd_link
  • schema_description - No description provided in schema for parameter: rosettafold2na_pdb100_link
  • schema_description - No description provided in schema for parameter: rosettafold2na_weights_link
  • schema_description - No description provided in schema for parameter: rfam_full_region_link
  • schema_description - No description provided in schema for parameter: rfam_cm_link
  • schema_description - No description provided in schema for parameter: rnacentral_rfam_annotations_link
  • schema_description - No description provided in schema for parameter: rnacentral_id_mapping_link
  • schema_description - No description provided in schema for parameter: rnacentral_sequences_link
  • schema_description - No description provided in schema for parameter: rosettafold2na_uniref30_path
  • schema_description - No description provided in schema for parameter: rosettafold2na_bfd_path
  • schema_description - No description provided in schema for parameter: rosettafold2na_pdb100_path
  • schema_description - No description provided in schema for parameter: rosettafold2na_weights_path
  • local_component_structure - post_processing.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_rosettafold_all_atom_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_rosettafold2na_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_alphafold3_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_colabfold_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - aria2_uncompress.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_esmfold_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_helixfold3_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_boltz_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_alphafold2_dbs.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure

❔ Tests ignored:

✅ Tests passed:

Run details

  • nf-core/tools version 3.5.1
  • Run at 2026-01-29 05:22:48

@tlitfin
Copy link
Copy Markdown
Contributor

tlitfin commented Jan 29, 2026

Somehow I didn't see this WIP when I started working on the same thing. I have a completed/linted/tested version using the same ideas but I won't submit a PR if you want to keep working to finish this one.

@keiran-rowell
Copy link
Copy Markdown

keiran-rowell commented Jan 29, 2026

Somehow I didn't see this WIP when I started working on the same thing. I have a completed/linted/tested version using the same ideas but I won't submit a PR if you want to keep working to finish this one.

If you have a testing and linting version feel free to throw a PR up.

I just wanted a quick with-code discussion of the sensible way to split output before people divvied up #393 & #394

@tlitfin tlitfin mentioned this pull request Jan 29, 2026
Merged
3 tasks
@keiran-rowell-unsw
Copy link
Copy Markdown
Contributor Author

keiran-rowell-unsw commented Feb 5, 2026 •
edited
Loading

Somehow I didn't see this WIP when I started working on the same thing. I have a completed/linted/tested version using the same ideas but I won't submit a PR if you want to keep working to finish this one.

For the next minor release, I can refine the publishDir patterns to use program-specific globs (*.json vs *.pkl) instead of broad ones (*.*, **), while still retaining the discussed simplicity of a single save_intermediates flag. My concerns in the meeting are unforeseen overmatching, particularly if other current or future underlying programs produce tmp files or symlinks.

This more refined control implementation would let users and institutes better manage storage for bulk runs, and provide fewer files for MultiQC to process.

As we add nf-tests, the test asserts will self-document which files are 'official' and well formatted, and what the user can focus on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@keiran-rowell-unsw @tlitfin @keiran-rowell