Show standard error for failing CWL jobs that capture standard error by adamnovak · Pull Request #5555 · DataBiosphere/toil

adamnovak · 2026-07-16T21:17:57Z

This fixes #2994 by logging standard error from CWL tasks to the Toil log, when the tasks fail, even if standard error for the task is being captured to a file.

(When standard error is not captured, we already log it to the Toil log.)

The result looks just like if it wasn't being captured:

[2026-07-16T17:15:05-0400] [Thread-4 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] fail_with_log.cwl.hello._:640ebd69-f2a0-4ed6-a923-4b450915dd90.stderr follows:
=========>
	Message: This is a test
<=========

We don't do this when the job succeeds, because the user might be capturing the log because it's inconveniently large.

Changelog Entry

To be copied to the draft changelog by merger:

CWL job error logs are now reported for failing jobs even if the job's standard error is captured to a file.

Reviewer Checklist

Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
- If it is coming from an external repo, make sure to pull it in for CI with:
```
contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
```
- If there is no associated issue, create one.
Read through the code changes. Make sure that it doesn't have:
- Addition of trailing whitespace.
- New variable or member names in camelCase that want to be in snake_case.
- New functions without type hints.
- New functions or classes without informative docstrings.
- Changes to semantics not reflected in the relevant docstrings.
- New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
- New features without tests.
Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
Finish the review with an overall description of your opinion.

Merger Checklist

Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.
Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.
Merge with the Github "Squash and merge" feature.
- If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
Copy its recommended changelog entry to the Draft Changelog.
Append the issue number in parentheses to the changelog entry.

… log files for failing commands

…standard-error

…ast MyPy

mr-c

Epic PR, nice re-org!

mr-c · 2026-07-17T13:52:27Z

+# This section deals with the CWL promise system.
+#
+# This is not to be confused with the Toil promise system, except under
+# fconfusing circumstances.


Suggested change

# fconfusing circumstances.

# confusing circumstances.

mr-c · 2026-07-17T13:54:45Z

+# Processes in general can generate a lot of types of jobs, but CommandLineTools and ExpressionTools are typed as generating a more restricted set of possible types of jobs.
+# We want to be able to wrap them with the same wrapper.
+#
+# ExperssionTool's job() is also able to take a None for output_callbacks, while CommandLineTool's can't.


Suggested change

# ExperssionTool's job() is also able to take a None for output_callbacks, while CommandLineTool's can't.

# ExpressionTool's job() is also able to take a None for output_callbacks, while CommandLineTool's can't.

mr-c · 2026-07-17T13:55:02Z

+#
+# We can't just say the wrapper mixin generates the more permissive type range because then it won't be substitutable for a CommandLineTool, etc.
+#
+# So to type this we have to do a bunch of ugly type stuff, and event hen we need to cast (see https://github.com/python/mypy/issues/17192#issuecomment-4995017670)


Suggested change

# So to type this we have to do a bunch of ugly type stuff, and event hen we need to cast (see https://github.com/python/mypy/issues/17192#issuecomment-4995017670)

# So to type this we have to do a bunch of ugly type stuff, and even then we need to cast (see https://github.com/python/mypy/issues/17192#issuecomment-4995017670)

mr-c · 2026-07-17T13:55:28Z

-        stage_listing: bool = False,
-        streaming_allowed: bool = True,
-    ):
+    Ypu also need to tell it the types that the mixed-with Process uses on its


Suggested change

Ypu also need to tell it the types that the mixed-with Process uses on its

You also need to tell it the types that the mixed-with Process uses on its

mr-c · 2026-07-17T13:55:54Z

+        Attach the Toil tool to the Toil file store it is running against.
+
+        This is needed in order to allow hooks we add into the cwltool
+        tool-running code to talk to the file stroe for things like logging.


Suggested change

tool-running code to talk to the file stroe for things like logging.

tool-running code to talk to the file store for things like logging.

mr-c · 2026-07-17T14:00:41Z


+#####
+#
+# In this section, we have helper functions for loading and setting uop a


Suggested change

# In this section, we have helper functions for loading and setting uop a

# In this section, we have helper functions for loading and setting up a

adamnovak added 6 commits July 9, 2026 17:04

Add a CWL file we could use as a test for getting back standard error…

7b67f86

… log files for failing commands

Merge remote-tracking branch 'upstream/master' into issues/2994-show-…

152368b

…standard-error

Reorder and document cwltoil.py

4784261

Hook the job method of all ToilTool Processes in a way we can sneak p…

9fe1705

…ast MyPy

Add test to make sure log is returned.

26888a1

Add code in case something goes really wrong and the log is not there

7b06592

mr-c requested changes Jul 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Show standard error for failing CWL jobs that capture standard error#5555

Show standard error for failing CWL jobs that capture standard error#5555
adamnovak wants to merge 6 commits into
masterfrom
issues/2994-show-standard-error

adamnovak commented Jul 16, 2026

Uh oh!

mr-c left a comment

Uh oh!

mr-c Jul 17, 2026

Uh oh!

mr-c Jul 17, 2026

Uh oh!

mr-c Jul 17, 2026

Uh oh!

mr-c Jul 17, 2026

Uh oh!

mr-c Jul 17, 2026

Uh oh!

mr-c Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	# ExperssionTool's job() is also able to take a None for output_callbacks, while CommandLineTool's can't.
	# ExpressionTool's job() is also able to take a None for output_callbacks, while CommandLineTool's can't.

	# So to type this we have to do a bunch of ugly type stuff, and event hen we need to cast (see https://github.com/python/mypy/issues/17192#issuecomment-4995017670)
	# So to type this we have to do a bunch of ugly type stuff, and even then we need to cast (see https://github.com/python/mypy/issues/17192#issuecomment-4995017670)

	Ypu also need to tell it the types that the mixed-with Process uses on its
	You also need to tell it the types that the mixed-with Process uses on its

	tool-running code to talk to the file stroe for things like logging.
	tool-running code to talk to the file store for things like logging.

	# In this section, we have helper functions for loading and setting uop a
	# In this section, we have helper functions for loading and setting up a

Uh oh!

Conversation

adamnovak commented Jul 16, 2026

Changelog Entry

Reviewer Checklist

Merger Checklist

Uh oh!

mr-c left a comment

Choose a reason for hiding this comment

Uh oh!

mr-c Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

mr-c Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

mr-c Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

mr-c Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

mr-c Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

mr-c Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants