What happened?

Description
Since ea9f02b and #10571, the following snippet hangs indefinitely — the files are opened successfully, but xr.open_mfdataset seems to hang and never completes. I end up having to kill the process. Reverting to the commit just before this change works fine.

open_mfdataset_params = {
    "engine": engine,
    "parallel": True,
    "preprocess": partial_preprocess,
    "data_vars": "all",
    "concat_characters": True,
    "mask_and_scale": True,
    "decode_cf": True,
    "decode_times": self.time_coder,
    "decode_coords": True,
    "compat": "override",
    "coords": "minimal",
    "drop_variables": drop_vars_list,
}

ds = xr.open_mfdataset(batch_files, **open_mfdataset_params)

Context

• Engine: h5netcdf
• Scheduler: Dask running on a Coiled cluster
• Input: batch_files is a fileset array of NetCDF objects stored in S3 (fileset = [s3_fs.open(file) for file in s3_paths]
• preprocess runs successfully on all files, but open_mfdataset itself gets stuck

What I’ve checked

• I’m aware of Regression: "h5py objects cannot be pickled" with cloudpickle #10712, and like @rabernat, that change broke my code. I actually caught it in my own integration tests, whereas the xarray unit tests didn’t expose it at the time. More generally, I think some of these regressions could be avoided if there were higher-level integration tests in addition to the existing unit tests. That particular issue was fixed about two weeks ago and resolved the pickle error I was seeing. However, I’m still encountering the hang I describe here.

• Rolling back to the commit before ea9f02b avoids the problem. In my case, 938e186 is the latest working commit

What did you expect to happen?

xr.open_mfdataset should complete successfully, as it does on earlier commits.

Minimal Complete Verifiable Example

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "xarray[complete]@git+https://github.com/pydata/xarray.git@main",
# ]
# ///
#
# This script automatically imports the development branch of xarray to check for issues.
# Please delete this header if you have _not_ tested this script with `uv run`!

import xarray as xr
xr.show_versions()
# your reproducer code ...

Steps to reproduce

No response

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Anything else we need to know?

No response

Environment