Add SaveBackground task and ingest_background_cf suite for GEOS-CF JDI backgrounds

[ftgoktas]

Adds a SaveBackground task and stores GEOS-CF JDI background files into R2D2 as symlinks. The task loops over 24 hourly steps (PT0H–PT23H), resolves each file's path via strftime on a configurable background_source_path template, checks the file exists, and calls r2d2.store(..., store_as_symlink=...). The model is read from self.__model__. Also names the suite folder suites/r2d2_ingest/ since the suite now covers both observations and backgrounds.

How to test:

1. swell create ingest_jdi_cf
2. Set dry_run in experiment.yaml false to actually store files, or keep true to preview which files you will be storing.
3. swell launch ...

Note on R2D2 store_as_symlink: After creating the symlink, file_util._set_permissions calls os.chmod which follows the symlink to the source file and throws PermissionError since we don't own it. The symlink and DB entry are both created before the chmod. We catch and verify the symlink before continuing. I think this is a bug in the R2D2 client (_set_permissions should be skipped when store_as_symlink=True) and should be fixed there. Created an issue about this here: https://github.com/JCSDA-internal/r2d2-client/issues/112

Closes #788

[mranst]

One small comment, but looks good. I was able to test the dry run

[jeromebarre]

quick comment:
since this ingest suite is not observations but model backgrounds this not really fits under suites/ingest_obs.
A solution to this is to remove the _obs and call all these simply ingest or more specifically r2d2_ingest because that is what it is about.

[ftgoktas]

quick comment: since this ingest suite is not observations but model backgrounds this not really fits under suites/ingest_obs. A solution to this is to remove the _obs and call all these simply ingest or more specifically r2d2_ingest because that is what it is about.

I agree, just pushed a commit renaming the relevant file/folder names.

[mranst]

Looks good, thanks!

[Dooruk]

Some comments:

• Can you create an r2d2 issue on JCSDA repo on the symlink problem you described above?

• It's jedi not jdi 👀

• In terms of task names, store jedi (or jdi 😄 ) is very ambiguous, an alternative might be ingest_background since we have the ingest_obs already. I also see this task is written more towards CF in mind (totally fine), so two options:

1. Name the task IngestCFBackground
2. Use method @mranst introduced and store model specific tasks under geos_cf folder, e.g., /tasks/geos_cf/ingest_background.py`

• Another suggestion regarding store_symlink. This should be optional, in other words it should be a key in experiment.yaml. Not sure what the default bool should be. This is another issue & PR though.

• Another note for the pipeline key. I've been meaning to create an issue regarding this, which we can discuss this over a dedicated R2D2 meeting as well. Currently ingest have two options. Either ingest to R2D2 directly or download, convert and ingest. For marine and atmosphere applications another option might be convert and ingest if there are netcdf files already on Discover. This maybe the case for operational applications where files are downloaded automatically via D-BOSS (GESDISC). So my suggestion is to have that pipeline key as a list. Let me think about it or let's discuss it so I can describe it better under an issue.

Sorry, long list 😟

[ftgoktas]

Some comments:

• Can you create an r2d2 issue on JCSDA repo on the symlink problem you described above?

Done! See https://github.com/JCSDA-internal/r2d2-client/issues/112

• It's jedi not jdi 👀

That got me too 😄, @mer-a-o confirmed the geos-cf collection is called jdi.

• In terms of task names, store jedi (or jdi 😄 ) is very ambiguous, an alternative might be ingest_background since we have the ingest_obs already. I also see this task is written more towards CF in mind (totally fine), so two options:

1. Name the task IngestCFBackground
2. Use method @mranst introduced and store model specific tasks under geos_cf folder, e.g., /tasks/geos_cf/ingest_background.py`

Switched to SaveBackground, which is model agnostic and cleaner. Happy to rename if we want to go in a different direction.

• Another suggestion regarding store_symlink. This should be optional, in other words it should be a key in experiment.yaml. Not sure what the default bool should be. This is another issue & PR though.

Done, added as a TaskQuestion (default True) and written to experiment.yaml. We have a workaround to mitigate the symlink PermissionError so it never triggers.

• Another note for the pipeline key. I've been meaning to create an issue regarding this, which we can discuss this over a dedicated R2D2 meeting as well. Currently ingest have two options. Either ingest to R2D2 directly or download, convert and ingest. For marine and atmosphere applications another option might be convert and ingest if there are netcdf files already on Discover. This maybe the case for operational applications where files are downloaded automatically via D-BOSS (GESDISC). So my suggestion is to have that pipeline key as a list. Let me think about it or let's discuss it so I can describe it better under an issue.

Agreed, we can discuss and open up a follow up PR.

Sorry, long list 😟

🫡

[Dooruk]

Thanks! One quick comment as I'm working on my marine ingest PR. Can you make this key get used for ingest_obs as well?

[ftgoktas]

Should we rename SaveBackground -> IngestBackground to match the IngestObs naming convention? They're both r2d2.store operations essentially.

[mer-a-o]

I suggest using "ingest" for suite and then "save" for tasks, saveObservation, saveBackground or saveRestart etc. I think of "ingest" as a suite that includes several tasks such as (download), (convert), save, etc.

[Dooruk]

Hmm, I consider Save.. tasks more for online cycling storing and Ingest for offline storing (i.e., existing experiment or observations). So it would look like:

• Ingest for suite names (ingest_background, ingest_obs) -> uses Store tasks (storebackground, storeobservations)

• Cycling suites -> uses Save tasks

Does that make sense?

this means ingest_obs task would be renamed store_obs.. maybe another PR.

Also FYI Furkan, the documentation sidebar was messed up and I addressed that in my merged PR.