Remove uninstallable revisions from lock files by arash77 · Pull Request #929 · usegalaxy-eu/usegalaxy-eu-tools

arash77 · 2025-10-22T18:28:03Z

This could be an alternative to #838

bgruening · 2025-10-22T21:43:03Z

Thanks @arash77 can you maybe open a PR with the result of such a run.

bernt-matthias

Wonderful to have two approaches. Now we can compare. If there is disagreement we need to investigate and otherwise we can be more confident.

Maybe the output (of both approaches) should contain a reasoning why a revision is removed ...

bernt-matthias · 2025-10-23T07:17:07Z

+            skipped += 1
+            continue
+
+        all_revs = list(uninstallable) + list(installable)


all_revs = list(revisions)?

That wouldn't include what we miss from the installable list.

Isn't uninstallable = set(revisions) - set(installable) implying revisions = uninstallable + installable.

If so, set(all_revs) = uninstallable?

What if installable has something that wasn’t in the lock file’s revisions?
Then not exactly! set(all_revs) becomes the union of uninstallable and installable, which can include extra entries not present in revisions.

For example:

revisions = ['r1', 'r2', 'r3', 'r4'] installable = ['r5', 'r2'] uninstallable = set(revisions) - set(installable) all_revs = list(uninstallable) + list(installable) print(set(all_revs)) # {'r1', 'r2', 'r3', 'r4', 'r5'} → includes 'r5' not in revisions

bernt-matthias · 2025-10-23T07:27:16Z

+                logger.warning(
+                    f"{name},{owner}: no matching installable revision for {cur}"
+                )
+                continue


exit 1 here as well?

@bernt-matthias, I'm not sure if we should do this here.
Now we have an error here, and it didn't process the rest of them.
https://github.com/arash77/usegalaxy-eu-tools/actions/runs/19669042380/job/56333058944#step:6:16

bernt-matthias · 2025-10-23T07:57:13Z

+                (
+                    cand
+                    for cand in installable
+                    if version_cache.get(cand) == cur_versions


Hrm. What if the set of tools in the repo changed? (Not sure if this is covered in my solution :))

What do you mean by changed?

If a tool was added to the tool repo, i.e. for a repo containing more than one tool.

Then should we run this regularly?

What I meant was that tools the set of tools in a tool repository might change between revisions.

Hmm, I don't know how to deal with those! Do you have any suggestions?

arash77 · 2025-10-24T10:24:33Z

Thanks @arash77 can you maybe open a PR with the result of such a run.

This is the test PR arash77#2

arash77 · 2025-10-24T10:36:26Z

I found out that some tools might not have any revisions to install after this run. Maybe we could add the next revision to replace them, or, if we haven’t already, handle the revisions in a separate workflow.

bernt-matthias · 2025-10-27T10:12:16Z

I found out that some tools might not have any revisions to install after this run.

How come? Do you have an example?

arash77 · 2025-10-27T10:35:05Z

How come? Do you have an example?

https://github.com/arash77/usegalaxy-eu-tools/pull/2/files#diff-e6a5a1ad82a6dcd5528e501ed88c67f70152e1587e4d73a12364afdc3b3f2825R98

bernt-matthias · 2025-10-27T10:50:31Z

The revision list of all these tools seems non-empty, i.e. there are revisions to install?

arash77 · 2025-10-27T11:04:33Z

The revision list of all these tools seems non-empty, i.e. there are revisions to install?

Yes, there are revisions to install, but not present in lock files.
The ones that have been removed are really returning errors when installed.
For example:

(134/144) Installing repository data_manager_sortmerna_database_downloader from rnateam to section "Data Managers" at revision 30bb49887172 (TRT: 0:00:28.772095)
	Repository data_manager_sortmerna_database_downloader is already installed.
	repository data_manager_sortmerna_database_downloader installed successfully (in 0:00:00.891637) at revision 30bb49887172

arash77 · 2025-10-28T15:15:00Z

With the new changes, it replaces the removed ones with a newer revision, even if it does not exist in the lock files.
It is also possible to replace it with scripts/update-tool.py.

bernt-matthias · 2025-10-28T16:31:18Z

I'm running my version here https://github.com/bernt-matthias/usegalaxy-eu-tools/actions/runs/18881254213/job/53884716367

since @martenson was also working on this (CESNET/galaxy_tools#21) I'm also pinging him here.

arash77 · 2025-10-29T15:21:15Z

There are some differences between your run and my run.
I could create a report for some of them:

📄 `bgruening.yaml.lock`

Tool: `flye` (owner: bgruening)

Removed revisions in Arash run:

➖ de24438c9988

Tool: `jbrowse2` (owner: fubar)

Removed revisions in Arash run:

➖ 137e08517410
➖ cec274db51c0

Added revisions in Arash run:

➕ 49f3d3878413

Tool: `flexynesis_cbioportal_import` (owner: bgruening)

Removed revisions in Arash run:

➖ 0e9e3e083350
➖ 2b414a6c35af

Added revisions in Arash run:

➕ e9a7cb5a3c63

Tool: `nanopolish_eventalign` (owner: bgruening)

Removed revisions in Arash run:

➖ 8fdb079ddaf0

📄 `cheminformatics.yaml.lock`

Tool: `gmx_energy` (owner: chemteam)

Duplicated in Matthias run:

🔁 dae135dfd519

📄 `eirene.yaml.lock`

Tool: `mfassignr_mfassign` (owner: recetox)

Duplicated in Matthias run:

🔁 4422c60a4b71

Tool: `mfassignr_mfassigncho` (owner: recetox)

Duplicated in Matthias run:

🔁 5ea9dd423109

📄 `epigenetics.yaml.lock`

Tool: `hifive` (owner: sauria)

Added revisions in Matthias run:

➕ cc95aa05f643

Added revisions in Arash run:

➕ 0a9c76d80e33

📄 `rnateam.yaml.lock`

Tool: `graphprot_predict_profile` (owner: rnateam)

Duplicated in Matthias run:

🔁 ddcf35a868b8

📄 `single-cell-ebi-gxa.yaml.lock`

Tool: `scmap_scmap_cell` (owner: ebi-gxa)

Removed revisions in Arash run:

➖ 65b719530dd5

bernt-matthias · 2025-10-29T17:03:29Z

I'm just running again. I needed to add --add in my script. Lets see :)

bernt-matthias · 2025-10-29T21:18:34Z

@arash77 we are getting closer. I looked at a few examples and in all cases I would prefer the results of my branch.

arash77 · 2025-11-07T13:10:16Z

The only problems I have with my script are these that I can’t figure out what’s going on!

`bgruening/bgruening/flye`

Removed revisions:
de24438c9988 5 -> 8d4f03b5fe9d 7 (why not 0284be52bfcf 6)

`bgruening/fubar/jbrowse2`

Removed revisions:

dc3fe98e7b37 103 -> a074cd6b5905 107 (why not 9e3f69d9fed1 104?)

`bgruening/bgruening/flexynesis_cbioportal_import`

Removed revisions:

0e9e3e083350 1 -> e9a7cb5a3c63 5 (why not ee7e61ab554d 2?)
2b414a6c35af 3 -> e9a7cb5a3c63 5 (why not 9c57d6edd3c1 4?)

`bgruening/bgruening/nanopolish_eventalign`

Removed revisions:

8fdb079ddaf0 5 -> 85a394edc247 7 (why not 855b7ba0ce9c 6?)
bee42f615f28 0 -> 0938217b3221 3 (why not c8c5caecfacc 1?)

`single-cell-ebi-gxa/ebi-gxa/scmap_scmap_cell`

Removed revisions:

65b719530dd5 12 -> 32ae4a600167 14 (why not c8c5caecfacc 13?)

bernt-matthias · 2025-11-07T14:21:14Z

bgruening/bgruening/flye
de24438c9988 5 -> 8d4f03b5fe9d 7 (why not 0284be52bfcf 6)

in the update from 6 to 7 the version was not bumped, i.e. 6 became uninstallable

bgruening/bgruening/flexynesis_cbioportal_import
0e9e3e083350 1 -> e9a7cb5a3c63 5 (why not ee7e61ab554d 2?)
2b414a6c35af 3 -> e9a7cb5a3c63 5 (why not 9c57d6edd3c1 4?)

1 = 0.2.20 + 0
2 = 0.2.20 + 0
3 = 0.2.20 + 0
4 = 0.2.20 + 1
5 = 0.2.20 + 1

so 1, 2, and 4 are uninstallable

I do not understand why 3 is removed.

Will need more time to look into this, but maybe it helps you?

bernt-matthias · 2025-11-07T15:20:51Z

bgruening/bgruening/nanopolish_eventalign

0 0.1.0
1 0.1.0
2 0.1.0 + tool-data/all_fasta.loc.sample
3 0.1.0
4 0.11.1
5 0.13.2+galaxy0
6 0.13.2+galaxy1 - tool-data/all_fasta.loc.sample
7 0.13.2+galaxy1 + tool-data/all_fasta.loc.sample

So 0->3 is expected

I do not understand why 5 is removed.

bi-gxa/scmap_scmap_cell

12 1.6.3+galaxy1
13 1.6.4+galaxy1 Invalid
14 1.6.4+galaxy2

I do not understand why 12 is removed. It is also not shown in the dropdown on the top here

arash77 · 2025-11-14T11:04:58Z

1 = 0.2.20 + 0
2 = 0.2.20 + 0
3 = 0.2.20 + 0
4 = 0.2.20 + 1
5 = 0.2.20 + 1

In the API, we don’t see tool versions in all revisions, and the tool version shown there doesn’t seem to match what you found.

revision 1 -> 0.2.20+galaxy1
revision 2 -> doesn't have any tool version
revision 3 -> 0.2.20+galaxy1
revision 4 -> doesn't have any tool version
revision 5 -> 0.2.20+galaxy1

bernt-matthias · 2025-11-18T15:58:56Z

In the API, we don’t see tool versions in all revisions, and the tool version shown there doesn’t seem to match what you found.

I basically tried to get the versions from the diffs reported in the toolshed. Probably I made some mistakes.

In the drop down menu in the TS (which I just discovered) 1 and 3 are not shown at all. If I construct the TS URL manually e.g. 3 it shows This revision can be installed: False (for 1 and 3). Wondering why this is?

The revisions 2 and 4 which can be selected show that the tools are invalid and also This revision can be installed: False.
Edit: This might also explain why you don't get versions from the API?

In summary this would explain why we go from revision 0 to 5, but I do not get yet, why 1 and 3 are not installable.

arash77 · 2025-11-18T16:18:53Z

This might also explain why you don't get versions from the API?

I think yes this is why the data is not right and it seems that it should get fixed from TS (maybe in v2) but for now it seems that it is not possible to find it out with API calls!

bgruening · 2025-11-18T20:50:12Z

I don't think we can fix the old TS and the new TS - who knows. I'm ready to be pragmatic here and if its just a few revisions that don't match implement special handling, or that I remove/hide tools

arash77 · 2025-11-19T13:45:03Z

So which way should we go? With the API or by cloning the repos?

bernt-matthias · 2025-11-23T10:21:25Z

How about using the approach implemented in this PR and storing removed revisions in an extra key of the lock files, eg uninstallable_revisions. Then we can keep track. Could also be separate files.

bgruening · 2025-11-23T12:29:38Z

Seperate file would be nice

…able revisions

…perations

… and improve logging clarity

…and path management

…nd ignore if no files found

arash77 · 2025-11-25T11:54:12Z

Should it be like this:
https://github.com/arash77/usegalaxy-eu-tools/pull/2/files

…isions

arash77 · 2025-11-27T12:04:32Z

Should this run on a schedule, or is it enough to have only workflow_dispatch?

bgruening · 2025-11-27T17:58:42Z

once a month?

…ow and script

Co-authored-by: Copilot <[email protected]>

…ad of a global constant

bgruening

Ok, we need to move forward here :)

Its git so we can rollback. Thanks everyone!

bernt-matthias reviewed Oct 23, 2025

View reviewed changes

arash77 force-pushed the fix-outdated branch from 12e062d to 1baad47 Compare October 24, 2025 10:27

arash77 added 6 commits November 24, 2025 17:06

Add workflow to fix outdated tools and implement script for uninstall…

d550002

…able revisions

Handle errors by exiting the script in fix_outdated.py

0b0e5a6

Remove ephemeris subproject reference

9d953a3

Implement retry logic with exponential backoff for repository fetch o…

6263fb6

…perations

Remove comment step from PR update workflow in fix-outdated-tools.yml

f926b66

Refactor tool removal logic to ensure revisions are updated correctly…

732e484

… and improve logging clarity

arash77 added 3 commits November 24, 2025 17:06

reverse over installable to get the latest revisions

ac1664c

Fix: Add missing github-token to uv installation step

8c920f8

Fix: Refactor fix_uninstallable function for improved error handling …

c7696df

…and path management

arash77 force-pushed the fix-outdated branch from 4945fb0 to c7696df Compare November 24, 2025 16:06

Fix: Update upload artifact step to include uninstallable revisions a…

84418c8

…nd ignore if no files found

Fix: Only write uninstallable revisions file if there are removed rev…

7991eb4

…isions

arash77 requested a review from bernt-matthias November 27, 2025 11:57

arash77 added 2 commits November 28, 2025 12:08

Add scheduled trigger for the fix outdated tools workflow

d30bb8d

Rename uninstallable revisions to not-installable revisions in workfl…

7548fad

…ow and script

Copilot AI review requested due to automatic review settings December 3, 2025 11:44

Copilot started reviewing on behalf of arash77 December 3, 2025 11:44 View session

Copilot finished reviewing on behalf of arash77 December 3, 2025 11:46

This comment was marked as off-topic.

Sign in to view

arash77 and others added 2 commits January 7, 2026 14:09

Apply suggestions from code review

b976dbc

Co-authored-by: Copilot <[email protected]>

Refactor retry_with_backoff to use a local max_retries variable inste…

c499cd0

…ad of a global constant

bgruening approved these changes Feb 4, 2026

View reviewed changes

bgruening merged commit bc3ed0e into usegalaxy-eu:master Feb 4, 2026
1 check passed

arash77 deleted the fix-outdated branch February 4, 2026 13:38

Conversation

arash77 commented Oct 22, 2025

Uh oh!

bgruening commented Oct 22, 2025

Uh oh!

bernt-matthias left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arash77 Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arash77 commented Oct 24, 2025

Uh oh!

arash77 commented Oct 24, 2025

Uh oh!

bernt-matthias commented Oct 27, 2025

Uh oh!

arash77 commented Oct 27, 2025

Uh oh!

bernt-matthias commented Oct 27, 2025

Uh oh!

arash77 commented Oct 27, 2025

Uh oh!

arash77 commented Oct 28, 2025

Uh oh!

bernt-matthias commented Oct 28, 2025

Uh oh!

arash77 commented Oct 29, 2025

📄 bgruening.yaml.lock

Tool: flye (owner: bgruening)

Tool: jbrowse2 (owner: fubar)

Tool: flexynesis_cbioportal_import (owner: bgruening)

Tool: nanopolish_eventalign (owner: bgruening)

📄 cheminformatics.yaml.lock

Tool: gmx_energy (owner: chemteam)

📄 eirene.yaml.lock

Tool: mfassignr_mfassign (owner: recetox)

Tool: mfassignr_mfassigncho (owner: recetox)

📄 epigenetics.yaml.lock

Tool: hifive (owner: sauria)

📄 rnateam.yaml.lock

Tool: graphprot_predict_profile (owner: rnateam)

📄 single-cell-ebi-gxa.yaml.lock

Tool: scmap_scmap_cell (owner: ebi-gxa)

Uh oh!

bernt-matthias commented Oct 29, 2025

Uh oh!

bernt-matthias commented Oct 29, 2025

Uh oh!

arash77 commented Nov 7, 2025

bgruening/bgruening/flye

bgruening/fubar/jbrowse2

bgruening/bgruening/flexynesis_cbioportal_import

bgruening/bgruening/nanopolish_eventalign

single-cell-ebi-gxa/ebi-gxa/scmap_scmap_cell

arash77 Nov 25, 2025 •

edited

Loading

📄 `bgruening.yaml.lock`

Tool: `flye` (owner: bgruening)

Tool: `jbrowse2` (owner: fubar)

Tool: `flexynesis_cbioportal_import` (owner: bgruening)

Tool: `nanopolish_eventalign` (owner: bgruening)

📄 `cheminformatics.yaml.lock`

Tool: `gmx_energy` (owner: chemteam)

📄 `eirene.yaml.lock`

Tool: `mfassignr_mfassign` (owner: recetox)

Tool: `mfassignr_mfassigncho` (owner: recetox)

📄 `epigenetics.yaml.lock`

Tool: `hifive` (owner: sauria)

📄 `rnateam.yaml.lock`

Tool: `graphprot_predict_profile` (owner: rnateam)

📄 `single-cell-ebi-gxa.yaml.lock`

Tool: `scmap_scmap_cell` (owner: ebi-gxa)

`bgruening/bgruening/flye`

`bgruening/fubar/jbrowse2`

`bgruening/bgruening/flexynesis_cbioportal_import`

`bgruening/bgruening/nanopolish_eventalign`

`single-cell-ebi-gxa/ebi-gxa/scmap_scmap_cell`

bernt-matthias commented Nov 18, 2025 •

edited

Loading