Extend HyperOpt backend to support the itwinai interface

[Radonirinaunimi]

This extends the hyperopt backend in view of interfacing with itwinai. Such an interface will provide several advantages as this will allow us to:

• run distributed HPO on a Ray cluster without mongoDB
• use some of the Ray Tune search algorithms
• use advanced scheduler such as SHA (Successive Halving Algorithm) for example to stop unpromising trials early
• have a genuine memory isolation that'd have fully prevented Fix-ish hyperopt leak #2511
• a true automatic resume of crashed/interrupted trials
• have live-monitoring through which one can see trials-status and resource usages
• ...

These minor changes are sufficient for the plugin (to be hosted in separate repo) while still allowing the old hyperopt to run exactly as before.

[Radonirinaunimi]

For some reasons, there are still a couple of data that fail...

[scarlehoff]

It's onle dijets now, pobably need to crank the precision up a bit. Although they are so bad that I'm not sure there's enough digits in the known universe 😬

I'll open a branch and just keep testing in a loop until I'm sated.

[scarlehoff]

Ah, no, scratch that. Good news, I missed two covmat_to_artfuncs!

[scarlehoff]

Rebase on master again. I've rerun the test now 3 times and hasn't failed yet so hopefully it's fixed now.

ps: there's one of the star - 2jet that I'm not sure about and if it fails again I will just "disconnect" that filter.py because I don't think it is worth more time.

[Radonirinaunimi]

Perfect! Thanks.