Bluebird gymnasium test agents

[presidente20]

This PR covers work done in my attempt to develop a test agent to drive bluebird gymnasium and in doing so flush out any issues and check that the project is supportive of attempts to train agents. (Ported from BluebirdDT).

I've made it a draft PR as not all of the contents necessarily need to be merged (e.g. the agent notebooks).

The best agent notebook by far is ppo_curriculum_agent.ipynb which manages to learn some lateral avoidance for 2 aircraft. Getting to this point was hard enough so I didn't attempt vertical separation or >2 aircraft. I can continue and try to do so if that is deemed useful.

The PR consists of:

• New notebooks using a couple of different RL algorithms and different airspaces (ISector and XSectorPlus).
• A new environment for FlightSchool and registration of it.
• A small set of reward changes were ported to support the curriculum and avoid known failure modes observed during training. (custom_reward.py)
• fixed the sign/direction of the progress reward so moving toward exit is rewarded correctly (expeditious.py)
• added a guard for degenerate route segments to avoid divide-by-zero failures (lateral_next_fix_proximity.py)

[dlpbc]

Other than my comments above, it looks good to me. Thanks.

[dlpbc]

Does this mean the FlightSchoolEnv is the same as SectorXPlusEnv?
I can't see the key difference.

[presidente20]

FlightSchoolEnv inherits the X-plus airspace implementation, but swaps in a different scenario generator/default scenario configuration. Flight School specifically asserts that its scenario manager is Infinite.

[dlpbc]

Is it better to move this scenario manager in bluebird_dt?

[presidente20]

Possibly but I am unsure if it is needed.

[nbarlowATI]

Looks good!
I think it is a good idea to have a FlightSchool env specifying Xplus + Infinite + default parameters for things like spawn rate. From running examples, the spawn rate still looks too high perhaps (or maybe just ramping up too fast...)

I don't think I'm qualified to comment on the reward stuff, but it looks reasonable to me from my limited knowledge.

I also think the notebooks are really good - would be a shame to just lose them by not including them in the final PR, though I recognise we do also need to be careful about having too much stuff in the "examples" directory. Maybe we organise into subdirectories? I would defer to @dlpbc on what would be the best approach.

I did get a crash in cell 7 ("Run the curriculum") of ppo_curriculum_agent though:

TypeError: route_progress_terminal_reward() takes 2 positional arguments but 3 were given

[nbarlowATI]

Is this value of spawn_rate_increase_interval a bit on the low side? I could imagine the rate would ramp up quite quickly, but maybe it's manageable if that's what you've been using?

[presidente20]

You're probably right but as my notebooks got to a maximum of two aircraft I'm not sure.

[nbarlowATI]

I think this will all be doable with the "Custom" scenario manager in bluebird-dt once the "sector-agnostic-scenario-managers" PR is merged.

[presidente20]

Looks good! I think it is a good idea to have a FlightSchool env specifying Xplus + Infinite + default parameters for things like spawn rate. From running examples, the spawn rate still looks too high perhaps (or maybe just ramping up too fast...)

I don't think I'm qualified to comment on the reward stuff, but it looks reasonable to me from my limited knowledge.

I also think the notebooks are really good - would be a shame to just lose them by not including them in the final PR, though I recognise we do also need to be careful about having too much stuff in the "examples" directory. Maybe we organise into subdirectories? I would defer to @dlpbc on what would be the best approach.

I did get a crash in cell 7 ("Run the curriculum") of ppo_curriculum_agent though:

TypeError: route_progress_terminal_reward() takes 2 positional arguments but 3 were given

Thanks for the comments Nick, I fixed the positional arguments crash in my latest commit. Not sure about the spawn rate, as I didn't get past two aircraft it never became an issue.