Airflow Python 3.12 support

[dirrao]

Description

Right now, airflow is not officially supporting the Python 3.12 version

Use case/motivation

Python software stack update.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Taragolis]

This kind of task not a straight forward, required to someone check what kind of core / providers dependencies not ready for 3.12 yet, or we use outdated dependencies or some deprecated stuff:

At current moment we know at lest incompatible with that pendulum 2 doesn't support of python 3.12, but this one could be solved by #36281

[potiuk]

At current moment we know at lest incompatible with that pendulum 2 doesn't support of python 3.12, but this one could be solved by #36281

Yeah. I believe this is the only "serious" blocker. We have option to exclude certain providers if they turn out to be incompatible ("excluded-python-versions" flag in provider.yaml) and we've done that in the past. So I'd say @dirrao if you want to take the lead on that - open a PR with added Python 3.12 in Breeze and we can keep it open and rebased until all the tests pass (might need to exclude some providers).

I think it's about the right time to attempt it - usually it takes 3-4 months for most important dependencies to catch up. We might also need to bump some more deps, not only Pendulum, but yeah - Pendulum is the "hard" blocker.

[dirrao]

At current moment we know at lest incompatible with that pendulum 2 doesn't support of python 3.12, but this one could be solved by #36281

Yeah. I believe this is the only "serious" blocker. We have option to exclude certain providers if they turn out to be incompatible ("excluded-python-versions" flag in provider.yaml) and we've done that in the past. So I'd say @dirrao if you want to take the lead on that - open a PR with added Python 3.12 in Breeze and we can keep it open and rebased until all the tests pass (might need to exclude some providers).

I think it's about the right time to attempt it - usually it takes 3-4 months for most important dependencies to catch up. We might also need to bump some more deps, not only Pendulum, but yeah - Pendulum is the "hard" blocker.

Yes. I started looking into updating Breeze.

[dirrao]

I have updated the breeze. However when I run this test with 3.12, I am seeing python3.12 image not found. How to publish this image?

Error response from daemon: No such image: ghcr.io/apache/airflow/main/ci/python3.12:latest

[potiuk]

You should be able to build it locally with breeze ci-image build --python 3.12 (after you add it to configuration)

[potiuk]

You might see some missing cache etc. but they should be just warnings - and you can always disable cache with build command flags (use --help to see the flags)

[dirrao]

I can run after making the changes in Breeze. However, I faced two problems as follows.

1. The build is looking for the remote constraints file. I can resolve it by using a local constraints file.
   https://raw.githubusercontent.com/apache/airflow/constraints-3.12.txt/constraints-source-providers-3.12.txt
2. Pip is not able to resolve the remote apache airflow package version. Not sure we can if can bypass it.
   #0 22.01 ERROR: Package 'apache-airflow' requires a different Python: 3.12.1 not in '<3.12,~=3.8'

Can you let me know the next steps?

[potiuk]

Oh yeah.

1. I just solved the first one by pushing copy of 3.11 constraints which are "good enough" to start with - this should be fixed now
2. the second is I think the result of the build optimilsation we have for the build to use main for our. I opened PR Remove Python 3.12 limitation in order to test Python 3.12 builds #36595 to remove the limt from main and once merged this problem should be gone.

[potiuk]

For 1: see this commit 3f3ed81

(and generally main constraints history is here https://github.com/apache/airflow/commits/constraints-main/)

[potiuk]

Merged. You can continue your quest @dirrao