PIP Requirements Install is taking so long - MacOs Sillicon - Sequoia OS version

[alerimoficial]

Description

Dear All,
After upgrading to the new versions of PIP it is taking so long to install the requirements and it used to take a few seconds our a minute.

Expected behavior

No response

pip version

24.2

Python version

3.13

OS

Sequoia OS version

How to Reproduce

pip3 install --upgrade -r requirements.in

Output

echo "Upgrading Package Manager"
Upgrading Package Manager
python3 -m pip install --upgrade pip
Requirement already satisfied: pip in ./env/lib/python3.13/site-packages (24.2)
echo "Exporting Python project dependencies"
Exporting Python project dependencies
pip3 install --upgrade -r requirements.in
Collecting azure-identity (from -r requirements.in (line 1))
...
...
pandas
pydantic
python-dotenv
openai
plotly
dash[celery]
dash[diskcache]
dash_bootstrap_components
dash_extensions
dash_iconify
dash-mantine-components
dash_cytoscape
iso3166
numpy
gunicorn
bs4
dash-ag-grid
...
taking more than 10 minutes I reinstall my computer from scratch just to test it at all.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[alerimoficial]

make requirements
echo "Upgrading Package Manager"
Upgrading Package Manager
python3 -m pip install --upgrade pip
Requirement already satisfied: pip in ./env/lib/python3.13/site-packages (24.2)
echo "Exporting Python project dependencies"
Exporting Python project dependencies
pip3 install --upgrade -r requirements.in
Collecting azure-identity (from -r requirements.in (line 1))
Downloading azure_identity-1.19.0-py3-none-any.whl.metadata (80 kB)
Collecting msgraph-sdk (from -r requirements.in (line 2))
Downloading msgraph_sdk-1.10.0-py3-none-any.whl.metadata (13 kB)
Collecting openpyxl (from -r requirements.in (line 3))
Downloading openpyxl-3.1.5-py2.py3-none-any.whl.metadata (2.5 kB)
Collecting requests (from -r requirements.in (line 4))
Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting sqlitedict (from -r requirements.in (line 5))
Downloading sqlitedict-2.1.0.tar.gz (21 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting bs4 (from -r requirements.in (line 6))
Using cached bs4-0.0.2-py2.py3-none-any.whl.metadata (411 bytes)
Collecting pandas (from -r requirements.in (line 7))
Using cached pandas-2.2.3-cp313-cp313-macosx_11_0_arm64.whl.metadata (89 kB)
Collecting typing_extensions (from -r requirements.in (line 8))
Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting pyodbc (from -r requirements.in (line 9))
Downloading pyodbc-5.1.0.tar.gz (115 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [22 lines of output]
Traceback (most recent call last):
File "/Users/alerim/Documents/GitHub/imed-earth/env/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
~~~~^^
File "/Users/alerim/Documents/GitHub/imed-earth/env/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/alerim/Documents/GitHub/imed-earth/env/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/private/var/folders/j4/9twvr7ys6_13m0dr8mh8lxy40000gn/T/pip-build-env-carjqckn/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/j4/9twvr7ys6_13m0dr8mh8lxy40000gn/T/pip-build-env-carjqckn/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
self.run_setup()
~~~~~~~~~~~~~~^^
File "/private/var/folders/j4/9twvr7ys6_13m0dr8mh8lxy40000gn/T/pip-build-env-carjqckn/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 318, in run_setup
exec(code, locals())
~~~~^^^^^^^^^^^^^^^^
File "", line 179, in
File "", line 35, in main
File "", line 135, in get_compiler_settings
ValueError: not enough values to unpack (expected 2, got 0)
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
make: *** [requirements] Error 1

[notatallshaw]

You need to provide your full requirements.in for anyone to be able to investigate why it is taking so long. I see this in your log:

Collecting pyodbc (from -r requirements.in (line 9))

But you've not listed pyodbc as one of your requirements in your original post, so you're missing some of the requirements.

As for the error, the latest version of pyodbc does not support Python 3.13: https://pypi.org/project/pyodbc/#files. You must use Python 3.12 or lower to install pyodbc.

[alerimoficial]

@notatallshaw , you can find bellow:

openpyxl
requests
sqlitedict
bs4
pandas
typing_extensions
pyodbc
pdfquery
google-generativeai
chevron
openai
pydantic
pypdf

[alerimoficial]

I will downgrade to check if it is why taking so long. Thanks for your reply.

[alerimoficial]

No, it is taking so long with any requirements I try to install.

[potiuk]

No, it is taking so long with any requirements I try to install.

I think it might be your environmental issue @alerimoficial . One of the problems I saw people had on Mac OS (and you might experience it) is when they migrate their Python installation (created using pyenv or other tools) from Intel to ARM based laptop. When you do it by backup/restore mechanism, you might end up with Intel binaries and Python installation on ARM based Mac OS.

Unfortunately, the problem is that such migration works, because MacOS has a built-in emulation and you can run Intel binaries on ARM device. It works but generally 16x or so slower. I guess you are experiencing this - but this is wild guess only.

You can check it by runing:

python -c "import platform; print(platform.uname().machine)"

If you are on ARM and it prints amd64 or x86, this is is your problem.

[potiuk]

And if that is your problem - reinstalling all the software stack on a clean MacOS is highly recommended - you can have other software having similar problems. I saw some cases where people had "ARM python" but "Intel" IntelliJ (where IntelliJ both Java and Python libraries were running using emulation. The result was very sluggish IDE.

[alerimoficial]

@potiuk @notatallshaw , I just found that python 3.13 is may not compiled all the packages and libraries and it is taking too long for many users. I did not find a correction for it yet.

[potiuk]

@potiuk @notatallshaw , I just found that python 3.13 is may not compiled all the packages and libraries and it is taking too long for many users. I did not find a correction for it yet.

Obviously. Because Airlfow does not support Python 3.13 yet. And it will not - until almost all the 700+ dependencies of Airflow will support Python 3.13. Usually, it takes about 6 months before the "critical mass" of those dependencies catch-up, and taking into account a number of breaking changes (mostly removals of some "batteries") in Python 3.13, it is likely to take maybe even longer this time.

In the few recent versions of Airflow 2 we even explicitly excluded Python 3.13 https://github.com/apache/airflow/blob/v2-10-test/pyproject.toml#L42 (this is the tip of v2-10-test branch which we use to release 2.10.* patchlevel releases from.

So - this will take quite some time to get Python 3.13 support, but if you want - feel free to open and monitor a PR that will take care of this. I have not done it yet, because I know there is no point currently - maybe in February/ March - or maybe a bit earlier, because with Airlfow 3 we are likely to remove some dependencies that were problematic.

Similar PR for 3.12 has been opened in March 2024 (when we knew we are close to get the right set of dependencies to support it) and merged in April 2024 apache/airflow#38025 (so pretty much exactly 6 months after 3.12 was available) and even until now, Python 3.12 is excluded for Apache Beam because dill they are pinned to is not supporting 3.12 apache/airflow#41990

Feel free to attempt to try it @alerimoficial