BUG: read parquet files with older pytz (DEP: keep lower pytz minimum version)

[jorisvandenbossche]

When we made pytz an optional dependency (#59089), we also bumped the minimum version (and later bumped it once more #62241). This causes issues with reading parquet files when someone does not have this required minimum version (the reported bug in #64978).

While we could solve this by improving the error message you get (so it is clear you have to update pytz), I also think there is not really a need to bump the minimum version here (pytz is mostly in maintenance mode AFAIK, and so the newer versions I assume are mostly updating the tz data)

For tzdata we actually decided to remove the minimum version altogether (#63335), but since pytz is still an API in addition to the tz data, I just kept the minimum version we had before in pandas 2.x (that should then at least not give problems for people upgrading from pandas 2 to 3 without upgrading pytz).

• closes BUG: read_parquet fails with tz aware data #64978
• Tests added and passed if fixing a bug or adding a new feature
  • Not easy to test, but verified it locally with installing an older pytz version.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[jorisvandenbossche]

For the case you would still have an older pytz than 2020.1, I also want to improve the behaviour or error message. At the moment, we have many cases where we don't actually check exactly that we have a pytz object (using treat_tz_as_pytz without checking pytz can be imported).
That means some parts of our API do kind of work with an older version of pytz, although might give wrong results, eg:

>>> import pytz
>>> pd.Timestamp(2012, 1, 1).tz_localize("UTC").tz_convert(pytz.timezone("Europe/Brussels"))
Timestamp('2012-01-01 00:18:00+0018', tz='Europe/Brussels')
# should be Timestamp('2012-01-01 01:00:00+0100', tz='Europe/Brussels')

Although then some other parts raise a error (like what read_parquet ran into).

I could easily make read_parquet also "work" (but return those wrong values), but so it seems to be a better behaviour to actually raise a proper error message up front when trying to use a pytz timezone when your pytz version is too old.

That is what I added in the last commit (but since this is somewhat of a breaking change, I could also keep that for a separate PR for 3.1)

[jbrockmendel]

Is it viable to write a test for this?

[jorisvandenbossche]

Not directly, since it is for the case where pytz is too old, and we don't have a CI build for that. I don't know if it would be possible to mock the pytz version in a test?

[jbrockmendel]

I don't know if it would be possible to mock the pytz version in a test?

I don't think so, especially if pyarrow is going to move to zoneinfo in the foreseeable future. Thanks for taking a look

[mroeschke]

Thanks @jorisvandenbossche

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 3.0.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 398e59c04ac30f4930bdbcdb0208e93e71d5a25a

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #65133: BUG: read parquet files with older pytz (DEP: keep lower pytz minimum version)'

4. Push to a named branch:

git push YOURFORK 3.0.x:auto-backport-of-pr-65133-on-3.0.x

5. Create a PR against branch 3.0.x, I would have named this PR:

"Backport PR #65133 on branch 3.0.x (BUG: read parquet files with older pytz (DEP: keep lower pytz minimum version))"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.