Merge remote-tracking branch 'upstream/main' into missing-value-after-comparison

* upstream/main:
  PERF: use lookup instead of hash_inner_join for merge with unique right keys (pandas-dev#64691)
  BUG : update `SeriesGroupBy.ohlc()` to honor `as_index=False` (pandas-dev#65141)
  PERF: Use DataFrame-level reductions in DataFrame.agg with list of funcs (pandas-dev#65031)
  DOC: document required external libraries in read_* I/O docstrings (pandas-dev#65143)
  DOC: improve MultiIndex.is_monotonic_increasing/decreasing docstrings (pandas-dev#65154)
  BUG: Raise ValueError for non-boolean numeric_only in DataFrame/Series reductions (GH#53098) (pandas-dev#65131)
  BUG: Timedelta.round() raises ZeroDivisionError when internal unit is 's' and target frequency is sub-second (pandas-dev#64836)
  ENH: Add replace method to Index (closes pandas-dev#19495) (pandas-dev#65099)
  PERF: improve StringArray.isna (pandas-dev#57733)
  BUG: read parquet files with older pytz (DEP: keep lower pytz minimum version) (pandas-dev#65133)
  DEPR: deprecate dates-with-datetime64 in _maybe_downcast_for_indexing (pandas-dev#64871)
  DOC: note that DataFrame.values is not writeable (pandas-dev#65142)
  CLN: Update groupby observed defaults (pandas-dev#65148)
  PERF: avoid materializing values[indexer] in Block.setitem (pandas-dev#64251)
  DOC: update GroupBy.sum/min/max See Also sections (pandas-dev#65144)

Author: Sharl0tteIsTaken

ci/deps/actions-311-minimum_versions.yaml

@@ -50,7 +50,7 @@ dependencies:
   - pyreadstat=1.2.8
   - pytables=3.10.1
   - python-calamine=0.3.0
-  - pytz=2024.2
+  - pytz=2020.1
   - pyxlsb=1.0.10
   - s3fs=2024.10.0
   - scipy=1.14.1

ci/deps/actions-311.yaml

@@ -49,7 +49,7 @@ dependencies:
   - pyreadstat>=1.2.8
   - pytables>=3.10.1
   - python-calamine>=0.3.0
-  - pytz>=2024.2
+  - pytz>=2020.1
   - pyxlsb>=1.0.10
   - s3fs>=2024.10.0
   - scipy>=1.14.1

ci/deps/actions-312.yaml

@@ -49,7 +49,7 @@ dependencies:
   - pyreadstat>=1.2.8
   - pytables>=3.10.1
   - python-calamine>=0.3.0
-  - pytz>=2024.2
+  - pytz>=2020.1
   - pyxlsb>=1.0.10
   - s3fs>=2024.10.0
   - scipy>=1.14.1

ci/deps/actions-313-downstream_compat.yaml

@@ -48,7 +48,7 @@ dependencies:
   - pyreadstat>=1.2.8
   - pytables>=3.10.1
   - python-calamine>=0.3.0
-  - pytz>=2024.2
+  - pytz>=2020.1
   - pyxlsb>=1.0.10
   - s3fs>=2024.10.0
   - scipy>=1.14.1

ci/deps/actions-313.yaml

@@ -50,7 +50,7 @@ dependencies:
   - pyreadstat>=1.2.8
   - pytables>=3.10.1
   - python-calamine>=0.3.0
-  - pytz>=2024.2
+  - pytz>=2020.1
   - pyxlsb>=1.0.10
   - s3fs>=2024.10.0
   - scipy>=1.14.1

ci/deps/actions-314.yaml

@@ -48,7 +48,7 @@ dependencies:
   - pyreadstat>=1.2.8
   - pytables>=3.10.1
   - python-calamine>=0.3.0
-  - pytz>=2024.2
+  - pytz>=2020.1
   - pyxlsb>=1.0.10
   - s3fs>=2024.10.0
   - scipy>=1.14.1

doc/source/getting_started/install.rst

@@ -379,5 +379,5 @@ Installable with ``pip install "pandas[timezone]"``
 ========================================== ================== =================== ==============================================
 Dependency                                 Minimum Version    pip extra           Notes
 ========================================== ================== =================== ==============================================
-`pytz <https://github.com/stub42/pytz>`__  2024.2             timezone            Alternative timezone library to ``zoneinfo``.
+`pytz <https://github.com/stub42/pytz>`__  2020.1             timezone            Alternative timezone library to ``zoneinfo``.
 ========================================== ================== =================== ==============================================

doc/source/whatsnew/v3.0.3.rst

@@ -14,6 +14,8 @@ including other versions of pandas.
 Fixed regressions
 ~~~~~~~~~~~~~~~~~
 
+- Fixed reading of Parquet files with timezone-aware timestamps or localizing of a timeseries with a ``pytz`` timezone when an older version of ``pytz`` was installed (:issue:`64978`)
+
 .. ---------------------------------------------------------------------------
 .. _whatsnew_303.bug_fixes:
 

doc/source/whatsnew/v3.1.0.rst

@@ -31,6 +31,7 @@ Other enhancements
 - :class:`Period` now supports f-string formatting via ``__format__``, e.g. ``f"{period:%Y-%m}"`` (:issue:`48536`)
 - :meth:`.DataFrameGroupBy.agg` now allows for the provided ``func`` to return a NumPy array (:issue:`63957`)
 - Added :meth:`ExtensionArray.count` (:issue:`64450`)
+- Added :meth:`Index.replace` method to support value replacement functionality similar to :meth:`Series.replace` (:issue:`19495`)
 - Display formatting for float sequences in DataFrame cells now respects the ``display.precision`` option (:issue:`60503`).
 - Improved the precision of float parsing in :func:`read_csv` (:issue:`64395`)
 - Improved the string ``repr`` of :class:`pd.core.arrays.SparseArray` (:issue:`64547`)
@@ -103,6 +104,7 @@ Deprecations
 - Deprecated arithmetic operations between pandas objects (:class:`DataFrame`, :class:`Series`, :class:`Index`, and pandas-implemented :class:`ExtensionArray` subclasses) and list-likes other than ``list``, ``np.ndarray``, :class:`ExtensionArray`, :class:`Index`, :class:`Series`, :class:`DataFrame`. For e.g. ``tuple`` or ``range``, explicitly cast these to a supported object instead. In a future version, these will be treated as scalar-like for pointwise operation (:issue:`62423`)
 - Deprecated automatic dtype promotion when reindexing with a ``fill_value`` that cannot be held by the original dtype. Explicitly cast to a common dtype instead (:issue:`53910`)
 - Deprecated grouping by an index level name when the name matches multiple levels of a :class:`MultiIndex`. Use the level number instead (:issue:`49434`)
+- Deprecated implicit conversion of ``datetime.date`` objects to :class:`Timestamp` when indexing or joining a :class:`DatetimeIndex`. Use :func:`to_datetime` to explicitly convert to :class:`DatetimeIndex` instead (:issue:`62158`)
 - Deprecated passing a ``dict`` to :meth:`DataFrame.from_records`, use the :class:`DataFrame` constructor or :meth:`DataFrame.from_dict` instead (:issue:`22025`)
 - Deprecated passing a non-dict (e.g. a list of dicts) to :meth:`DataFrame.from_dict`. Use the :class:`DataFrame` constructor instead (:issue:`58862`)
 - Deprecated passing unnecessary ``*args`` and ``**kwargs`` to :meth:`.GroupBy.cumsum`, :meth:`.GroupBy.cumprod`, :meth:`.GroupBy.cummin`, :meth:`.GroupBy.cummax`, :meth:`.SeriesGroupBy.skew`, :meth:`.DataFrameGroupBy.skew`, :meth:`.SeriesGroupBy.take`, and :meth:`.DataFrameGroupBy.take`. The ``skipna`` parameter for the cum* methods is now an explicit keyword argument (:issue:`50407`)
@@ -114,6 +116,12 @@ Deprecations
 - Deprecated the ``xlrd`` and ``pyxlsb`` engines in :func:`read_excel`. Use ``engine="calamine"`` instead (:issue:`56542`)
 - Deprecated the default value of ``exact`` in :func:`assert_index_equal`; in a future version this will default to ``True`` instead of "equiv" (:issue:`57436`)
 - Deprecated the default value of ``track_times`` in :meth:`HDFStore.put`. In a future version, the default will change from ``True`` to ``False`` so that HDF5 files are deterministic by default (:issue:`51456`)
+- Deprecated passing a non-boolean value for ``numeric_only`` to :meth:`DataFrame.mean`,
+  :meth:`DataFrame.min`, :meth:`DataFrame.max`, :meth:`DataFrame.median`,
+  :meth:`DataFrame.skew`, :meth:`DataFrame.kurt`, :meth:`DataFrame.std`,
+  :meth:`DataFrame.var`, :meth:`DataFrame.sem`, :meth:`DataFrame.sum`,
+  :meth:`DataFrame.prod` (and their :class:`Series` equivalents); this will raise in
+  a future version of pandas (:issue:`53098`)
 - Deprecated the inference of ``datetime64`` dtype from data containing ``datetime.date`` objects when used in comparisons or :meth:`Index.equals` with :class:`DatetimeIndex`. Use :func:`pandas.to_datetime` to explicitly convert to ``datetime64`` instead (:issue:`65056`)
 -
 
@@ -131,6 +139,7 @@ Performance improvements
 - Performance improvement in :func:`bdate_range` and :func:`date_range` with ``freq="B"`` or ``freq="C"`` (business day frequencies) (:issue:`16463`)
 - Performance improvement in :func:`infer_freq` (:issue:`64463`)
 - Performance improvement in :func:`merge` and :meth:`DataFrame.join` for many-to-many joins with ``sort=False`` (:issue:`56564`)
+- Performance improvement in :func:`merge` for many-to-one joins with unique right keys (:issue:`38418`)
 - Performance improvement in :func:`merge` with ``how="cross"`` (:issue:`38082`)
 - Performance improvement in :func:`merge` with ``how="left"`` (:issue:`64370`)
 - Performance improvement in :func:`merge` with ``sort=False`` for single-key ``how="left"``/``how="right"`` joins when the opposite join key is sorted, unique, and range-like (:issue:`64146`)
@@ -171,6 +180,9 @@ Performance improvements
 - Performance improvement in :meth:`DataFrame.loc` and :meth:`DataFrame.iloc`
   setitem with a 2D list-of-lists value by avoiding a wasteful round-trip
   through an intermediate object array (:issue:`64229`).
+- Performance improvement in :meth:`Series.iloc` and :meth:`DataFrame.iloc`
+  when setting datetimelike values into object-dtype data with list-like
+  indexers (:issue:`64250`).
 
 .. ---------------------------------------------------------------------------
 .. _whatsnew_310.bug_fixes:
@@ -208,7 +220,7 @@ Timedelta
 ^^^^^^^^^
 - Bug in :class:`DateOffset` where ``DateOffset(1)`` and ``DateOffset(days=1)`` returned different results near daylight saving time transitions (:issue:`61862`)
 - Bug in :func:`to_timedelta` where passing ``np.str_`` objects would fail in Cython string parsing (:issue:`48974`)
--
+- Fixed regression in :meth:`Timedelta.round`, :meth:`Timedelta.floor`, and :meth:`Timedelta.ceil` raising ``ZeroDivisionError`` for sub-second ``freq`` (:issue:`64828`)
 
 Timezones
 ^^^^^^^^^
@@ -295,6 +307,7 @@ Groupby/resample/rolling
 - Bug in :meth:`.GroupBy.any` and :meth:`.GroupBy.all` returning ``NaN`` with ``float64`` dtype for unobserved categorical groups on NumPy ``bool`` data instead of the boolean identity value with ``bool`` dtype (:issue:`65100`)
 - Bug in :meth:`.Rolling.skew` and :meth:`.Rolling.kurt` (and their :class:`GroupBy` counterparts) returning ``0.0`` and ``-3.0`` respectively for degenerate windows or groups; these now return ``NaN`` (:issue:`62864`)
 - Bug in :meth:`.Rolling.skew` and :meth:`.Rolling.kurt` returning ``NaN`` for low-variance windows (:issue:`62946`)
+- Bug in :meth:`.SeriesGroupBy.ohlc` ignoring ``as_index=False`` (:issue:`65140`)
 - Bug in :meth:`DataFrame.groupby` with a :class:`Grouper` with ``freq`` raising ``AttributeError`` when all grouping keys are ``NaT`` (:issue:`43486`)
 - Bug in :meth:`Series.resample` and :meth:`DataFrame.resample` where same-frequency resampling with monthly, quarterly, or annual frequencies bypassed aggregation, returning the original values instead of the aggregation result (:issue:`18553`)
 

environment.yml

@@ -49,7 +49,7 @@ dependencies:
   - pyreadstat>=1.2.8
   - pytables>=3.10.1
   - python-calamine>=0.3.0
-  - pytz>=2024.2
+  - pytz>=2020.1
   - pyxlsb>=1.0.10
   - s3fs>=2024.10.0
   - scipy>=1.14.1