PERF: Use optimized take path for non-nanosecond datetime64/timedelta64#65165

jbrockmendel wants to merge 2 commits intopandas-dev:mainpandas-dev/pandas:mainfrom

jbrockmendel:ref-special-2jbrockmendel/pandas:ref-special-2Copy head branch name to clipboard