[hoangchunghien]

typeshed's "stub uploader: tests" job failed on python/typeshed#16623 with a network error unrelated to that change:

FAILED tests/test_integration.py::test_build_wheel[Authlib] - requests.exceptions.SSLError:
HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/Authlib/json
(Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1032)')))

The failing frame is metadata.py:423, the requests.get(f"{project_url}/json") in get_upstream_release_metadata. get_version.fetch_pypi_versions already retries through a Session with a urllib3 Retry, but the three requests.get() calls in metadata.py have no retry and no timeout. One TLS hiccup in any of the ~675 test_build_wheel fetches fails the job.

This moves that retry policy into stub_uploader/pypi.py as pypi_get() and uses it for all four PyPI requests:

• Same policy as get_version today: Retry(total=5, status_forcelist=[429, 500, 502, 503, 504]) with a 3-second timeout. urllib3 counts connection and TLS errors against total, so an SSLEOFError like the one above is retried.
• raise_on_status=False: after the last retry the final response comes back rather than a RetryError. validate_pypi_response() therefore still reports got 503 as before (test_pypi_error covers this), and fetch_pypi_versions still raises its own error.
• One cached Session, so connections are reused across lookups. This also matters for the sdist download, which streams (stream=True, then resp.raw.read()) and needs the pool to stay open after get() returns. That call keeps timeout=None, as before, so a large sdist isn't cut off.

Tests:

• New tests/test_pypi.py runs against a local HTTP server: a 503 that recovers, a dropped connection that recovers, retries running out (the 503 is returned after RETRIES + 1 requests), and the shared session. With total=0, the three retry tests fail.
• test_release_metadata.py now patches stub_uploader.metadata.pypi_get instead of requests.get, with the same call assertions.
• mypy --strict -p stub_uploader -p tests and black 26.5.1 are clean. python -Werror -m pytest tests against typeshed main: 679 passed, including the live PyPI integration tests.

[JelleZijlstra]

Could we use a generous finite timeout here? The read timeout measures inactivity, not total download duration, so large downloads don't require timeout=None.

[JelleZijlstra]

Could we add a small backoff, e.g. backoff_factor=0.2, so a brief outage doesn’t exhaust all retries immediately? Retry-After is already respected, but isn’t always present.

[hoangchunghien]

Thanks, both done in 776223b:

• Downloads now use a 60 s read timeout (DOWNLOAD_TIMEOUT). Since it limits inactivity rather than the whole transfer, a large sdist only fails if PyPI stops sending.
• Retries back off with backoff_factor=0.2, so the waits are 0.4, 0.8, 1.6 and 3.2 s after the first immediate retry, about 6 s in total before giving up. Retry-After still takes precedence when it is sent.

The tests now record urllib3's sleeps instead of waiting for them, so the suite stays fast, and one test checks the backoff sequence. Another checks that the sdist download passes the finite timeout.

[srittau]

It makes sense to me to extract the fetching logic and retry more aggressively.

We don't need extensive tests that basically just test that requests works as expected, i.e. I don't think we need the test_pypi.py file.

[srittau]

I hope we don't do any fetching of http resources. Let's remove this adapter. (I think it was needed for the tests.)

[hoangchunghien]

Thanks, both done in f3b0550: the http:// mount is gone, so the retrying session only serves https://, and tests/test_pypi.py is removed. The suite passes and mypy is clean.

[srittau]

Thanks!