(cli): deploy waits forever when an AWS API call gets no response, because requestTimeout only logs a warning #2097

Describe the bug

cdk deploy can wait forever on a single AWS API call that never gets a response.

sdkRequestHandler in packages/@aws-cdk/toolkit-lib/lib/api/aws-auth/awscli-compatible.ts builds every SDK client's request handler with connectionTimeout: 10_000 and requestTimeout: 300_000, but without throwOnRequestTimeout and without socketTimeout. In Smithy's NodeHttpHandler, a requestTimeout without throwOnRequestTimeout only logs a warning. The request is never destroyed, so the SDK's retry strategy never sees an error. If the connection has silently died after it was established, the CLI waits on it forever.

We have hit this twice in production deploys from GitHub Actions in the last two weeks, out of about 1,000 deploys since July. Both times, the CLI stopped reporting progress for one stack, logged this warning, and then printed nothing more:

@smithy/node-http-handler - [WARN] a request has exceeded the configured 300000 ms requestTimeout. Init client requestHandler with throwOnRequestTimeout=true to turn this into an error.

• Oct 19 2026: the warning appeared at 13:31:03 UTC. The job was cancelled by hand about 42 minutes later.
• Oct 27 2026: the stack reached UPDATE_COMPLETE in CloudFormation at 02:47:24 UTC. The warning appeared at 02:49:35, and the job ran until GitHub's 6-hour job timeout cancelled it at 08:43:59. Because our deploy job is serialized, it also blocked the next release for almost 6 hours.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

An AWS API call that gets no response should time out, be retried by the SDK's retry strategy like any other transient network error, and fail the command with an error if the retries run out.

Current Behavior

The CLI keeps waiting on the same request with no further output. It never retries and never exits, until something outside the CLI kills it. On 2.1107.0 it logs @smithy/node-http-handler - [WARN] a request has exceeded the configured 300000 ms requestTimeout after 5 minutes. On 2.1146.1 it printed nothing at the default verbosity.

Reproduction Steps

1. Start a server that accepts TCP connections and never replies. Save this as blackhole.py and run python3 blackhole.py:

import socket, threading

s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("127.0.0.1", 8999))
s.listen(50)
held = []

def hold(c):
    held.append(c)
    try:
        while c.recv(65536):
            pass
    except OSError:
        pass

while True:
    c, _ = s.accept()
    print("accepted a connection; never replying", flush=True)
    threading.Thread(target=hold, args=(c,), daemon=True).start()

2. In another shell, point the CLI at it. The credentials are dummies and nothing reaches AWS:

AWS_ACCESS_KEY_ID=AKIAEXAMPLEEXAMPLE AWS_SECRET_ACCESS_KEY=dummy AWS_REGION=us-east-1 \
AWS_ENDPOINT_URL=http://127.0.0.1:8999 CDK_DISABLE_CLI_TELEMETRY=1 \
npx aws-cdk@2.1146.1 bootstrap aws://123456789012/us-east-1 --no-notices

3. The server accepts one connection, and the CLI prints Bootstrapping environment aws://123456789012/us-east-1... and then waits.

We ran this on 2.1107.0 and 2.1146.1 and killed each after 7 minutes. Neither version opened a second connection, so the call was never retried. 2.1107.0 logged the requestTimeout warning at the 5-minute mark. 2.1146.1 printed no warning at the default verbosity, so on the latest release the hang has no output at all.

Possible Solution

Set a socketTimeout in sdkRequestHandler, for example 60 to 120 seconds. When no data moves on the connection for that long, Smithy destroys the request and rejects with a TimeoutError. isTransientError classifies that as retryable, so the existing retry strategy handles it. Unlike requestTimeout, it does not fire on a slow request that is still sending data, such as a large asset upload.

Setting throwOnRequestTimeout: true would also work, because the resulting error (name TimeoutError, code ETIMEDOUT) is retryable too. But requestTimeout runs from the start of the request until the response headers arrive, including the whole request body, so it would also abort a healthy upload that takes longer than 5 minutes.

Either way, it would help to make these timeouts configurable. Today the only HTTP setting the CLI accepts from outside is the proxy agent.

Additional Information/Context

Both occurrences were on the same third-party GitHub Actions runner provider, and we have not seen it on GitHub-hosted runners. Whatever drops the connection, the CLI should recover rather than wait forever.

Related, but different:

• (cli) CDK CLI hangs indefinitely after command completion in CI environments (GitLab/Kubernetes) - v2.1034.0+ #1217 is about the process not exiting after the command has finished. Here the command never finishes: the CLI is still waiting for a stack's status.
• fix(toolkit-lib): CLI hangs when CreateChangeSet returns InternalFailure #1500 capped the retry backoff for calls that return an error. A call that gets no response at all never reaches the retry logic.

CDK CLI Version

2.1107.0 (also reproduced on 2.1146.1)

Framework Version

aws-cdk-lib 2.240.0

Node.js Version

24.22.1

OS

Ubuntu 24.04 (GitHub Actions runner)

Language

TypeScript

Language Version

TypeScript 5.9.3

Other information

No response