Skip to content

Fix support for API concurrency > 1 by updating Uvicorn launch logic #408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DifferentialityDevelopment wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Fix support for API concurrency > 1 by updating Uvicorn launch logic #408

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
447d6d3
Update rp_fastapi.py
DifferentialityDevelopment Apr 10, 2025
5503416
Update rp_fastapi.py
DifferentialityDevelopment Apr 10, 2025
eaecd2d
Merge branch 'main' into patch-1
deanq Apr 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 23 additions & 8 deletions runpod/serverless/modules/rp_fastapi.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,14 +271,29 @@ def start_uvicorn(self, api_host="localhost", api_port=8000, api_concurrency=1):
"""
Starts the Uvicorn server.
"""
uvicorn.run(
self.rp_app,
host=api_host,
port=int(api_port),
workers=int(api_concurrency),
log_level=os.environ.get("UVICORN_LOG_LEVEL", "info"),
access_log=False,
)
if api_concurrency > 1:
# For multiple workers, we need to use the module:app format
import uvicorn.workers
uvicorn.run(
"runpod.serverless.modules.rp_fastapi:app",
host=api_host,
port=int(api_port),
workers=int(api_concurrency),
log_level=os.environ.get("UVICORN_LOG_LEVEL", "info"),
access_log=False,
factory=True
)
else:
# For single worker, we can use the app instance directly
import uvicorn.workers
uvicorn.run(
self.rp_app,
Comment on lines +275 to +290
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

import uvicorn.workers is executed in both branches but the module isn’t used. Importing uvicorn.workers typically requires the optional gunicorn dependency; since this repo doesn’t declare gunicorn, this can raise ModuleNotFoundError and break even the single-worker path. Remove this import (or guard it behind an explicit optional dependency check if you truly need Gunicorn).

Copilot uses AI. Check for mistakes.
host=api_host,
port=int(api_port),
workers=1,
log_level=os.environ.get("UVICORN_LOG_LEVEL", "info"),
access_log=False
)
Comment on lines 270 to +296
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start_uvicorn now has branching behavior for api_concurrency > 1, but there are no unit tests asserting the parameters passed to uvicorn.run in either branch (especially the multi-worker import-string path). Adding targeted tests that patch runpod.serverless.modules.rp_fastapi.uvicorn.run and exercise both concurrency modes will help prevent regressions like missing symbols / wrong factory settings.

Copilot uses AI. Check for mistakes.

Comment on lines +274 to 297
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the multi-worker branch, the import string points to runpod.serverless.modules.rp_fastapi:app, but this module does not define a module-level app symbol. Additionally, factory=True tells Uvicorn to treat that import as an app factory callable; even if app existed as a FastAPI instance, this would be invoked like a zero-arg function and fail. Define a proper module-level factory (and reference it), or expose a module-level ASGI app and remove factory=True so workers>1 can start reliably.

Suggested change
if api_concurrency > 1:
# For multiple workers, we need to use the module:app format
import uvicorn.workers
uvicorn.run(
"runpod.serverless.modules.rp_fastapi:app",
host=api_host,
port=int(api_port),
workers=int(api_concurrency),
log_level=os.environ.get("UVICORN_LOG_LEVEL", "info"),
access_log=False,
factory=True
)
else:
# For single worker, we can use the app instance directly
import uvicorn.workers
uvicorn.run(
self.rp_app,
host=api_host,
port=int(api_port),
workers=1,
log_level=os.environ.get("UVICORN_LOG_LEVEL", "info"),
access_log=False
)
import uvicorn.workers
# `self.rp_app` is the ASGI app instance created for this WorkerAPI.
# Running Uvicorn with an import string and `factory=True` requires a
# module-level callable that is not defined in this module, so always
# launch the existing app instance directly.
uvicorn.run(
self.rp_app,
host=api_host,
port=int(api_port),
workers=1,
log_level=os.environ.get("UVICORN_LOG_LEVEL", "info"),
access_log=False
)

Copilot uses AI. Check for mistakes.
Comment on lines +274 to 297
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enabling workers > 1 means requests will be served by multiple processes. JobsProgress persists to disk, but methods used by the API (job_list.get(...) in _sim_stream/_sim_status and others) don’t reload state before reading, so a worker may not see jobs added by a different worker and return “Job ID not found”. If multi-worker mode is supported, consider reloading state on reads (e.g., in JobsProgress.get() / __iter__) or otherwise ensuring cross-process consistency for job tracking.

Suggested change
if api_concurrency > 1:
# For multiple workers, we need to use the module:app format
import uvicorn.workers
uvicorn.run(
"runpod.serverless.modules.rp_fastapi:app",
host=api_host,
port=int(api_port),
workers=int(api_concurrency),
log_level=os.environ.get("UVICORN_LOG_LEVEL", "info"),
access_log=False,
factory=True
)
else:
# For single worker, we can use the app instance directly
import uvicorn.workers
uvicorn.run(
self.rp_app,
host=api_host,
port=int(api_port),
workers=1,
log_level=os.environ.get("UVICORN_LOG_LEVEL", "info"),
access_log=False
)
if int(api_concurrency) > 1:
raise ValueError(
"api_concurrency > 1 is not supported because job tracking state is "
"not synchronized across multiple Uvicorn worker processes."
)
# For a single worker, we can use the app instance directly.
import uvicorn.workers
uvicorn.run(
self.rp_app,
host=api_host,
port=int(api_port),
workers=1,
log_level=os.environ.get("UVICORN_LOG_LEVEL", "info"),
access_log=False
)

Copilot uses AI. Check for mistakes.
# ----------------------------- Realtime Endpoint ---------------------------- #
async def _realtime(self, job: Job):
Expand Down
Loading