Skip to content

Tutorial for async batched inference using sagemaker#2637

Open
dwarez wants to merge 1 commit into
mainfrom
async-inference-tutorial
Open

Tutorial for async batched inference using sagemaker#2637
dwarez wants to merge 1 commit into
mainfrom
async-inference-tutorial

Conversation

@dwarez

@dwarez dwarez commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Note

Low Risk
Documentation-only addition (Jupyter notebook); no production code, auth, or library behavior changes.

Overview
Adds a new SageMaker SDK tutorial notebook under docs/sagemaker/notebooks/sagemaker-sdk/async-inference-embedding-tei/ that walks through batch corpus embedding for semantic search using asynchronous inference and Text Embeddings Inference (TEI).

The notebook explains async vs sync inference, then implements an end-to-end flow: load a slice of sentence-transformers/natural-questions, build and deploy BAAI/bge-small-en-v1.5 via ModelBuilder with AsyncInferenceConfig, configure scale-to-zero autoscaling (backlog target tracking plus wake-from-zero alarms), submit batched invoke_async jobs via S3, poll results, write a JSONL index to S3, and run a cosine-similarity retrieval smoke test. Optional SNS hooks and a CLEANUP-controlled teardown of endpoint, model, autoscaling, and CloudWatch alarms are included.

Reviewed by Cursor Bugbot for commit 151d62d. Bugbot is set up for automated code reviews on this repo. Configure here.

Signed-off-by: DWarez <dario.salvati@huggingface.co>
@dwarez dwarez requested review from alvarobartt and ehcalabres July 9, 2026 15:20
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants