Skip to content
BranchesTags

Repository files navigation

Autogluon-Productionized

This project implements a serverless MLOps pipeline using AWS SageMaker, AWS Lambda, and AutoGluon.

Strict Requirement: This entire infrastructure is designed to be provisioned using the AWS CLI. Follow the steps below to set up your environment from scratch.

Prerequisites

  1. Git Clone the repository:
git clone https://github.com/Chi-SquareX/autogluon-productionized.git
cd <YOUR_REPO_NAME>
  1. Install Dependencies:
    • AWS CLI v2 installed and configured (aws configure).
    • Docker Desktop running.
    • Python 3.9+ & pip.
    • zip utility (for packaging Lambda).

Step 1: Environment Setup (Variables)

export AWS_REGION="us-east-1" # change this accordingly
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export BUCKET_NAME="ag-pipeline-${ACCOUNT_ID}"
export LAMBDA_ROLE_NAME="AutoGluonLambdaRole"
export SAGEMAKER_ROLE_NAME="AutoGluonSageMakerRole"
export LAMBDA_FUNC_NAME="AutoGluonOrchestrator"

Step 2: Storage Setup (S3)

aws s3api create-bucket --bucket $BUCKET_NAME --region $AWS_REGION
cd scripts
tar -czvf scripts.tar.gz \
    inference_object_detection.py \
    inference_segmentation.py \
    inference_tabular.py \
    train_object_detection.py \
    train_segmentation.py \
    train_tabular.py \
    train_timeseries.py \
    inference_timeseries.py
    
aws s3 cp scripts.tar.gz s3://$BUCKET_NAME/code/scripts.tar.gz
cd ..tar.gz

Step 3: IAM Roles & Permissions

1. Create Trust Policies

Create two JSON files locally for the trust relationships.

echo '{
  "Version": "2012-10-17",
  "Statement": [{"Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]
}' > trust-policy-lambda.json

echo '{
  "Version": "2012-10-17",
  "Statement": [{"Effect": "Allow", "Principal": {"Service": "sagemaker.amazonaws.com"}, "Action": "sts:AssumeRole"}]
}' > trust-policy-sagemaker.json

2. Create Roles and Attach Policies

aws iam create-role --role-name $LAMBDA_ROLE_NAME --assume-role-policy-document file://trust-policy-lambda.json

aws iam attach-role-policy --role-name $LAMBDA_ROLE_NAME --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
aws iam attach-role-policy --role-name $LAMBDA_ROLE_NAME --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
aws iam attach-role-policy --role-name $LAMBDA_ROLE_NAME --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

aws iam create-role --role-name $SAGEMAKER_ROLE_NAME --assume-role-policy-document file://trust-policy-sagemaker.json

aws iam attach-role-policy --role-name $SAGEMAKER_ROLE_NAME --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
aws iam attach-role-policy --role-name $SAGEMAKER_ROLE_NAME --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess

Step 4: Docker & ECR (Container Registry)

We create 3 repositories (one per domain). Each repository will host two image tags: :latest (for training) and :inference (for inference).

CAUTION : If you face storage issues or timeouts building locally, it is highly recommended to use AWS CodeBuild to build and push these images directly within the cloud. Please refer to the AWS CodeBuild Official Documentation for setup instructions.

1. Login to ECR

aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com

2. Create ECR Repositories

aws ecr create-repository --repository-name autogluon-vision --region $AWS_REGION
aws ecr create-repository --repository-name autogluon-tabular --region $AWS_REGION
aws ecr create-repository --repository-name autogluon-timeseries --region $AWS_REGION

3. Build, Tag & Push Images

  • Vision Images
# Build & Push Training (Tagged as :latest)
docker build -t autogluon-vision:latest -f Dockerfile.vision-training .
docker tag autogluon-vision:latest $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-vision:latest
docker push $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-vision:latest

# Build & Push Inference (Tagged as :inference)
docker build -t autogluon-vision:inference -f Dockerfile.vision-inference .
docker tag autogluon-vision:inference $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-vision:inference
docker push $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-vision:inference
  • Tabular Images
docker build -t autogluon-tabular:latest -f Dockerfile.tabular-training .
docker tag autogluon-tabular:latest $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-tabular:latest
docker push $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-tabular:latest

docker build -t autogluon-tabular:inference -f Dockerfile.tabular-inference .
docker tag autogluon-tabular:inference $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-tabular:inference
docker push $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-tabular:inference
  • Time-Series Images
docker build -t autogluon-timeseries:latest -f Dockerfile.timeseries-training .
docker tag autogluon-timeseries:latest $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-timeseries:latest
docker push $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-timeseries:latest

docker build -t autogluon-timeseries:inference -f Dockerfile.timeseries-inference .
docker tag autogluon-timeseries:inference $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-timeseries:inference
docker push $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/autogluon-timeseries:inference

Step 5: Lambda Setup

Deploy the orchestrator function.

1. Package the Code

zip -j lambda_function.zip scripts/lambda_function.py

2. Create the Function

LAMBDA_ROLE_ARN=$(aws iam get-role --role-name $LAMBDA_ROLE_NAME --query 'Role.Arn' --output text)
SAGEMAKER_ROLE_ARN=$(aws iam get-role --role-name $SAGEMAKER_ROLE_NAME --query 'Role.Arn' --output text)

aws lambda create-function \
    --function-name $LAMBDA_FUNC_NAME \
    --runtime python3.10 \
    --role $LAMBDA_ROLE_ARN \
    --handler lambda_function.lambda_handler \
    --timeout 300 \
    --memory-size 128 \
    --zip-file fileb://lambda_function.zip \
    --environment "Variables={SAGEMAKER_ROLE_ARN=$SAGEMAKER_ROLE_ARN,OUTPUT_BUCKET=$BUCKET_NAME}" \
    --region $AWS_REGION

Step 6: Connect S3 Trigger (Filtered to 'data/')

aws lambda add-permission \
    --function-name $LAMBDA_FUNC_NAME \
    --statement-id s3-trigger \
    --action "lambda:InvokeFunction" \
    --principal s3.amazonaws.com \
    --source-arn "arn:aws:s3:::$BUCKET_NAME" \
    --region $AWS_REGION

echo '{
  "LambdaFunctionConfigurations": [
    {
      "LambdaFunctionArn": "'"$(aws lambda get-function --function-name $LAMBDA_FUNC_NAME --query 'Configuration.FunctionArn' --output text)"'",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "data/"
            }
          ]
        }
      }
    }
  ]
}' > s3-notification.json

aws s3api put-bucket-notification-configuration \
    --bucket $BUCKET_NAME \
    --notification-configuration file://s3-notification.json

Step 7: Usage Guide

Upload files to specific paths inside the data/ folder to trigger jobs.

Folder Structure Reference

  • Code: s3://BUCKET/code/

  • Data (Triggers):

    • s3://BUCKET/data/tabular/training/

    • s3://BUCKET/data/tabular/inference/

    • s3://BUCKET/data/timeseries/training/

    • s3://BUCKET/data/vision/detection/training/

    ...etc

  • Outputs (Automatic):

    • s3://BUCKET/models/

    • s3://BUCKET/predictions/

    • s3://BUCKET/metrics/

How to Run This Application

To run this application, follow these steps:

  1. Install Dependencies:

    npm install

  2. Run Development Server:

    npm run dev

  3. Build for Production (Optional):

    npm run build

  4. Start Production Server (Optional):

    npm run start

Working Functionalities

Currently, the following functionalities are working:

  • Tabular
  • Object Detection
  • Image Segmentation
  • Time Series Forecasting
  • Text Classification (AI-Automated)

Functionality Details

Time Series Forecasting

The Time Series forecasting functionality uses AutoGluon's TimeSeriesPredictor to train models that predict future values based on historical time-stamped data.

Data Format

Your CSV or Parquet file should contain:

Column Required Description
timestamp Yes DateTime column (e.g., 2024-01-01 00:00:00)
target Yes The numeric value to forecast
item_id No Identifier for multiple time series (optional)

Example CSV:

timestamp,target,item_id
2024-01-01 00:00:00,100.5,series_A
2024-01-01 01:00:00,102.3,series_A
2024-01-01 02:00:00,98.7,series_A
2024-01-01 00:00:00,50.2,series_B
2024-01-01 01:00:00,51.8,series_B

Configuration Options

Parameter Default Description
Target Column target Column containing values to forecast
Timestamp Column timestamp Column with datetime values
Item ID Column item_id For multiple time series (optional)
Prediction Length 50 Number of future timesteps to predict
Frequency H (Hourly) Data frequency: T (minutely), H (hourly), D (daily), W (weekly), M (monthly)

Training Workflow

  1. Upload your time series CSV/Parquet file
  2. Configure column mappings and forecasting parameters
  3. Train - File is uploaded to S3 → Lambda triggers SageMaker training job
  4. Poll - UI polls /api/timeseries/status every 30 seconds
  5. Complete - Model saved to S3, metrics displayed in UI

Inference Workflow

  1. Enter Training Job ID from your completed training
  2. Upload historical data for forecasting
  3. Process - SageMaker batch transform generates predictions
  4. View Results - Next 50 timesteps displayed with visualization chart

API Endpoints

Endpoint Method Description
/api/upload POST Upload training/inference data (task: timeseries-training or timeseries-inference)
/api/timeseries/status GET Check training job status and fetch metrics
/api/timeseries/download GET Download model (type=model) or metrics (type=metrics)
/api/timeseries/inference-status GET Check inference job status and fetch forecasts
/api/timeseries/inference-download GET Download forecast predictions
/api/timeseries/stop-training POST Stop a running training job

S3 Paths

Type S3 Path
Training Data data/timeseries/training/{timestamp}-{filename}
Inference Data data/timeseries/inference/{timestamp}-{filename}
Models models/ag-timeseries-{jobId}/output/model.tar.gz
Metrics metrics/timeseries/training/ag-timeseries-{jobId}.json
Predictions predictions/timeseries/

Metrics Output

After training, the following metrics are available:

{
  "summary": { "MASE": 0.85, "MAPE": 0.12, ... },
  "leaderboard": [
    { "model": "AutoETS", "score_val": -0.85, "fit_time": 12.5, "MASE": 0.85 },
    { "model": "DeepAR", "score_val": -0.92, "fit_time": 45.2, "MASE": 0.92 }
  ],
  "prediction_length": 50,
  "freq": "H",
  "num_time_series": 3
}

Forecast Output

Predictions include point forecasts and confidence intervals:

timestamp mean lower upper item_id
2024-02-01 00:00:00 105.2 98.5 112.1 series_A
2024-02-01 01:00:00 107.8 100.2 115.4 series_A

Object Detection

For Object detection the dataset structure should be

my_dataset.zip/
├── images/               <-- Folder containing all your .jpg/.png images
│   ├── image_01.jpg
│   ├── image_02.jpg
│   └── ...
└── annotations/
    └── train_coco.json   <-- The Critical File (contains labels & bounding boxes)

A sample .json file would look like this:

{
    "images": [
        {
            "file_name": "../images/image_01.jpg",  <-- Relative path from this JSON file
            "height": 480,
            "width": 640,
            "id": 1
        },
    ],
    "annotations": [
        {
            "image_id": 1,        <-- Matches "id" in "images" list
            "bbox": [100, 100, 50, 80],  <-- [x_min, y_min, width, height]
            "category_id": 1,     <-- Matches "id" in "categories" list
            "id": 1,
            "iscrowd": 0,
            "area": 4000
        }
    ],
    "categories": [
        {
            "id": 1,
            "name": "cat"
        },
        {
            "id": 2,
            "name": "dog"
        }
    ]
}

Image Segmentation

The image segmentation functionality allows you to train a model to identify and outline objects in images.

Training

To train an image segmentation model, you need to upload a .zip file with the following structure:

dataset.zip/
├── images/
│   ├── image_01.jpg
│   ├── image_02.jpg
│   └── ...
└── masks/
    ├── image_01.png
    ├── image_02.png
    └── ...
  • images/: This folder should contain the input images in .jpg format.
  • masks/: This folder should contain the corresponding segmentation masks in .png format.
  • The filenames of the images and masks must match.

When you upload the zip file, it is sent to an S3 bucket under the data/vision/segmentation/training/ prefix. This triggers a Lambda function that starts a SageMaker training job.

Inference

To perform inference, you can upload a single image (.jpg or .png). The uploaded image is sent to the S3 bucket under the data/vision/segmentation/inference/ prefix. This triggers a Lambda function that uses the trained model to generate a segmentation mask for the image.

Text Classification

The text classification functionality uses AI to automatically label unstructured text and provides an interactive AI chatbot to further explore and analyze the results.

Training

1.API Key Validation The user first provides their LLM API key, which is securely validated before any data processing begins.

2.Dataset Upload After successful validation, the user uploads a CSV file containing the text data to be classified.

3.AI-Driven Analysis The AI analyzes the text corpus and automatically extracts semantic labels and relevant keywords.

4.Interactive Exploration Once classification is complete, an AI chatbot becomes available, allowing the user to ask questions and gain deeper insights into the text classification results.

New Features

1. Job ID Display & Copy Button

  • Job ID is shown in a large, easy-to-read format (not buried in logs)

2. Stop Training Button

  • Stop Training button for all task types directly calls AWS SageMaker StopTrainingJob API to terminate running jobs
  • Button appears when training is in progress

3. Inference Pipeline

  • Tabular inference
    • /api/tabular/inference-status - Check tabular inference job status
    • /api/tabular/inference-download - Download tabular predictions
    • /api/vision/inference-status - Check vision inference job status (detection & segmentation)
    • /api/vision/inference-download - Download vision predictions

4. Stop Training APIs

  • /api/tabular/stop-training - Stop tabular SageMaker training jobs
  • /api/vision/stop-training - Stop vision (detection/segmentation) SageMaker training jobs
  • /api/timeseries/stop-training - Stop time series SageMaker training jobs

5. Time Series Pipeline

  • /api/timeseries/status - Check time series training job status
  • /api/timeseries/download - Download time series model or metrics
  • /api/timeseries/inference-status - Check time series inference job status
  • /api/timeseries/inference-download - Download time series forecasts

Technical Details

S3 Paths

Task Training Data Inference Data Predictions
Tabular data/tabular/training/ data/tabular/inference/ predictions/tabular/
Time Series data/timeseries/training/ data/timeseries/inference/ predictions/timeseries/
Object Detection data/vision/detection/training/ data/vision/detection/inference/ predictions/vision-detection/
Segmentation data/vision/segmentation/training/ data/vision/segmentation/inference/ predictions/vision-segmentation/

Lambda Triggers

The Lambda function (scripts/lambda_function.py) routes S3 uploads to SageMaker jobs:

S3 Path Pattern Action Script
data/tabular/training/ Training Job train_tabular.py
data/tabular/inference/ Batch Transform inference_tabular.py
data/timeseries/training/ Training Job train_timeseries.py
data/timeseries/inference/ Batch Transform inference_timeseries.py
data/vision/detection/training/ Training Job train_object_detection.py
data/vision/detection/inference/ Batch Transform inference_object_detection.py
data/vision/segmentation/training/ Training Job train_segmentation.py
data/vision/segmentation/inference/ Batch Transform inference_segmentation.py

Docker Images

Task ECR Image
Tabular autogluon-tabular:latest
Time Series autogluon-timeseries:latest
Vision autogluon-vision:latest

Building & Pushing Docker Images

# Authenticate with ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com

# Build and push Time Series image
cd scripts
docker build -f Dockerfile.timeseries -t autogluon-timeseries .
docker tag autogluon-timeseries:latest <account-id>.dkr.ecr.us-east-1.amazonaws.com/autogluon-timeseries:latest
docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/autogluon-timeseries:latest

Deployment:-

https://autogluon-productionized.web.app

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

1 fork
Report repository

Releases

Packages

Contributors

Languages