Training & Deploying SageMaker ML Models using AWS Lambda (NodeJS)

Question

I am using AWS Lambda (NodeJS) for creating a sagemaker training job and deploy it using the Sagemaker Javascript SDK.

I am following the below AWS JavaScript SDK docs

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SageMaker.html

I am using the below script for creating the Training job.

Create Training Job:
=====================

    let TrainingJobName = 'Training-' + curr_date_time
    let TrainingImage   = 'XXXXXX.dkr.ecr.us-east-1.amazonaws.com/xxxx:latest'
    let S3Uri           = 's3://xxx.xxxx.sagemaker/csv'

    console.log(`TrainingJobName: ${TrainingJobName}`);

    let params = {
        AlgorithmSpecification: { /* required */
            TrainingInputMode: 'File', /* required */
            TrainingImage: TrainingImage
        },
        OutputDataConfig: { /* required */
            S3OutputPath: 's3://xxx.xxxx.sagemaker/xxxx/output', /* required */
        },
        ResourceConfig: { /* required */
            InstanceCount: 1, /* required */
            InstanceType: 'ml.m4.xlarge', /* required */
            VolumeSizeInGB: 1, /* required */
        },
        RoleArn: 'arn:aws:iam::xxxxx:role/service-role/AmazonSageMaker-ExecutionRole-xxxx', /* required */
        StoppingCondition: { /* required */
            MaxRuntimeInSeconds: 86400
        },
        TrainingJobName: TrainingJobName, /* required */
        InputDataConfig: [
            {
                ChannelName: 'training', /* required */
                DataSource: { /* required */
                    S3DataSource: {
                        S3DataType: 'S3Prefix', /* required */
                        S3Uri: S3Uri, /* required */
                        S3DataDistributionType: 'FullyReplicated'
                    }
                },
                CompressionType: null,
                ContentType: '',
                RecordWrapperType: null,
            }
        ]
    };

    return await sagemaker.createTrainingJob(params).promise();

After the training job is created, i query the job status using the sagemaker describeTrainingJob function. I get the status as "InProgress"

After that I call the sagemaker waitFor function to wait for the completion of the training job using the below method:

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SageMaker.html#trainingJobCompletedOrStopped-waiter

let waitFor_result = await sagemaker.waitFor('trainingJobCompletedOrStopped', {TrainingJobName: training_job_name}).promise();
console.log(`waitFor_result : ${JSON.stringify(waitFor_result)}`);

I find the sagemaker waitFor creates the second training job before the first training job is completed, and it goes on creating subsequent training jobs with the same job name.

I think this is due to the StoppingCondition parameter (MaxRuntimeInSeconds:86400) in the createTrainingJob function.

I want to know if there is any solution which creates a single training job and return the results after the trainining job is completed ?

========================================================== Update:

I am following the "Scheduling the training of a SageMaker model with a Lambda function" https://www.youtube.com/watch?v=FJaykbAtGTM.

I am able to create a training job if i am using the below code in my lambda function.

let training_job_result = await start_model_training();
console.log(`Sagemaker training result : ${JSON.stringify(training_job_result)}`);

let training_job_arn = training_job_result["TrainingJobArn"];
let training_job_name = training_job_arn.split("/")[1];


let desc_training_job = await sagemaker.describeTrainingJob({TrainingJobName: training_job_name}).promise();
let desc_status = desc_training_job["TrainingJobStatus"];
console.log(`Training job desc_status 1 : ${JSON.stringify(desc_status)}`);

But I need to wait till the training job is completed and invoke the sagemaker deploy method for creating/updating the endpoint.

If I use the below code then it keeps on creating multiple training jobs and the lambda function never terminates.

let waitFor_result = await sagemaker.waitFor('trainingJobCompletedOrStopped', {TrainingJobName: training_job_name}).promise();
console.log(`waitFor_result : ${JSON.stringify(waitFor_result)}`);


desc_training_job = await sagemaker.describeTrainingJob({TrainingJobName: training_job_name}).promise();
desc_status = desc_training_job["TrainingJobStatus"];
console.log(`Training job desc_status 2 : ${JSON.stringify(desc_status)}`);

I want to deploy/update the endpoint once the training is completed.

Julien Simon Julien Simon · Accepted Answer · 2019-07-11T12:00:04

I don't know much about JS, but shouldn't you pass the id of the training job you're waiting for? According to the doc, that's the proper usage:

var params = {
  TrainingJobName: 'STRING_VALUE' /* required */
};
sagemaker.waitFor('trainingJobCompletedOrStopped', params, function(err, data) {
  if (err) console.log(err, err.stack); // an error occurred
  else     console.log(data);           // successful response
});

Training & Deploying SageMaker ML Models using AWS Lambda (NodeJS)

1 Answers