Create Amazon SageMaker Hyperparameter Tuning Job with Scikit Learn

Question

I'm wondering how to automatically tune my scikit learn random forest model with Amazon Sagemaker. For now, I would like to tune a single hyperparameter called "max_depth". I'll dump my code first and express some concerns after.

FILE: notebook.ipynb

estimator = sagemaker.estimator.Estimator(image, role,
              train_instance_count=1,
              train_instance_type='ml.m4.xlarge',
              output_path=output_location,
              sagemaker_session=sagemaker_session,
              )

hyperparameter_ranges = {'max_depth': IntegerParameter(20, 30)}
objective_metric_name = 'score'
metric_definitions = [{'Name': 'score', 'Regex': 'score: ([0-9\\.]+)'}]

tuner = HyperparameterTuner(estimator,
                        objective_metric_name,
                        hyperparameter_ranges,
                        metric_definitions,
                        max_jobs=9,
                        max_parallel_jobs=3)
tuner.fit({'train': train_data_location, 'test': test_data_location})

FILE: train (located in docker container)

def train():
    with open(param_path, 'r') as tc:
        hyperparams = json.load(tc)
    print("DEBUG VALUE: ", hyperparams)
    data, class = get_data() #abstraction
    X, y = train_data.drop(['class'], axis=1), train_data['class']
    clf = RandomForestClassifier()
    clf.fit(data, class)
    print("score: " + str(evaluate_model(clf)) + "\n")

I see two issues with this code. First, If I put a json object {'max_value':2} in a file named hyperparameters.json at the necessary path, the print statement outputs {} as if the file is empty.

Issue number 2 is the fact that train() does not allow for hyperparameters to affect the code in any way shape or form. As far as I can tell, amazon has no documentation on the inner workings of the tuner.fit() method. This means I can't figure out how train() accesses the hyperparameters to test.

Any help is appreciated, let me know if I can provide more code or clarify anything.

This is not really answer but an alternative approach. Have you considered to build your own docker container? This willlet you dive one layer deeper in how aws sagemaker handles everthing for you. For me it was a much more transparent approach. Here is an example github.com/awslabs/amazon-sagemaker-examples/tree/master/… — dennis-w
We actually do that. The only reason we're using the notebook is to kick off training jobs via the above code. Is there some way to train without using such a notebook? — user10096127
Yes with boto3 you can start training jobs(and hyperparameter tuning jobs) from everywhere you have access to your aws account via python. In my use case I start my training jobs directly from lambda functions. — dennis-w

Andre Andre · Accepted Answer · 2018-07-20T07:22:20

For issue #1, the service will write the hyperparameters.json for you at /opt/ml/input/config/hyperparameters.json -- you don't need to write this yourself.

For issue #2, tuner.fit(), the fit() function will kick off a SageMaker Tuning job that will run multiple SageMaker Training jobs - each of which will call your train() function. Each training job will get a different set of hyperparameters and so your train() function's responsibility is to simply read the file and use the values therein to parameterize your algorithm. Your algorithm should then emit the objective metric values and model specific to the single set of hyperparameters it received.

The Tuning job will look at the objective metrics from each successfully completed training job and figure out the optimal hyperparameters from the hyperparameter search space to run the next job(s) or to stop tuning and emit the best model/hyperparameters found so far.

We've open-sourced the SageMaker Python SDK, so you can see the code for more details: https://github.com/aws/sagemaker-python-sdk/blob/2fa160c44d92c9eacdb8f79265676eca42832233/src/sagemaker/tuner.py#L225

You can read more about launching Tuning Jobs from the Python SDK here: https://github.com/aws/sagemaker-python-sdk#sagemaker-automatic-model-tuning

Hope this helps!

Create Amazon SageMaker Hyperparameter Tuning Job with Scikit Learn

1 Answers