How to ensure software package version consistency in AWS SageMaker serverless compute?

0
votes

I am learning AWS SageMaker which is supposed to be a serverless compute environment for Machine Learning. In this type of serverless compute environment, who is supposed to ensure the software package consistency and update the versions?

For example, I ran the demo program that came with SageMaker, deepar_synthetic. In this second cell, it executes the following: !conda install -y s3fs

However, I got the following warning message:

Solving environment: done ==> WARNING: A newer version of conda exists. <== current version: 4.4.10 latest version: 4.5.4 Please update conda by running $ conda update -n base conda

Since it is serverless compute, am I still supposed to update the software packages myself?

Another example is as follows. I wrote a few simple lines to find out the package versions in Jupyter notebook:

import platform

import tensorflow as tf

print(platform.python_version())

print (tf.version)

However, I got the following warning messages:

/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6 return f(*args, **kwds) /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters

The prints still worked and I got the results shown beolow:

3.6.4 1.4.0

I am wondering what I have to do to get the package consistent so that I don't get the warning messages. Thanks.

1
amazon-web-servicesserverlessamazon-sagemaker

1 Answers

0
votes

Today, SageMaker Notebook Instances are managed EC2 instances but users still have full control over the the Notebook Instance as root. You have full capabilities to install missing libraries through the Jupyter terminal.

To access a terminal, open your Notebook Instance to the home page and click the drop-down on the top right: “New” -> “Terminal”. Note: By default, conda installs to the root environment.

The following are instructions you can follow https://conda.io/docs/user-guide/tasks/manage-environments.html on how to install libraries in the particular conda environment.

In general you will need following commands, 

conda env list

which list all of your conda environments 

source activate <conda environment name>

e.g. source activate python3 

conda list | grep <package>

e.g. conda list | grep numpy list what are the current package versions 

pip install numpy

Or 

conda install numpy

Note: Periodically the SageMaker team releases new versions of libraries onto the Notebook Instances. To get the new libraries, you can stop and start your Notebook Instance.

If you have recommendations on libraries you would like to see by default, you can create a forum post under https://forums.aws.amazon.com/forum.jspa?forumID=285 . Alternatively, you can bootstrap your Notebook Instances with Lifecycle Configurations to install custom libraries. More details here: https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateNotebookInstanceLifecycleConfig.html