Using cloudwatch api get_metric_data to get sagemaker endpoints invocation metrics

Question

I am trying to use aws cloudwatch api get_metric_data to get sagemaker endpoint invocation metrics in python, and it returns me empty timestamps and Values, but there are some invocations between the time I specified, so there is something going wrong. Below is the code I write in python.

cloudwatch.get_metric_data(
MetricDataQueries=[
    {
        'Id': 'm1',
        'MetricStat': {
            'Metric': {
                'Namespace': 'AWS/SageMaker',
                'MetricName': 'Invocations',
                'Dimensions': [
                    {
                        'Name': 'EndpointName',
                        'Value': 'users-hcl-2',
                    },
                    {
                        'Name': 'VariantName',
                        'Value': 'AllTraffic',
                    },
                ]
            },
            'Period': 3600,
            'Stat': 'Sum',
            'Unit': 'None'
        },
        'ReturnData': True,
    },
],
StartTime=datetime(2019, 2, 1),
EndTime=datetime(2019,2,13),
)

And it returns below:

    {'MetricDataResults': [{'Id': 'm1',
   'Label': 'Invocations',
   'Timestamps': [],
   'Values': [],
   'StatusCode': 'Complete'}],
 'ResponseMetadata': {'RequestId': '8dd847eb-3b43-11e9-b50f-5f6fedb3e07d',
  'HTTPStatusCode': 200,
  'HTTPHeaders': {'x-amzn-requestid': '8dd847eb-3b43-11e9-b50f-5f6fedb3e07d',
   'content-type': 'text/xml',
   'content-length': '494',
   'date': 'Thu, 28 Feb 2019 10:28:13 GMT'},
  'RetryAttempts': 0}}

As I said, the timestamp and values shouldn't be empty, can you help me to sort out where I did wrong, I found some useful links below:

cloudwatch concepts: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html

invocation metrics info: https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch.html

cloudwatch get_metric_data api: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudwatch.html#CloudWatch.Client.get_metric_data

For what I have already tried, changing "Period" entity to different values, but it does no help. Thanks in advance.

Dejan Peretin Dejan Peretin · Accepted Answer · 2019-02-28T12:09:10

Assuming you got all of the dimensions correctly, two things to do first would be:

Try changing the unit to Count or remove the unit completely.
Check that you're calling the correct region. Set it explicitly in your client to the target region, like this for example: cloudwatch = boto3.client('cloudwatch', region_name='TARGET-REGION')

Here are general instructions on how to debug empty data responses: How to retrieve AWS Cloudwatch metrics using AWSSDK.CloudWatch?

Using cloudwatch api get_metric_data to get sagemaker endpoints invocation metrics

2 Answers