I have a very simple AWS Lambda function that has been firing every few seconds for 3 months without fail
import boto3
shadow_client = boto3.client('iot-data')
def lambda_handler(event, context):
response = shadow_client.list_named_shadows_for_thing(thingName='XXXXXX')
...
As of 2021-01-26T18:53:05.415+00:00 this has started timing out after 3000ms
I'm also getting notifications of SSL errors suddenly.
I see from the boto3 release history that yesterday (2021-01-26) was when boto3 1.16.60 was released. Maybe unrelated. At the same time there's this post of people reporting SSL errors as of 2021-01-26. They point to AWS maybe switching from one boto3 version to another.
any calls to any boto3.client('iot-data') functions time out after 3000ms:
boto3.client('iot-data').list_named_shadows_for_thing()
boto3.client('iot-data').get_thing_shadow()
Can anyone help me:
- fix it
- avoid changes to boto3 stopping my code running in the future?
Error Logs:
[ERROR] SSLError: SSL validation failed for https://data.iot.eu-central-1.amazonaws.com/things/XXXX/shadow [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091) Traceback (most recent call last): File "/var/task/lambda_function.py", line 37, in lambda_handler client_shadow = get_shadow(clientID)['state']['desired'] File "/var/task/lambda_function.py", line 94, in get_shadow response = iot_data_client.get_thing_shadow(thingName=thing_name) File "/var/runtime/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/var/runtime/botocore/client.py", line 663, in _make_api_call
operation_model, request_dict, request_context) File "/var/runtime/botocore/client.py", line 682, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/var/runtime/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model) File "/var/runtime/botocore/endpoint.py", line 137, in _send_request
success_response, exception): File "/var/runtime/botocore/endpoint.py", line 256, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict) File "/var/runtime/botocore/hooks.py", line 356, in emit return self._emitter.emit(aliased_event_name, **kwargs) File "/var/runtime/botocore/hooks.py", line 228, in emit return self._emit(event_name, kwargs) File "/var/runtime/botocore/hooks.py", line 211, in _emit response = handler(**kwargs) File "/var/runtime/botocore/retryhandler.py", line 183, in call if self._checker(attempts, response, caught_exception): File "/var/runtime/botocore/retryhandler.py", line 251, in call caught_exception) File "/var/runtime/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/var/runtime/botocore/retryhandler.py", line 317, in call
caught_exception) File "/var/runtime/botocore/retryhandler.py", line 223, in call attempt_number, caught_exception) File "/var/runtime/botocore/retryhandler.py", line 359, in _check_caught_exception raise caught_exception File "/var/runtime/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request) File "/var/runtime/botocore/endpoint.py", line 269, in _send return self.http_session.send(request) File "/var/runtime/botocore/httpsession.py", line 281, in send raise SSLError(endpoint_url=request.url, error=e) @timestamp 1611687531606
Actually the timeouts happen silently and are only visible from the logs by searching for timeout
ingestionTime
1611687185615 log 535942143265:/aws/lambda/XXXXXXXXXXX logStream
2021/01/26/[$LATEST]xxxxxxxxxxxxxxxxxxxxx message 2021-01-26T18:53:05.415Z 0b454f65-1366-4525-8288-940d6f667e6c Task timed out after 3.00 seconds requestId
0b454f65-1366-4525-8288-940d6f667e6c timestamp
1611687185415
