SSL: CERTIFICATE_VERIFY_FAILED error when curator access elasticsearch

0
votes

Im trying to setup elasticsearch-curator (version 5.6.0) to delete indices in elasticsearch (version 7.3.1).

Theirs versions should be compatible (https://www.elastic.co/guide/en/elasticsearch/client/curator/current/version-compatibility.html).

Elasticseach is protected by SSL using self-signed certificate, so I need to turn off certificate verification.

This is my curator.yml conf:

client:
  hosts:
    - 127.0.0.1
  port: 9201
  url_prefix:
  use_ssl: True
  certificate: /opt/elastic-stack/curator/security/ca.crt
  client_cert:
  client_key:
  ssl_no_validate: True
  http_auth: curator:************
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile: /var/log/elastic-stack/curator/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

When I run

curator --config /opt/elastic-stack/curator/curator.yml  /opt/elastic-stack/curator/actions.yml

Even though ssl_no_validate is set to True, I am getting:

/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py:53: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  return yaml.load(read_file(path))
/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py:175: UserWarning: Connecting to 127.0.0.1 using SSL with verify_certs=False is insecure.
  % host
Traceback (most recent call last):
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 394, in connect
    ssl_context=context,
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
    return context.wrap_socket(sock)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 217, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 376, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 394, in connect
    ssl_context=context,
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
    return context.wrap_socket(sock)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 899, in get_client
    check_version(client)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 685, in check_version
    version_number = get_version(client)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 658, in get_version
    version = client.info()['version']['number']
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 245, in info
    return self.transport.perform_request("GET", "/", params=params)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/transport.py", line 353, in perform_request
    timeout=timeout,
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 226, in perform_request
    raise SSLError("N/A", str(e), e)
elasticsearch.exceptions.SSLError: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/python36/python/opt/rh/rh-python36/root/usr/bin//curator", line 11, in <module>
    sys.exit(cli())
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/cli.py", line 213, in cli
    run(config, action_file, dry_run)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/cli.py", line 160, in run
    client = get_client(**client_args)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 906, in get_client
    'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch.  Error: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777))

I got successful response, when I run

curl -k -u curator:******** https://127.0.0.1:9201

Also kibana and logstash communicates with elasticsearch properly.

Does anyone knows, what might be the problem or how to get more info?

Edit 1:

unfortunately I dont have permissions to use yum, when I installed elasticsearch-curator-5.8.1-1.x86_64.rpm to my home dir using 

cd {{ python_installation_dest }} && rpm2cpio ../elasticsearch-curator-5.8.1-1.x86_64.rpm| cpio -idmB

and then run curator, I got: 

Fatal Python error: initfsencoding: Unable to get the locale encoding Traceback (most recent call last): File "/opt/python/3.7.4/lib/python3.7/encodings/__init__.py", line 31, in <module> zipimport.ZipImportError: can't decompress data; zlib not available

I dont use pythin 3.7.4, how can I change path to python?

2
pythonelasticsearchelastic-stackurllib3elasticsearch-curator
If you're planning on not verifying the SSL certificate, why even provide one in the first place? It seems clear that Curator is trying to verify it because you provided it. Also, recommend upgrading Curator to 5.8.x – untergeek
I see that you also installed via pip, as that is not the RPM version of Curator, which I recommend using, if possible. – untergeek
@untergeek when I left certificate field empty I got the same error – Matúš Bartko
upgrade to 5.8.0 fixed the problem, you can post it as an official solution for other users – Matúš Bartko

2 Answers

1
votes

The error appears to be tied to Python issues within RedHat and associated variants. As the official RPM version of Curator 5.8.x now bundles both its own version of Python 3.7.4 and its own up-to-date OpenSSL shared library (1.1.1c in Curator 5.8.1), you will get better results using the official RPM build.

0
votes

I also faced a similar issue when I was using Curator 5.5.4 for my Elastic Search 5.6. The issue seems to be tied to some python dependency associated with Curator with versions below 5.8.

To fix this, I first installed Curator 5.8 which internally upgraded all the other Python dependencies of Curator. I then uninstalled Curator 5.8 which just removed Curator 5.8 and not its upgraded dependencies.

In the end, I installed Curator 5.5.4. It worked like a charm after that.

Just FYI, it has been working well on production.

pip install --no-cache-dir elasticsearch_curator==5.8 --user
pip uninstall elasticsearch_curator==5.8
pip install --no-cache-dir elasticsearch_curator==5.5.4 --user