Helm rabbitmq endpoints does not exist

1
votes

I installed helm chart by using this command:

helm install stable/rabbitmq --name rabbitmq --set rabbitmq.username="admin" --set rabbitmq.password="admin" --set rabbitmq.erlangCookie=secretcookie --set rbacEnabled=true --set service.type=ClusterIP --set ingress.enabled=true --set ingress.hostName="rabbitmq.example.com" --set ingress.tls=true --set ingress.tlsSecret="rabbitmq-tls" --set ingress.annotations."kubernetes.io/ingress.class"="nginx" --set resources.limits.memory="256Mi" --set resources.limits.cpu="100m"

When I tried to open hostname it's giving me 503 (service temporarily unavailable) error.

In kubectl get ep, there is no ip address for rabbitmq service.

In kubectl describe pods/rabbitmq-0 => Readiness probe failed: curl: (22) The requested URL returned error: 401, Liveness probe failed: curl: (22) The requested URL returned error: 401, Readiness probe failed: curl: (7) Failed to connect to 127.0.0.1 port 15672: Connection refused

Please advise, what went wrong here? Service should run on given hostname with https. But, it's not working.

Version of Helm and Kubernetes:

helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.8", GitCommit:"a89f8c11a5f4f132503edbc4918c98518fd504e3", GitTreeState:"clean", BuildDate:"2019-04-23T04:41:47Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Logs

 2019-06-20 14:10:41.580 [info] <0.282.0>
 Starting RabbitMQ 3.7.14 on Erlang 21.3
 Copyright (C) 2007-2019 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/

  ##  ##
  ##  ##      RabbitMQ 3.7.14. Copyright (C) 2007-2019 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
  ######  ##
  ##########  Logs: <stdout>

          Starting broker...
2019-06-20 14:10:41.581 [info] <0.282.0>
 node           : [email protected]
 home dir       : /opt/bitnami/rabbitmq/.rabbitmq
 config file(s) : /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf
 cookie hash    : 2AnXraASrd+rMV02+02Jew==
 log(s)         : <stdout>
 database dir   : /opt/bitnami/rabbitmq/var/lib/rabbitmq/mnesia/[email protected]
2019-06-20 14:10:45.085 [info] <0.282.0> Running boot step pre_boot defined by app rabbit
2019-06-20 14:10:45.085 [info] <0.282.0> Running boot step rabbit_core_metrics defined by app rabbit
2019-06-20 14:10:45.177 [info] <0.282.0> Running boot step rabbit_alarm defined by app rabbit
2019-06-20 14:10:45.198 [info] <0.288.0> Memory high watermark set to 2778 MiB (2913550336 bytes) of 6946 MiB (7283875840 bytes) total
2019-06-20 14:10:45.278 [info] <0.290.0> Enabling free disk space monitoring
2019-06-20 14:10:45.278 [info] <0.290.0> Disk free limit set to 50MB
2019-06-20 14:10:45.293 [info] <0.282.0> Running boot step code_server_cache defined by app rabbit
2019-06-20 14:10:45.293 [info] <0.282.0> Running boot step file_handle_cache defined by app rabbit
2019-06-20 14:10:45.304 [info] <0.293.0> Limiting to approx 65436 file handles (58890 sockets)
2019-06-20 14:10:45.304 [info] <0.294.0> FHC read buffering:  OFF
2019-06-20 14:10:45.304 [info] <0.294.0> FHC write buffering: ON
2019-06-20 14:10:45.305 [info] <0.282.0> Running boot step worker_pool defined by app rabbit
2019-06-20 14:10:45.305 [info] <0.282.0> Running boot step database defined by app rabbit
2019-06-20 14:10:45.378 [info] <0.282.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-06-20 14:10:45.678 [info] <0.282.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2019-06-20 14:10:45.678 [info] <0.282.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping registration.
2019-06-20 14:10:45.678 [info] <0.282.0> Running boot step database_sync defined by app rabbit
2019-06-20 14:10:45.678 [info] <0.282.0> Running boot step codec_correctness_check defined by app rabbit
2019-06-20 14:10:45.678 [info] <0.282.0> Running boot step external_infrastructure defined by app rabbit
2019-06-20 14:10:45.678 [info] <0.282.0> Running boot step rabbit_registry defined by app rabbit
2019-06-20 14:10:45.678 [info] <0.282.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2019-06-20 14:10:45.678 [info] <0.282.0> Running boot step rabbit_queue_location_random defined by app rabbit
2019-06-20 14:10:45.679 [info] <0.282.0> Running boot step rabbit_event defined by app rabbit
2019-06-20 14:10:45.679 [info] <0.282.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2019-06-20 14:10:45.679 [info] <0.282.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2019-06-20 14:10:45.679 [info] <0.282.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2019-06-20 14:10:45.679 [info] <0.282.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2019-06-20 14:10:45.679 [info] <0.282.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2019-06-20 14:10:45.680 [info] <0.282.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2019-06-20 14:10:45.680 [info] <0.282.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2019-06-20 14:10:45.680 [info] <0.282.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2019-06-20 14:10:45.680 [info] <0.282.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2019-06-20 14:10:45.680 [info] <0.282.0> Running boot step rabbit_priority_queue defined by app rabbit
2019-06-20 14:10:45.680 [info] <0.282.0> Priority queues enabled, real BQ is rabbit_variable_queue
2019-06-20 14:10:45.680 [info] <0.282.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2019-06-20 14:10:45.680 [info] <0.282.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2019-06-20 14:10:45.680 [info] <0.282.0> Running boot step kernel_ready defined by app rabbit
2019-06-20 14:10:45.680 [info] <0.282.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2019-06-20 14:10:45.681 [info] <0.282.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2019-06-20 14:10:45.686 [info] <0.282.0> Running boot step guid_generator defined by app rabbit
2019-06-20 14:10:45.698 [info] <0.282.0> Running boot step rabbit_node_monitor defined by app rabbit
2019-06-20 14:10:45.698 [info] <0.318.0> Starting rabbit_node_monitor
2019-06-20 14:10:45.698 [info] <0.282.0> Running boot step delegate_sup defined by app rabbit
2019-06-20 14:10:45.699 [info] <0.282.0> Running boot step rabbit_memory_monitor defined by app rabbit
2019-06-20 14:10:45.699 [info] <0.282.0> Running boot step core_initialized defined by app rabbit
2019-06-20 14:10:45.699 [info] <0.282.0> Running boot step upgrade_queues defined by app rabbit
2019-06-20 14:10:45.984 [info] <0.282.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2019-06-20 14:10:45.984 [info] <0.282.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2019-06-20 14:10:45.984 [info] <0.282.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2019-06-20 14:10:45.985 [info] <0.282.0> Running boot step rabbit_policies defined by app rabbit
2019-06-20 14:10:46.078 [info] <0.282.0> Running boot step rabbit_policy defined by app rabbit
2019-06-20 14:10:46.078 [info] <0.282.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2019-06-20 14:10:46.078 [info] <0.282.0> Running boot step rabbit_vhost_limit defined by app rabbit
2019-06-20 14:10:46.078 [info] <0.282.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2019-06-20 14:10:46.078 [info] <0.282.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2019-06-20 14:10:46.078 [info] <0.282.0> Management plugin: using rates mode 'basic'
2019-06-20 14:10:46.079 [info] <0.282.0> Running boot step recovery defined by app rabbit
2019-06-20 14:10:46.080 [info] <0.349.0> Making sure data directory '/opt/bitnami/rabbitmq/var/lib/rabbitmq/mnesia/[email protected]/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2019-06-20 14:10:46.094 [info] <0.349.0> Starting message stores for vhost '/'
2019-06-20 14:10:46.094 [info] <0.353.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2019-06-20 14:10:46.096 [info] <0.349.0> Started message store of type transient for vhost '/'
2019-06-20 14:10:46.096 [info] <0.356.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2019-06-20 14:10:46.178 [warning] <0.356.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2019-06-20 14:10:46.179 [info] <0.349.0> Started message store of type persistent for vhost '/'
2019-06-20 14:10:46.181 [info] <0.282.0> Running boot step load_definitions defined by app rabbitmq_management
2019-06-20 14:10:46.181 [info] <0.282.0> Running boot step empty_db_check defined by app rabbit
2019-06-20 14:10:46.181 [info] <0.282.0> Running boot step rabbit_looking_glass defined by app rabbit
2019-06-20 14:10:46.181 [info] <0.282.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2019-06-20 14:10:46.181 [info] <0.282.0> Running boot step background_gc defined by app rabbit
2019-06-20 14:10:46.181 [info] <0.282.0> Running boot step connection_tracking defined by app rabbit
2019-06-20 14:10:46.182 [info] <0.282.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local'
2019-06-20 14:10:46.182 [info] <0.282.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local'
2019-06-20 14:10:46.182 [info] <0.282.0> Running boot step routing_ready defined by app rabbit
2019-06-20 14:10:46.182 [info] <0.282.0> Running boot step pre_flight defined by app rabbit
2019-06-20 14:10:46.182 [info] <0.282.0> Running boot step notify_cluster defined by app rabbit
2019-06-20 14:10:46.182 [info] <0.282.0> Running boot step networking defined by app rabbit
2019-06-20 14:10:46.279 [warning] <0.379.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-20 14:10:46.280 [info] <0.393.0> started TCP listener on [::]:5672
2019-06-20 14:10:46.280 [info] <0.282.0> Running boot step direct_client defined by app rabbit
2019-06-20 14:10:46.318 [info] <0.439.0> Peer discovery: enabling node cleanup (will only log warnings). Check interval: 10 seconds.
2019-06-20 14:10:46.584 [info] <0.449.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2019-06-20 14:10:46.584 [info] <0.555.0> Statistics database started.
 completed with 5 plugins.
2019-06-20 14:10:50.779 [info] <0.8.0> Server startup complete; 5 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_peer_discovery_k8s
 * rabbitmq_peer_discovery_common
 * rabbitmq_management_agent
2
kubernetesrabbitmqkubernetes-helm
Can you add output from kubectl logs pods/rabbitmq-0? Also, what version of the chart? – Andy Shinn
@AndyShinn: Chart version is rabbitmq-5.5.0. Logs :updated in question. – Ronak Patel

2 Answers

0
votes

I opened the issue on Github and figured that it was an issue with Persistent Volume Claim.

This solved the error: https://github.com/helm/charts/issues/14893#issuecomment-507967842

kubectl get pvc
kubectl delete pvc <rabbitmq-name>
-1
votes

It seems to be a known github issue, and so far the culprit under the case investigation are invalid credentials used with the readiness probe - HTTP probe ended up with 401 code - unauthorized access.