OpsCenter 4.1.1 with cassandra 2.0.5 can't add existing cluster : Error creating cluster: Call to /cluster-configs timed out

0
votes

I install a simple 2 nodes cluster with cassandra v2.0.5 in aws ec2(cenos) and install opscenter v4.1.1. The config as follows:

  • Node1

    • public IP: 54.254.26.11
    • private IP: 10.144.138.119
    • Cassandra (cassandra.yaml)
      • broadcast_address: 54.254.26.11
      • listen_address: 10.144.138.119
    • Agent (address.yaml)
      • stomp_interface: 54.254.26.11
      • local_address: 54.254.26.11
      • agent_rpc_interface: 10.144.138.119
      • agent_rpc_broadcast_address: 54.254.26.11
    • OpsCenter (opscenterd.conf )
      • interface: interface = ec2-54-254-26-11.ap-southeast-1.compute.amazonaws.com

  • Node2

    • public IP: 175.41.186.242
    • private IP: 10.130.57.90
    • Cassandra (cassandra.yaml)
      • broadcast_address: 175.41.186.242
      • listen_address: 10.130.57.90
    • Agent (address.yaml)
      • stomp_interface: 54.254.26.11
      • local_address: 175.41.186.242
      • agent_rpc_interface: 10.130.57.90
      • agent_rpc_broadcast_address: 175.41.186.242

When I try to add a existing cluster by the opscenter web, it show

"Error creating cluster: Call to /cluster-configs timed out."

on the web. Then I checked the opscenterd.log as follows, it seems that opscenter can connect both nodes, but a warning:

"ProcessingError while calling CreateClusterConfController: Timeout while adding cluster. Please check the log for details on the problem."

Please help me.

  • 2014-03-18 10:32:24+0800 [JuikerCluster] INFO: Starting CassandraCluster service
  • 2014-03-18 10:32:24+0800 [JuikerCluster] INFO: agent_config items: {'cassandra_log_location': '/var/log/cassandra/system.log', 'thrift_port': 9160, 'thrift_ssl_truststore': None, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'metrics_ignored_solr_cores': '', 'api_port': '61621', 'metrics_enabled': 1, 'thrift_ssl_truststore_type': 'JKS', 'kerberos_use_ticket_cache': True, 'use_ssl': 0, 'kerberos_renew_tgt': True, 'rollups60_ttl': 604800, 'cassandra_install_location': '', 'rollups7200_ttl': 31536000, 'kerberos_debug': False, 'storage_keyspace': 'OpsCenter', 'ec2_metadata_api_host': '169.254.169.254', 'provisioning': 0, 'kerberos_use_keytab': True, 'metrics_ignored_column_families': '', 'thrift_ssl_truststore_password': None, 'metrics_ignored_keyspaces': 'system, system_traces, system_auth, dse_auth, OpsCenter'}
  • 2014-03-18 10:32:24+0800 [] INFO: Stopping factory
  • 2014-03-18 10:32:24+0800 [JuikerCluster] DEBUG: Successfully added connection to to the pool
  • 2014-03-18 10:32:24+0800 [JuikerCluster] DEBUG: Successfully added connection to to the pool
  • 2014-03-18 10:32:25+0800 [JuikerCluster] DEBUG: Successfully added connection to to the pool
  • 2014-03-18 10:32:25+0800 [JuikerCluster] DEBUG: Successfully added connection to to the pool
  • 2014-03-18 10:32:25+0800 [JuikerCluster] DEBUG: Successfully added connection to to the pool
  • 2014-03-18 10:32:25+0800 [] DEBUG: Exception found inspecting dse thrift.
  • 2014-03-18 10:32:25+0800 [JuikerCluster] INFO: Enterprise functionality: False
  • 2014-03-18 10:32:25+0800 [JuikerCluster] DEBUG: Collecting node/token list over Thrift
  • 2014-03-18 10:32:25+0800 [JuikerCluster] INFO: Snitch: org.apache.cassandra.locator.Ec2MultiRegionSnitch
  • 2014-03-18 10:32:25+0800 [JuikerCluster] INFO: Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
  • 2014-03-18 10:32:25+0800 [JuikerCluster] INFO: Cluster Name: JuikerCluster
  • 2014-03-18 10:32:25+0800 [JuikerCluster] INFO: Recognizing new node 175.41.186.242 ('-1586618039612323641')
  • 2014-03-18 10:32:25+0800 [JuikerCluster] INFO: Node 175.41.186.242 has multiple tokens (vnodes). Only one picked for display.
  • 2014-03-18 10:32:25+0800 [JuikerCluster] INFO: Recognizing new node 54.254.26.11 ('-4909356646437291937')
  • 2014-03-18 10:32:25+0800 [JuikerCluster] INFO: Node 54.254.26.11 has multiple tokens (vnodes). Only one picked for display.
  • 2014-03-18 10:34:31+0800 [] WARN: ProcessingError while calling CreateClusterConfController: Timeout while adding cluster. Please check the log for details on the problem.
  • 2014-03-18 10:34:31+0800 [] DEBUG: Average opscenterd CPU usage: 0.92%, memory usage: 39 MB
  • 2014-03-18 10:34:31+0800 [JuikerCluster] DEBUG: Processing node detail from agent for 54.254.26.11
  • 2014-03-18 10:34:31+0800 [JuikerCluster] DEBUG: Found node for 54.254.26.11, data is {u'os-load': 0.14000000000000001} 0.070000000000000007}
  • 2014-03-18 10:34:31+0800 [JuikerCluster] DEBUG: Processing node detail from agent for 54.254.26.11
  • 2014-03-18 10:34:31+0800 [JuikerCluster] DEBUG: Found node for 54.254.26.11, data is {u'os-load': 0.059999999999999998}
  • 2014-03-18 10:34:31+0800 [JuikerCluster] DEBUG: Processing node detail from agent for 54.254.26.11
  • 2014-03-18 10:34:31+0800 [JuikerCluster] DEBUG: Found node for 54.254.26.11, data is {u'os-load': 0.050000000000000003}
  • 2014-03-18 10:34:31+0800 [JuikerCluster] DEBUG: Processing node detail from agent for 54.254.26.11
1
cassandradatastaxopscenter
Do you have any user created keyspaces and tables? – Peter Halliday

1 Answers

0
votes

I had numerous issues configuring OpsCenter to work on the Amazon Linux we deploy to. I got past the issue that is the subject of this question by insuring that my security group for the OpsCenter node allowed inbound traffic (within the VPC only) to port 61620, and that my security group for Cassandra nodes allowed inbound traffic to port 61621. These ports are used to fetch config info from the agents running on the Cassandra nodes.

Other problems I ran into included: 1) configuring both JMX ports (the registry port and the rmi port can both be configured to fixed values).

2) installing sysstat so the iostat command would work

3) updating the installed python and adding several required python libraries

The installation of OpsCenter 4.1.2 on AWS Linux 2014-03 went a lot smoother, but forced a corresponding upgrade to DSE 4.0.2 to match protocols.