Cassandra Datastax OpsCenter “NO DATA” on some graphs

1
votes

I'm running on 1 node test cluster Cassandra Datastax OpsCenter 5.2.0, installed from Amazon Datastax AMI version 2.6.3 with Cassandra Community 2.2.0-1.

OpsCenter doesn't report any errors (All agents connected) yet on some graphs I see NO DATA (while I know that there been a lot of requests):

NO DATA on Write Requests

On some there is nothing:

enter image description here

some are working just fine, like OS: Load, Storage Capacity and OS: Disk Utilization.

What could be the reason for this? How to fix it?

EDIT:

Opscenter logs seems to be fine. In agent.log file, I've found the following errors (dozens of times):

 ERROR [jmx-metrics-2] 2015-09-21 06:50:30,910 Error getting CF metrics
 java.lang.UnsupportedOperationException: nth not supported on this type: PersistentArrayMap
    at clojure.lang.RT.nthFrom(RT.java:857)
    at clojure.lang.RT.nth(RT.java:807)
    at opsagent.rollup$process_metric_map.invoke(rollup.clj:252)
    at opsagent.metrics.jmx$cf_metric_helper.invoke(jmx.clj:96)
    at opsagent.metrics.jmx$start_pool$fn__15320.invoke(jmx.clj:159)
    at clojure.lang.AFn.run(AFn.java:24)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
ERROR [jmx-metrics-4] 2015-09-21 06:50:38,524 Error getting general metrics
 java.lang.UnsupportedOperationException: nth not supported on this type: PersistentHashMap
    at clojure.lang.RT.nthFrom(RT.java:857)
    at clojure.lang.RT.nth(RT.java:807)
    at opsagent.rollup$process_metric_map.invoke(rollup.clj:252)
    at opsagent.metrics.jmx$generic_metric_helper.invoke(jmx.clj:73)
    at opsagent.metrics.jmx$start_pool$fn__15334$fn__15335.invoke(jmx.clj:171)
    at opsagent.metrics.jmx$start_pool$fn__15334.invoke(jmx.clj:170)
    at clojure.lang.AFn.run(AFn.java:24)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

BTW. In Datastax agent configuration file (address.yaml), I have only stomp_interface parameter set to my node IP.

2
cassandradatastaxopscenter
Try this gist.github.com/philip-doctor/2b7c87f551a35a5c7c79 - phact
Unfortunately it isn't this case - I have all columns which this script tries to populate... - piotrwest
Can you show your agent logs and opscenterd log - phact
Your agent is able to retrieve OS counters, so the error seems to be related with the agent not being able to connect to your cassandra node. I think this is likely a configuration issue in your agent's address.yaml. Please post the datastax-agent.log - fishi0x01
I've included errors from agent.log file. I've also found stackoverflow.com/questions/31777084/… . Any information which Datastax Agent versions are affected? Can I update Datastax agents from Opscenter? You can post it as an answer. - piotrwest

2 Answers

1
votes

NO DATA in this case was caused by bug in Datastax agent. I've included related error in my question. The same error is also mentioned here: Opsagent UnsupportedOperationException with PersistentHashMap

After upgrading from Datastax Opscenter 5.2.0 to 5.2.1 and upgrading agent to the same version, problem went away.

Thank you @phact for our help!

0
votes

I had a similar error before and it was due to a wrongly configured address.yaml.

In my case the issue was that I only wrote a simple IP address in the hosts setting and it was fixed by using an array syntax instead. It seemed that without that, the agent was using a wrong data type, which then issued an UnsupportedOperationException. Try to add your local interface IP (or whatever IP your cassandra node is listening on on your machine) to the hosts setting like so in address.yaml:

hosts: ["<local-node-ip>"]

Maybe also try adding the following settings to address.yaml, just to be sure that the correct parameters are used by the agent:

rpc_address: <local-node-ip>

agent_rpc_interface: <local-node-ip>