How to interpret Fiware CYGNUS stats service output?

Question

Starting from my own installation of the following Fiware components: Orion Context Broker, CYGNUS NGSI, Fiware STH and MongoDB, after a while I got the following result consuming a stats service which I found inside CYGNUS management API.

Service: GET http://<cygnus_host>:<management_port>/v1/stats

Result:

{
  "success":"true",
  "stats":{
     "sources":[
        {
           "name":"http-source",
           "status":"START",
           "setup_time":"2018-05-10T13:35:06.194Z",
           "num_received_events":78,
           "num_processed_events":78
        }
     ],
     "channels":[
        {
           "name":"sth-channel",
           "status":"START",
           "setup_time":"2018-05-10T13:35:06.662Z",
           "num_events":1,
           "num_puts_ok":78,
           "num_puts_failed":0,
           "num_takes_ok":77,
           "num_takes_failed":112
        },
        {
           "name":"mongo-channel",
           "status":"START",
           "setup_time":"2018-05-10T13:35:06.662Z",
           "num_events":0,
           "num_puts_ok":78,
           "num_puts_failed":0,
           "num_takes_ok":78,
           "num_takes_failed":139
        },
        {
           "name":"hdfs-channel",
           "status":"START",
           "setup_time":"2018-05-10T13:35:06.662Z",
           "num_events":1,
           "num_puts_ok":78,
           "num_puts_failed":0,
           "num_takes_ok":77,
           "num_takes_failed":35
        }
     ],
     "sinks":[
        {
           "name":"hdfs-sink",
           "status":"START",
           "setup_time":"2018-05-10T13:35:06.341Z",
           "num_processed_events":77,
           "num_persisted_events":0
        },
        {
           "name":"mongo-sink",
           "status":"START",
           "setup_time":"2018-05-10T13:35:06.374Z",
           "num_processed_events":78,
           "num_persisted_events":78
        },
        {
           "name":"sth-sink",
           "status":"START",
           "setup_time":"2018-05-10T13:35:06.380Z",
           "num_processed_events":78,
           "num_persisted_events":77
        }
     ]
  } 
}

What caught my attention was the amount of num_takes_failed on each channel and here is my first question:

What exactly does this variable mean?

Looking into CYGNUS documentation I suppose that a "take" is something like a retry of a certain action in Flume Mongo channel but which action is that?

I looked at the MongoDB log files and did not find anything related to a connection saturation or similar problem, which brings me to my second question.

Should I worry about this statistic? If yes, how do I solve this problem?

Thank you very much in advance for any help.

anmunoz anmunoz · Accepted Answer · 2018-05-21T10:49:14

You don't have to be worried about the num_takes_failed if you see that the number of processed_events is the same than the number of persisted_events. The numb_takes_filed is the result of the subtraction between the values of the flume methods EventTakeAttemptCount and EventTakeSuccessCount where the EventTakeAttemptCount is the total number of times the sink(s) attempted to read the events from the channel. This doesn't mean that the events were returned each time since sinks might poll and the channel might not have any data, On the other hand,EventTakesuccessCount is the total number of events that were successfully taken by the sink(s).

Moreover, if you want to know more about how the events are processed by the channels and sinks, you can run Cygnus on debug mode and see the log output for ensuring that every event is processed and persisted in the correct way

How to interpret Fiware CYGNUS stats service output?

1 Answers