Telegraf daemon not receiving metrics from app deployed with Dokku

Question

I've already monitored NodeJS apps in the past with Telegraf+InfluxDB+Grafana, but it's the first time I try to monitor a Dokku app, without success so far.

I have the following setup

app server                                          monitoring serveur
- telegraf daemon (listening on udp 8125)   ------> InfluxDB + Grafana
- dokku
  - myapp : sending metrics on localhost:udp:8125

I have the issue that metrics sent from myapp on udp://localhost:8125 are not received by telegraf.

The connectivity from telegraf to influxdb and grafana is ok, because I ran the following command on the app server:

echo "foobar:1|c" | nc -u -w0 127.0.0.1 8125

And I could visualize inside Grafana the foobar counter increment.

So I'm guessing the issue is due to myapp or how I configured Dokku.

In myapp (a node JS app), I'm using the hot-shots package to send metrics (also tried with lynx without success).

This is the code I use to instanciate the statsd client.

var StatsD = require('hot-shots');
var client = new StatsD();
// Attaching an error handler to client's socket
client.socket.on('error', function(error) {
  console.error('StatsD client error in socket: ', error);
});

I increment a counter on app startup to signal a restart. This is done with the following code:

metrics.increment('server_restart', function(err, bytes) {
  if (err) {
    console.log(err);
  } else {
    console.log('StatsD sent : ', bytes);
  }
});

Inside the app server, I ran dokku logs myapp after performing a deployment.

I get the following output

$ dokku logs myapp
2017-05-06T14:51:19.977938845Z app[web.1]:
2017-05-06T14:51:19.977996938Z app[web.1]: > [email protected] start /app
2017-05-06T14:51:19.978002950Z app[web.1]: > node .
2017-05-06T14:51:19.978005766Z app[web.1]:
2017-05-06T14:51:21.849086537Z app[web.1]: StatsD sent :  18
$

So it seems that the metrics payload is properly sent on the right port, but it never reaches telegraf.

Is there any configuration to do in Dokku to authorize external UDP requests to localhost ?
What can I do to debug this further ?

Boynux Boynux · Accepted Answer · 2017-05-14T11:05:23

As the author requested in another thread the solution seems to be:

By default, can a docker container call host's localhost UDP?

Opening ports is only needed when you want to Listen for the requests not sending. By default Docker provides the necessary network namespace for your container to communicate to the host or outside world.

So, you could do it in two ways:

use --net host in your docker run and send requests to localhost:8125 in this case you containerized app is effectively sharing the host's network stack. So localhost points to the daemon that's already running in your host.
talk to the container network gateway (which is usually 172.17.0.1) or your host's hostname from your container. Then your are able to send packets to your daemon in your host.

Telegraf daemon not receiving metrics from app deployed with Dokku

1 Answers