Store data from an mqtt queue into two influxdb measurements

Question

I set up an influxdata architecture composed of telegraf, influxdb and chronograf to collect data from ambient sensors (temperature, pressure and hygrometry). For now, I am perfectly collecting data and metadata from mqtt to a measurement with infinite data retention.

Now, the users of the project would like that, from a record passing through mqtt, the data are stored infinitely in one measurement and the metadata are stored in a second measurement with a retention of 6 months.

Currently, I have no problem parsing json, decoding base64, converting values into tags and fields, renaming them or storing them in ONE measurement with telegraf. But I'm not very good at duplicating them to two measurements and processing them in parallel.

For example, here is the json frame that I capture from mqtt :

{
    "applicationID":"1",
    "applicationName":"chirp-app",
    "deviceName":"0200000001",
    "devEUI":"60c5a8fffe76ea89",
    "rxInfo":[
        {
            "gatewayID":"60c5a8fffe76154b",
            "uplinkID":"70254019-7f5b-42cd-8cdf-c6f1eaf421c8",
            "name":"rak7249",
            "time":"2020-10-23T14:29:14.260435Z",
            "rssi":-76,
            "loRaSNR":9,
            "location":{
                "latitude":43.50438,
                "longitude":1.52947,
                "altitude":293
            }
        }
    ],
    "txInfo":{
        "frequency":868100000,
        "dr":0
    },
    "adr":true,
    "fCnt":1290,
    "fPort":8,
    "data":"CAIBZwdosAZzJyYCZwCtBAIBtA==",
    "object":{
        "analogInput":{
            "4":4.36,
            "8":3.59
        },
        "barometer":{
            "6":1002.2
        },
        "humiditySensor":{
            "7":88
        },
        "temperatureSensor":{
            "2":17.3
        }
    }
}

Here is how the records created from the mqtt queue are stored in influxdb :

Connected to http://localhost:8086 version 1.8.3
InfluxDB shell version: 1.8.3
> use iot
Using database iot
> select time, device_id, longitude, latitude, altitude, temperature, pressure, humidity from device_0200000001 order by time desc limit 5
name: device_0200000001
time                device_id  longitude latitude altitude temperature pressure humidity
----                ---------  --------- -------- -------- ----------- -------- --------
1603698989487116640 0200000001 1.52989   43.50454 275      11.4        995.5    89
1603698383735575991 0200000001 1.52981   43.50444 267      11.4        995.5    89
1603697777987368971 0200000001 1.52983   43.5044  232      11.4        995.3    88.5
1603697172240129341 0200000001 1.52988   43.50445 235      11.5        995.2    88
1603696566494739058 0200000001 1.52999   43.50457 243      11.6        995      87

What I would like to get:

a series containing the data: device_02000001 (time, device_id, temperature, pressure, humidity)
a series containing data and metadata: device_meta_0200000001 (time, device_id, temperature, pressure, humidity, gatewayID, uplinkID, name, gw_time, rssi, loRaSNR, latitude, longitude, altitude)

Can someone tell me how I can process the data in parallel and then store them in two different measurements?

I've already tried to create two configuration files that would use the same mqtt file, but every time I restart telegraf, it crashes with the error systemd[1]: telegraf.service: Start request repeated too quickly.

Thanks for your help

Thierry

theirman theirman · Accepted Answer · 2020-10-26T15:38:44

The problem was simply a problem of file ownership. The conf file, created using sudo, belonged to the root user

Store data from an mqtt queue into two influxdb measurements

1 Answers