How to define which time_index attribute is used when notifing Quantumleap?

Question

I am using FIWARE for some time-series data of heat-pumps. I use Orion 2.5.2 and Quantumleap 0.7.6.

My entities have a lot of attributes which are reported in batches. Those data-batches have individual time-stamps for each attribute, so the exact time of a measurement is known (this is also rather important). I use a little python tool to split these batches and send them to the iot-agent seperately via http, using the time-stamp parameter.

I end up with an entity like this:

...

    "attrs": {
        "temp_outdoor": {
            "value": "-6.6",
            "type": "Number",
            "md": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": 1613148707.7509995
                }
            },
            "mdNames": [
                "TimeInstant"
            ],
            "creDate": 1612780352.3855166,
            "modDate": 1613148716.1449544
        },
        "temp_return_flow": {
            "value": "40.8",
            "type": "Number",
            "md": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": 1613149016.394001
                }
            },
            "mdNames": [
                "TimeInstant"
            ],
            "creDate": 1612780352.3855166,
            "modDate": 1613149021.5991328
        },
        "TimeInstant": {
            "value": 1613149101.1790009,
            "type": "DateTime",
            "mdNames": [],
            "creDate": 1612780352.3855166,
            "modDate": 1613149102.5100079
        },

...

I don't really care about the creDate and modDate but about the TimeInstant in "md" of each attribute. Also the bottom "TimeInstant" attribute is just the value of the last Data-Point I think? I would like to use the "md" TimeInstant to create the time_index in CrateDB. Hence, the reported time has to be the custom-metadata one. I tried some different values while subscribing to Quantumleap but can't get it right. Can someone tell me how to specify md->TimeInstant as value for time_index?

I find the documentation to be rather unconclusive on that topic and hope that someone has already solved that mistery and might let me in on it :)

Thanks!

Answer 1

Looking at your payload, it is not clear what's the NGSI model used, which would be the information need to help you. Anyhow, as reported by the documentation:

A fundamental element in the time-series database is the time index. You may be wondering... where is it stored? QuantumLeap will persist the time index for each notification in a special column called time_index.

The value that is used for the time index of a received notification is defined according to the following policy, which choses the first present and valid time value chosen from the following ordered list of options.

1. Custom time index. The value of the Fiware-TimeIndex-Attribute http header. Note that for a notification to contain such header, the corresponding subscription has to be created with an httpCustom block, as detailed in the Subscriptions and Custom Notifications section of the NGSI spec. This is the way you can instruct QL to use custom attributes of the notification payload to be taken as time index indicators.
2. Custom time index metadata. The most recent custom time index (the value of the Fiware-TimeIndex-Attribute) attribute value found in any of the attribute metadata sections in the notification. See the previous option about the details regarding subscriptions.
3. TimeInstant attribute. As specified in the FIWARE IoT agent documentation.
4. TimeInstant metadata. The most recent TimeInstant attribute value found in any of the attribute metadata sections in the notification. (Again, refer to the FIWARE IoT agent documentation.)
5. timestamp attribute.
6. timestamp metadata. The most recent timestamp attribute value found in any of the attribute metadata sections in the notification. As specified in the FIWARE data models documentation.
7. dateModified attribute. If you payed attention in the Orion Subscription section, this is the "dateModified" value notified by Orion. dateModified metadata. The most recent dateModified attribute value found in any of the attribute metadata sections in the notification.

Finally, QL will use the Current Time (the time of notification reception) if none of the above options is present or none of the values found can actually be converted to a datetime.

This means that (and if you understand NGSI model, the documentation is quite clear), with the following payload

{
    "id": "Room1",
    "type": "Room",
    "temperature": {
        "value": 24.2,
        "type": "Number",
        "metadata": {
            "myTime": {
                "type": "DateTime",
                "value": "2020-12-16T17:13:46.00Z"
            }
         }
    },
    "pressure": {
        "value": 720,
        "type": "Number",
        "metadata": {
            "TimeInstant": {
                "type": "DateTime",
                "value": "2020-12-16T17:13:46.00Z"
            }
         }
    },
    "dateObserved": "2021-02-02T00:00:00.00Z",
    "dateCreated": "2019-09-24T12:49:02.00Z",
    "dateModified": "2021-02-02T23:00:50.00Z",
    "TimeInstant": {
          "type": "DateTime",
          "value": "2020-12-16T17:13:46.00Z"
    }
}

If in the notification you set a custom header Fiware-TimeIndex-Attribute=dateObserved, time_index will be the value of dateObserved. If you set Fiware-TimeIndex-Attribute=myTime it will be the myTime attribute metadata linked to temperature. If not Fiware-TimeIndex-Attribute header is passed, the most recent value of metadata attribute TimeInstant will be picked. Suppose to remove the metadata attribute TimeInstant in the payload above, then attribute TimeInstant will be picked. If TimeInstant attribute is removed as well, dateModified value will be picked. In case that attribute is not received as well, current time is used.