Cygnus MySQL sink error (attr_persistence = column)

Question

I'm using cygnus to store data in both MySQL Database and Cosmos. Storing the data in cosmos works ok, but when storing in MySQL, especifically with the attribute "cygnusagent.sinks.mysql-sink.attr_persistence = column" instead of "cygnusagent.sinks.mysql-sink.attr_persistence = row" the data is not stored and I'm getting some errors in flume log.

As the table needs to be created previously when using the column attribute(more info : https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/design/OrionMySQLSink.md#important-notes-regarding-the-persistence-mode), I create the table:

CREATE TABLE def_servpath_sensorreading4_sensorreading(systemid int,value float, sensorid int, nodeid int);

DESCRIBE def_servpath_sensorreading4_sensorreading(systemid int,value float, sensorid int, nodeid int);
mysql> DESCRIBE def_servpath_sensorreading4_sensorreading;
+----------+---------+------+-----+---------+-------+
| Field    | Type    | Null | Key | Default | Extra |
+----------+---------+------+-----+---------+-------+
| systemid | int(11) | YES  |     | NULL    |       |
| value    | float   | YES  |     | NULL    |       |
| sensorid | int(11) | YES  |     | NULL    |       |
| nodeid   | int(11) | YES  |     | NULL    |       |
+----------+---------+------+-----+---------+-------+
4 rows in set (0.00 sec)

This is the flume log:

11 Aug 2015 11:40:31,977 INFO  [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionMySQLSink.persist:240)  - [mysql-sink] Persisting data at OrionMySQLSink. Database: def_serv, Table: def_servpath_sensorreading3_sensorreading, Timestamp: 2015-08-11T08:40:31.969, Data (attrs): {systemid=1, value=29.2, sensorid=2, nodeid=1}, (metadata): {sensorid_md=[], systemid_md=[], value_md=[], nodeid_md=[]}
11 Aug 2015 11:40:31,978 WARN  [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:182)  - Bad context data (Unknown column 'systemid' in 'field list')
11 Aug 2015 11:40:31,978 INFO  [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193)  - Finishing transaction (1439278647-655-0000000005)
11 Aug 2015 11:40:32,369 INFO  [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persist:356)  - [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/sensorreading3_sensorreading/sensorreading3_sensorreading.txt), Data ({"recvTime":"2015-08-11T08:40:31.969Z","nodeid":"1", "nodeid_md":[],"sensorid":"2", "sensorid_md":[],"systemid":"1", "systemid_md":[],"value":"29.2", "value_md":[]})
11 Aug 2015 11:40:32,858 INFO  [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:193)  - Finishing transaction (1439278647-655-0000000005)

Where im getting the following error:

Bad context data (Unknown column 'systemid' in 'field list')

Do I need to create any more columns in the table related with the metadata? Am I creating the table correctly?

Thanks in advance.

frb frb · Accepted Answer · 2015-08-11T09:31:13

Yes, this section of the documentation details that an additional column regarding the metadata must be added per each attribute (by sufixing the attribute's name with _md). Nevertheless, an example is not given (something to be improved, btw), so here it is (tailored to your specific case):

create table def_servpath_sensorreading4_sensorreading(recvTime text, systemid int, systemid_md text, value float, value_md text, sensorid int, sensorid_md text, nodeid int, nodeid_md text);

It is very important the attribute names match the fields in the table. I mean, your entity must have an attribute named systemid; if your attribute would be named system_id then that had to be the field name as well (being the metadata field system_id_md).

Ass you can see, there is an addition column regarding the reception time (recvTime), since Cygnus persists such a value each time a notification is received.

Regarding the table name, you have composed it perfectly: it must be the concatenation of the notified FIWARE service-path, the entity ID and the entity type ('_' as concatenation character). Such a table must exists within a databased named as the notified FIWARE service.

Cygnus MySQL sink error (attr_persistence = column)

1 Answers