Persist data to MongoDB in columns instead of rows

Question

I'm using mongo-sink and mongo-channel parameters to persist data from Cygnus to MongoDB. This is so cool because it was not permitted before.

However, unlike the others persistence channels (MySQL, CKAN and HDFS), it's not yet allowed to save data in column instead of rows. In other words, the configuration property:

cygnusagent.sinks.mongo-sink.attr_persistence = column

in the agent*.conf file is not allowed to specify this parameter.

Have you planned to extend this features to MongoDB channel too?

Thanks in advance.

Sergio

Answer 1

I'm glad you are using these two new sinks, and you like them. Regarding the "column"-like behaviour of the mongoDB-based sinks, we could add the feature to the OrionMongoSink for sure, since it is a let's say plain data serializer. It is not so clear for me this can be done for OrionSTHSink since the data organization is much more complex. I have to think on it.

Just created: https://github.com/telefonicaid/fiware-cygnus/issues/548

Please, give it a +1 (add a comment in the issue) in order it gains relevance in our roadmap.

Answer 2

in the meantime I looked at the source code. it was quite easy to implement functionality, I had already drafted. Right now it is in the testing phase and it seems to work.

So, if the issue has not yet been assigned to anyone, I would be happy to propose my implementation.

I have to compile and send the file ContributionPolicy.txt to submit a pull request, correct?

Thanks.