Fiware-Cygnus configuration to Postgresql, DynamoDB and CKAN

1
votes

I'm new to working with Fiware and have installed contextBroker and Cygnus. What I need to do now is configure to post data to PostgreSQL, DynamoDB and CKAN.

I have seen some exampled on posting to CKAN but haven't been able to try this out as yet.

For the Amazon DynamoDB sink, I'm getting an of error that states the region that I'm using "ap-southeast-2" is invalid. The log states that the valid region should be "ap-shouteast-2". Has anybody else had this issue and know how to address this?

Here is the extract from the log:-

time=2017-07-06T08:44:39.652Z | lvl=DEBUG | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=configure | msg=com.telefonica.iot.cygnus.sinks.NGSIDynamoDBSink[108] : [dynamodb-sink] Invalid configuration (region=ap-southeast-2) -- Must be 'us-east-1', 'us-west-1', 'us-west-2', 'eu-west-1', 'eu-central-1', 'ap-northeast-1', 'ap-northeast1', 'ap-shouteast-1', 'ap-shouteast-2' or 'sa-east-1'

I'm also struggling to find any examples on how to Post to PostgreSQL, including any mappings for creating the table.

Are there any good examples out there on how to post to PostgreSQL, Dynamodb and CKAN?

Any assistance with this would be appreciated.

1
postgresqlamazon-dynamodbfiwareckanfiware-cygnus
Have you tried to replace ap-southeast-2, which should be the correct, by ap-shouteast-2 as stated in the error message? Maybe it was a typo someone made on the code... - Dalton Cézane
Yes I did a test with ap-shouteast-2 and that failed as it should. But I managed to get the DynamoDB part working when I switched to us-east-2. I'm still stuck on how to setup and post to PostgreSQL though. - AndyMH
Did you ser this observation about a problem with PostgreSQL and cygnus: stackoverflow.com/questions/44658851/… ? - Dalton Cézane
Yes I have enable_cache set to true in the agent. My problem is that I don't have any example to work with which would help explain what I'm missing. I have psql running and can connect to the Amazon Postgres End Point to see the database, schema and tables. No data is getting into any tables I create and I'm just guessing how the columns and table names are supposed to be defined. - AndyMH
Did you see this documentation: github.com/telefonicaid/fiware-cygnus/blob/master/doc/… ? - Dalton Cézane

1 Answers

0
votes

Thanks for your post, it has revealed a bug regarding the AWS region names. Southeast regions are coded as APSHOUTEAST1 and APSHOUTEAST2, instead of APSOUTHEAST1 and APSOUTHEAST2 (after removing - and upper casing the configured region). You can see it here:

https://github.com/telefonicaid/fiware-cygnus/blob/master/cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/sinks/NGSIDynamoDBSink.java#L51

The only possible workaround is to use a different region than southeast ones.

With regards to the behaviour of the sinks, usually they are able to work in two different persistence modes, "row" and "column". The former enables Cygnus to create everything, i.e. schemas and tables. The later requires schemas and tables are previously created by the admin. Details can be found here.

Finally, the xXXXX-like encoding you are experiencing is the result of configuring enable_encoding = true. Just set it to false and you will see the underscores in the table names. More details can be seen here.