FIWARE Cygnus: File exists in HDFS, but could not write the data

Question

I am using NGSIHDFSSink to persist on an HDFS through Cygnus. When the first notification from Orion is received, it creates the directory and the correct file with the data:

/user/hdfs/openiot/Lamp_001_Lamp/Lamp_001_Lamp.txt

However, for the next notifications received it shows the following error:

There was some problem with the current endpoint, trying other one. Details: CygnusPersistenceError. /user/hdfs/openioLamp/Lamp_001_Lamp.txt file exists in HDFS, but could not write the data. Server response: 403 Forbidden.

This is my Sink configuration:

cygnus-ngsi.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.NGSIHDFSSink
cygnus-ngsi.sinks.hdfs-sink.channel = hdfs-channel
cygnus-ngsi.sinks.hdfs-sink.enable_encoding = false
cygnus-ngsi.sinks.hdfs-sink.enable_grouping = false
cygnus-ngsi.sinks.hdfs-sink.enable_lowercase = false
cygnus-ngsi.sinks.hdfs-sink.enable_name_mappings = false
cygnus-ngsi.sinks.hdfs-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.hdfs-sink.file_format = json-column
cygnus-ngsi.sinks.hdfs-sink.backend.impl = rest
cygnus-ngsi.sinks.hdfs-sink.backend.max_conns = 500
cygnus-ngsi.sinks.hdfs-sink.backend.max_conns_per_route = 100
cygnus-ngsi.sinks.hdfs-sink.hdfs_host = my_hdfs_ip
cygnus-ngsi.sinks.hdfs-sink.hdfs_port = 50070
cygnus-ngsi.sinks.hdfs-sink.hdfs_username = hdfs
cygnus-ngsi.sinks.hdfs-sink.hdfs_password =
cygnus-ngsi.sinks.hdfs-sink.oauth2_token =
cygnus-ngsi.sinks.hdfs-sink.service_as_namespace = false
cygnus-ngsi.sinks.hdfs-sink.batch_size = 100
cygnus-ngsi.sinks.hdfs-sink.batch_timeout = 30
cygnus-ngsi.sinks.hdfs-sink.batch_ttl = 10
cygnus-ngsi.sinks.hdfs-sink.batch_retry_intervals = 5000
cygnus-ngsi.sinks.hdfs-sink.hive = false
cygnus-ngsi.sinks.hdfs-sink.krb5_auth = false

Any ideas?

Thank you so much

It would be interesting to know how you subscribe and the data you send to Orion, because your question talks about: no Permissions for /user/hdfs/openioLamp/Lamp_001_Lamp.txt and the file /user/hdfs/openiot/Lamp_001_Lamp/Lamp_001_Lamp.txt which are obviously not the same. — jicarretero
@jicarretero I deployed the HDFS docker and the problem was solved. Because a lack of resources, I tried to make it work with a single node HDFS but it seems this doesn't do well.The fact you're pointing out may be an error copy/pasting. I've deleted this test environment already. I'm sorry I can't check this. — yeuk0

fgalan fgalan · Accepted Answer · 2019-03-07T13:17:40

According to HDFS sink documentation, Cygnus uses the WebHDFS to write in HDFS.

I'm not sure but maybe WebHDFS considers "create" and "write" as different permissions, so your WebHDFS instance is configured to allow only the former but not the latter. In that case, it would be an issue with permissions at the WebHDFS server.

I'd suggest to try the WebHDFS itself without Cygnus (i.e. using Postman or curl) to ensure that WebHDFS is working properly.

FIWARE Cygnus: File exists in HDFS, but could not write the data

1 Answers