Cygnus can not persist data on Cosmos global instance

Question

When trying to persist an entity from Cygnus to Cosmos global instance it fails.

Looking at the log file I see something like that:

2015 15:31:50,006 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:273)  - >> GET /webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&user.name=ms HTTP/1.1
12 Nov 2015 15:31:50,006 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276)  - >> X-Auth-Token: xxxxxxxxxxxxxxxxxxx
12 Nov 2015 15:31:50,006 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276)  - >> Host: cosmos.lab.fiware.org:14000
12 Nov 2015 15:31:50,006 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276)  - >> Connection: Keep-Alive
12 Nov 2015 15:31:50,007 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:276)  - >> User-Agent: Apache-HttpClient/4.2.1 (java 1.5)
12 Nov 2015 15:31:50,284 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.close:169)  - Connection 0.0.0.0:59834<->130.206.80.46:14000 closed
12 Nov 2015 15:31:50,285 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.client.DefaultRequestDirector.tryExecute:716)  - Closing the connection.
12 Nov 2015 15:31:50,285 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.close:169)  - Connection 0.0.0.0:59834<->130.206.80.46:14000 closed
12 Nov 2015 15:31:50,286 INFO  [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.client.DefaultRequestDirector.tryExecute:723)  - I/O exception (java.net.SocketException) caught when processing request: Connection reset
12 Nov 2015 15:31:50,287 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.client.DefaultRequestDirector.tryExecute:728)  - Connection reset
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:185)
    at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)

(I have obfuscated the token in the example above.)

for some reason the connection is closed.

Orion and Cygnus are installed on a fiware lab private instance in Spain2.

When trying to fire the same request from the command line from the Orion machine, I get the same disconnected response.

curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&user.name=ms" -H "X-Auth-Token: xxxxxxxxxxxxxxxxxxxxxx"
curl: (56) Failure when receiving data from the peer

But surprisingly, when doing the same request from the command line from the Cosmos global instance, it works!

See:

[ms@cosmosmaster-gi ~]$ curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/ms/def_serv/def_servpath/6_registervalues/6_registervalues.txt?op=getfilestatus&user.name=ms" -H "X-Auth-Token: xxxxxxxxxxxxxxxxx"
{"FileStatus":{"pathSuffix":"","type":"FILE","length":2,"owner":"ms","group":"ms","permission":"644","accessTime":1447332222908,"modificationTime":1447253745711,"blockSize":67108864,"replication":3}}
[ms@cosmosmaster-gi ~]$

Seems like Cosmos will not service connection arriving from a fiware lab instance running on Spain2.

Does anyone knows what's the issue here, and how to correct this.

Thanks!

Menachem S.

It seems a network issue... Just to have all the test cases: have you tried the curl from another remote machine not in FIWARE Lab, for instance, a laptop of your own? — frb
Thanks! You are correct, it did work from my workstation... so it seems that only from the fiware lab instance, Cosmos is not serving the request, btw, the connection from fiware lab instance to the cosmos global server does seems to work at the network level, and the reason I think it does is because I have tried the same request from fiware instance when using an invalid token and did get a response that authentication is invalid, so did get some response on that situation. question now remains who is in control of the Cosmos global server which can rectify this situation. — M. S.
I'm charge of the Cosmos instance :) Thus, if from your workstation it is properly working then there must be somethig weird with your FIWARE Lab machine... Can you run curl in verbose mode? (-v option) — frb
sure, though I can not post all the data in the comment since the system tells me comment is to long..., any idea how I can transfer the trace to you? — M. S.

M. S. M. S. · Accepted Answer · 2015-11-19T07:57:34

Issue was solved by a workaround.

The Cygnus server was installed on an external node ( a Machine on Azure), and everything works fine wrt Orion-->Cygnus-->Cosmos stream.

Cygnus can not persist data on Cosmos global instance

1 Answers