How can I improve socket hang up when connecting many devices?

1
votes
  • I am testing to connect many devices to FIWARE in the following environment.

  • Each component is deployed in a container on a physical server.

    +-------------------------------------------------+
|Comet - Cygnus - Orion - IoTAgentJSON - Mosquitto| - device*N
+-------------------------------------------------+

  • Under the condition that each device transmits data at 1 msg/sec, the following error occurs at IoTAgent when the number of devices is 350.(That is, 350 msg/sec)

    {"log":"time=2018-12-16T14:57:24.810Z | lvl=ERROR | corr=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | trans=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | op=IoTAgentNGSI.NGSIService | srv=n/a | subsrv=n/a | msg=Error found executing update action in Context Broker: Error: socket hang up | comp=IoTAgent\n","stream":"stdout","time":"2018-12-16T14:57:24.81037597Z"}
{"log":"time=2018-12-16T14:57:24.810Z | lvl=ERROR | corr=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | trans=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | op=IoTAgentNGSI.Alarms | srv=n/a | subsrv=n/a | msg=Raising [ORION-ALARM]: {\"code\":\"ECONNRESET\"} | comp=IoTAgent\n","stream":"stdout","time":"2018-12-16T14:57:24.810440213Z"}
{"log":"time=2018-12-16T14:57:24.810Z | lvl=ERROR | corr=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | trans=ec11c37f-5194-4cb3-8d79-e04a2d1e745c | op=IoTAgentJSON.MQTTBinding | srv=n/a | subsrv=n/a | msg=MEASURES-002: Couldn't send the updated values to the Context Broker due to an error: Error: socket hang up | comp=IoTAgent\n","stream":"stdout","time":"2018-12-16T14:57:24.810526916Z"}

  • The result of the requested ps ax | grep contextBroker command is as follows.

    ps ax | grep contextBroker
19766 ?        Ssl   29:02 /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -dbhost mongodb-orion-demo -statCounters -statSemWait -statTiming

Question 1: Where is the cause? IoTAgent? Or Orion? Or MongoDB? Or kernel parameter?

  • Error found executing update action in Context Broker: Error: socket hang up but there is no error log displayed in Orion.

Question 2: How can I improve the processing performance of FIWARE?

Question 3: Is there a batch operation on IoT Agent?

1
fiwarefiware-orion
Could you edit your question post to add the parameters that Orion process is using? Typically the result of the ps ax | grep contextBroker command. Thx! – fgalan
I edited the result of the requested command and added it. Thank you. @fgalan – hiro
Please tell me if there are other necessary commands and information. – hiro
When the throughput of your devices is << 350 tps (e.g. 35 tps) the system works correctly? – fgalan
In an hour test, the system will operate correctly until the throughput of the device is 320 tps. The correct operation of the system is judged by whether the message sent by the device is registered in Comet. – hiro

1 Answers

0
votes

It is difficult to provide a right answer, as performance depends on many factors specially in complicated setups involving several components interacting among them. However, I'll try to provide some thoughts and insights based in the information you provide and my previous experience.

With regards to Orion, I'd recommend you to have a look to the documentation on performance tunning. Following the indications in that page you can increase the performance of the component.

However, having said that, I don't think that Orion is the cause of the problem in your case, based on:

  • Even without performance optimization Orion typically reach a throughput in the order of the ~1,000 tps. It should cope updates at 350 tps without problems.
  • Orion is not showing error logs. The error logs you have are produced by IOTAgent component, as far as I understand.

Thus, focusing in IOTA, maybe it would be better to use IOTA-UL instead of IOTA-JSON. The UL encoding is more efficient that JSON encoding so you can gain in efficiency. In addition, IOTA-UL allows you to send multimeasures (using # as separator) which I don't know if fits your case but can be seen as a limited form of batch update (see UL documentation for more detail).

If that doesn't work another posibility is to send data directly to Orion using its NGSIv2 API. That would have several advantages:

  • Simplified setup (two pieces less: MQTT broker and IOTAgent)
  • Under same resource conditions, Orion native performance is typically higher than IOTAgents performance (as I mention before ~1,000 tps or even higher after applying performance optimizations)
  • NGSIv2 API provides a batch update operation (look for POST /v2/op/update in the NGSIv2 specification document cited above)