Orion Context Broker crashes

Question

We have an Orion instance which crashes about once each day or two.

In /var/log/contextBroker/contextBroker-service.out I have found:

log directory: '/var/log/contextBroker'
*** glibc detected *** /usr/bin/contextBroker: corrupted double-linked list: 0x00007f0ed92e3f70 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75f4e)[0x7f0eecdeaf4e]
/lib64/libc.so.6(+0x763d3)[0x7f0eecdeb3d3]
/lib64/libc.so.6(+0x78c88)[0x7f0eecdedc88]
/usr/lib64/libstdc++.so.6(_ZNSsD1Ev+0x39)[0x7f0eed6404c9]
/usr/bin/contextBroker(_Z9jsonParseP14ConnectionInfoPKcRKSsP8JsonNodeP9ParseData+0x539)[0x56fb99]
/usr/bin/contextBroker(_Z9jsonTreatPKcP14ConnectionInfoP9ParseData11RequestTypeRKSsPP11JsonRequest+0x17d)[0x56cf0d]
/usr/bin/contextBroker(_Z12payloadParseP14ConnectionInfoP9ParseDataP11RestServicePP10XmlRequestPP11JsonRequestP18JsonDelayedReleaseRSt6vectorISsSaISsEE+0x3f2)[0x564012]
/usr/bin/contextBroker(_Z11restServiceP14ConnectionInfoP11RestService+0x126c)[0x5654bc]
/usr/bin/contextBroker[0x55cbb6]
/usr/bin/contextBroker[0x55f987]
/usr/lib64/libmicrohttpd.so.10(+0x5599)[0x7f0eee1cf599]
/usr/lib64/libmicrohttpd.so.10(MHD_connection_handle_idle+0x518)[0x7f0eee1d0078]
/usr/lib64/libmicrohttpd.so.10(+0xc3c8)[0x7f0eee1d63c8]
/lib64/libpthread.so.0(+0x7a51)[0x7f0eec957a51]
/lib64/libc.so.6(clone+0x6d)[0x7f0eece5d93d]

And in /var/log/contextBroker/contextBroker-service.out.old the following:

log directory: '/var/log/contextBroker'
*** glibc detected *** /usr/bin/contextBroker: free(): invalid next size (fast): 0x00007fe6d4262110 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75f4e)[0x7fe6e8e9cf4e]
/lib64/libc.so.6(+0x78cf0)[0x7fe6e8e9fcf0]
/usr/bin/contextBroker(_ZN20ContextElementVector7releaseEv+0x2fa)[0x5f4a4a]
/usr/bin/contextBroker(_Z17postUpdateContextP14ConnectionInfoiRSt6vectorISsSaISsEEP9ParseDatab+0x1472)[0x4d6692]
/usr/bin/contextBroker(_Z11restServiceP14ConnectionInfoP11RestService+0x6d6)[0x564926]
/usr/bin/contextBroker[0x55cbb6]
/usr/bin/contextBroker[0x55f987]
/usr/lib64/libmicrohttpd.so.10(+0x5599)[0x7fe6ea281599]
/usr/lib64/libmicrohttpd.so.10(MHD_connection_handle_idle+0x518)[0x7fe6ea282078]
/usr/lib64/libmicrohttpd.so.10(+0xc3c8)[0x7fe6ea2883c8]
/lib64/libpthread.so.0(+0x7a51)[0x7fe6e8a09a51]
/lib64/libc.so.6(clone+0x6d)[0x7fe6e8f0f93d]

Data is sent to the Orion in batches each 5 minutes:

a request with around 500 contextElements
a request with around 10 contextElements
a request with a single contextElements

Orion has only 2 subscriptions (AFAIK) which send the data to Proton-CEP.

The Orion version is:

[centos@orion ~]$ /usr/bin/contextBroker --version
0.25.0 (git version: a8cf800d4e9fdd7b4293a886490c40309a5bb58c)
Copyright 2013-2015 Telefonica Investigacion y Desarrollo, S.A.U

Is there anything we can do to debug the issue?

How much RAM memory and CPU use the system in which CB is running? In additon, is there any other process with CPU/memory footprint running in the same system (e.g. Tomcat, JBoss, some Node.js processes, etc.)? — fgalan
@fgalan 2GB RAM, 1vCPU, 20GB HDD. It's based on an image provided in the FIWARE LAB, so other than Orion, nothing else is running in that VM. — Pedro
That seems to be below the recommended thresholds at fiware-orion.readthedocs.org/en/develop/admin/diagnosis/…. Could you test with 2 vCPU and 4GB RAM just to see how it goes, please? — fgalan

fgalan fgalan · Accepted Answer · 2015-11-30T17:45:56

Taking into account user inputs, Orion seems to be running below the recommended CPU and RAM thresholds (see recomendations). Thus, probably with more resources (e.g. 2 vCPU and 4GM RAM) it run better, specially if MongoDB runs in the same machine that Orion (MongoDB is known to be a memory-intensive process).

Orion Context Broker crashes

1 Answers