Syslog and Cloud Foundry/Bluemix

Question

I've been looking into defining a user service for SysLog. Basically, implement a third party logging service (Papertrail). It's simple to set up.

cf cups papertrail -l syslog://papertail_url:port

It seemed to work ok but I hadn't really looked in detail to see if the log was 'draining' properly. And.. my company would not allow us to store logs with a third party. So... I wrote my own SysLog server and set up a user service pointing to my server.

The log started to flow but pretty quickly, my read on the socket would just block for ages.. then some more data would come in.. but it was missing a lot of data in between. Keep in mind that my Web app that was doing the logging is pretty simple too..and only produces a few message and a stacktrace on a deliberate error.

But the issue is that the log simply does not flow often enough and when it does...it misses a lot of data.

So, I set up PaperTrail again to see if this was my code... or not. But... the same. Even flowing to PaperTrail doesn't work 100%. You get some log messages.. a long wait.. then some more. And many more have been missed.

It seems that the Cloud Foundry Loggregator service isn't quite working correctly here. Has anyone experienced an issue like I have been? This is from BlueMix.

Note: I did read that you will miss messages when many log messages are being created in a short period.. but that is not the case here. I also checked my Bluemix logs to see if there are any messages about SysLog failures.. but I couldn't find any.

Thanks in advance...

Answer 1

Other people have reported the same issue. There have been a number of changes to Loggregator to try and resolve this issue. It's being actively worked on.

Could you open a support ticket with this information and IBM will keep you updated?

Answer 2

This is a known issue that is under investigation by IBM. Unfortunately at the moment there isn't any workaround.

Answer 3

I resolved this by manually logging to papertrail using the modules winston and winston-papertrail

https://github.com/kenperkins/winston-papertrail

Editing to share my code on request

My code implementation was very basic to get this working manually - I will further enhance this. Basically I included both modules winston and winston-papertrail in the dependencies in my package.json and then I created the following logging helper file:

var winston = require('winston');

//
// Requiring `winston-papertrail` will expose
// `winston.transports.Papertrail`
//
require('winston-papertrail').Papertrail;

var winstonPapertrail, logger;

module.exports = {
    init: function (hostname) {
        return init(hostname);
    },
    info: function (logText){
        return log(logText,"info");
    },
    error: function (logText){
        return log(logText,"error");
    },
    log: function(logText){
        return log(logText,"debug")
    }
}

function init(program){
    winstonPapertrail = new winston.transports.Papertrail({
        host: <replacethiswithyourhostname> e.g.:'logsX.papertrailapp.com',
        port: <replacethiswithyourport>,
        program: program,
        handleExceptions: true
    });
    winstonPapertrail.on('error', function(err) {
        // Handle, report, or silently ignore connection errors and failures
    });

    logger = new winston.Logger({
        transports: [winstonPapertrail]
    });

    log(hostname+" starting","info");
}

function log(logText,level){
    if(level=="info"){
        console.info(logText);
        logger.info(logText);
    }else if(level=="error"){
        console.error(logText);
        logger.error(logText);
    }else if(level=="debug"){
        console.log(logText);
        logger.debug(logText);
    }
}

I then save this as a javascript file and import it into my main appand then can both externally and internally log by first using the exposed .init() method and then the .log, .info, .error etc

Not a very elegant solution im sure but its working for the moment (e.g. i can externally log all the messages without losing any)

James.