Logging to Splunk with Winston

I have to admit, I’ve still got a soft spot for Splunk in my heart. I spent several years developing apps there and it is still my go-to logging platform. Recently, I’ve been playing with ExpressJS and using Winston as my logger of choice, together with express-winston to hook the two pieces up. My projects inevitably start with this:

var express = require('express'),
    winston = require('winston'),
    expressLogger = require('express-winston');

var app = express();

app.use(expressLogger.logger({
    transports: [
        new winston.transports.Console()
    ],
    level: 'debug'
}));

// Do other setup here
app.listen(process.env.PORT || 3000);

This is all well and good, but what about Splunk? My prior version of this wrote the log to a file and then Splunk would consume the file. However, I’m operating inside of Azure App Service these days and my Splunk instance is operating on a different machine – it’s a virtual machine inside of Azure. So what am I to do?

Splunk recognized this and so they produced a high-performance HTTP event collector. This is a REST endpoint that allows you to submit data as long as you have a token. I’m not going to explain how to get a token (Splunk does a really good job of that). However, I need to handle the other side of things – the Winston transport.

Fortunately, Winston has a highly extensible transport system and I’ve done just that. You can download the module from npmjs.org or get it from my GitHub repository.

So, how do you use it? It doesn’t require Express, but I’m going to alter my prior example to show how easy it is. Note the highlighted lines:

var express = require('express'),
    winston = require('winston'),
    SplunkStreamEvent = require('winston-splunk-httplogger'),
    expressLogger = require('express-winston');

var app = express();

var splunkSettings = {
    host: 'my-splunk-host',
    token: 'MY-DATAINPUT-TOKEN'
};

app.use(expressLogger.logger({
    transports: [
        new winston.transports.Console(),
        new SplunkStreamEvent({ splunk: splunkSettings });
    ],
    level: 'debug'
}));

// Do other setup here
app.listen(process.env.PORT || 3000);

Block-by-block:

  1. Line 3 brings in the library – standard Node module management here
  2. Lines 8-11 define the options for the splunk-logging library
  3. Line 16 adds the transport to winston for logging

It’s as simple as that. There is one proviso. Underneath, it uses the excellent splunk-logging library. Winston expects that you send off each event individually. It doesn’t really stream events. As a result, setting any of the options in such a way that batching occurs will result in weird errors. That’s because Winston is expecting a callback for each event and the splunk-logging library doesn’t call the callback unless it actually writes to the channel. I haven’t done any high capacity tests to see what happens when batching does occur, so I’d avoid that for now.

If you find any bugs or wish to ask a question, please let me know through the GitHub Repository.