JSON logging, to Splunk

[Mark Martinec]

Patrick,

> I've given up on ELK (ElasticSearch/Logstash/Kibana), and I'm moving
> to Splunk. Amavisd-new ability to log in JSON format is a very great
> feature, and I would like to be able to pipe my JSON logs to Splunk.
[...]

Joolee wrote:
> It wouldn't be that hard to create a plugin for that using the amavis
> custom hooks api.

Obtaining a JSON report by a custom hook is possible, although it would
miss some last-minute updates, e.g. on the timing report, as it is not
run late enough in the processing. You'd still need a way to merge
reports from multiple concurrent child processes into a single stream,
which involves locking or some other approach (e.g. message passing).
Also some queuing is desired to decouple feeders from consumers.
Using a file as an intermediate medium to feed Splunk seems like
a poor choice.

2014-10-05 20:17, Jernej Porenta wrote:
> a while ago, Mark Martinec wrote a script that pulls Redis logs out to
> standard output, which can be easily fed into splunk.
> 
> With a little help of a skilled perl programmer, I am totally sure you
> can extend attached script to do whatever you want ;)

Indeed, my little program offers all that: locking and queuing is
handled by Redis, so the consumer process (e.g. Splunk) would be
nicely decoupled from amavisd. Even better would be to persuade
Splunk folks to provide an input module to pull JSON records from
a Redis queue directly.

   Mark