JSON logging, to Splunk

[Patrick Proniewski]

Hello,

I've given up on ELK (ElasticSearch/Logstash/Kibana), and I'm moving to Splunk. Amavisd-new ability to log in JSON format is a very great feature, and I would like to be able to pipe my JSON logs to Splunk.

The redis output is still defined, from my past tests with ELK and I have defined this:

$log_templ = <<'EOD';
[:report_json]
EOD

Unfortunately I've got some problem feeding logs into Splunk:

- Splunk won't pull data from a Redis server. It just does not have proper connector for that.
- Amavisd-new will not log pure JSON into a file, there's always regular log lines (start/stop for example) and every mail analysis log entry is prefixed with "time-stamp hostname binary-path[PID]: (thread-number)", JSON comes only after all those informations. Hence, Splunk fails to recognize proper JSON, and won't index the log file.
- Using Syslog with JSON output is not an option, on FreeBSD syslogd can't handle lines longer than 1000 Bytes.

Any help is greatly appreciated.

I'm registered to digest, feel free to {B}Cc me.

Patrick PRONIEWSKI
-- 
Responsable pôle Opérations - DSI - Université Lumière Lyon 2
Responsable Sécurité des Systèmes d'Information