JSON logging, to Splunk

[Patrick Proniewski]

Hi,
Not that hard... well, it depends on who you are talking about :)I'm afraid I can barely read Perl. So writing a proper plugin, ready for production on 3 servers, it kind of a challenge for me. An easier approach would be to write a shell/python/ruby/whatever script for Splunk to pull data from Redis, and then define a "script data source" using this script. But I want to make sure that I've not missed anything. May be I can leverage some Amavisd-new functionality or setting to get closer from my goal.

Joolee <amavisd at joolee.nl> wrote:It wouldn't be that hard to create a plugin for that using the amavis custom hooks api. I'm planning on writing one myself to feed KairosDB with statistical information and log some extra information about a mail to db/file.

On 5 October 2014 13:11, Patrick Proniewski <patrick.proniewski at univ-lyon2.fr> wrote:
Hello,

I've given up on ELK (ElasticSearch/Logstash/Kibana), and I'm moving to Splunk. Amavisd-new ability to log in JSON format is a very great feature, and I would like to be able to pipe my JSON logs to Splunk.

The redis output is still defined, from my past tests with ELK and I have defined this:

$log_templ = <<'EOD';
[:report_json]
EOD

Unfortunately I've got some problem feeding logs into Splunk:

- Splunk won't pull data from a Redis server. It just does not have proper connector for that.
- Amavisd-new will not log pure JSON into a file, there's always regular log lines (start/stop for example) and every mail analysis log entry is prefixed with "time-stamp hostname binary-path[PID]: (thread-number)", JSON comes only after all those informations. Hence, Splunk fails to recognize proper JSON, and won't index the log file.
- Using Syslog with JSON output is not an option, on FreeBSD syslogd can't handle lines longer than 1000 Bytes.

Any help is greatly appreciated.

I'm registered to digest, feel free to {B}Cc me.

Patrick PRONIEWSKI
--
Responsable pôle Opérations - DSI - Université Lumière Lyon 2
Responsable Sécurité des Systèmes d'Information

Patrick PRONIEWSKI
-- 
Responsable pôle Opérations - DSI - Université Lumière Lyon 2
Responsable Sécurité des Systèmes d'Information
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.amavis.org/pipermail/amavis-users/attachments/20141005/3e65a715/attachment.html>