JSON logging, to Splunk

[Jernej Porenta]

Dear Patrick,

a while ago, Mark Martinec wrote a script that pulls Redis logs out to 
standard output, which can be easily fed into splunk.

With a little help of a skilled perl programmer, I am totally sure you 
can extend attached script to do whatever you want ;)

cheers, Jernej

On 05/10/14 17:47, Patrick Proniewski wrote:
> Hi,
>
> Not that hard... well, it depends on who you are talking about :)
> I'm afraid I can barely read Perl. So writing a proper plugin, ready for
> production on 3 servers, it kind of a challenge for me. An easier
> approach would be to write a shell/python/ruby/whatever script for
> Splunk to pull data from Redis, and then define a "script data source"
> using this script. But I want to make sure that I've not missed
> anything. May be I can leverage some Amavisd-new functionality or
> setting to get closer from my goal.
>
>
> Joolee <amavisd at joolee.nl> wrote:
>> It wouldn't be that hard to create a plugin for that using the amavis
>> custom hooks api. I'm planning on writing one myself to feed KairosDB
>> with statistical information and log some extra information about a
>> mail to db/file.
>>
>> On 5 October 2014 13:11, Patrick Proniewski
>> <patrick.proniewski at univ-lyon2.fr
>> <mailto:patrick.proniewski at univ-lyon2.fr>> wrote:
>>
>>     Hello,
>>
>>     I've given up on ELK (ElasticSearch/Logstash/Kibana), and I'm
>>     moving to Splunk. Amavisd-new ability to log in JSON format is a
>>     very great feature, and I would like to be able to pipe my JSON
>>     logs to Splunk.
>>
>>     The redis output is still defined, from my past tests with ELK and
>>     I have defined this:
>>
>>     $log_templ = <<'EOD';
>>     [:report_json]
>>     EOD
>>
>>     Unfortunately I've got some problem feeding logs into Splunk:
>>
>>     - Splunk won't pull data from a Redis server. It just does not
>>     have proper connector for that.
>>     - Amavisd-new will not log pure JSON into a file, there's always
>>     regular log lines (start/stop for example) and every mail analysis
>>     log entry is prefixed with "time-stamp hostname binary-path[PID]:
>>     (thread-number)", JSON comes only after all those informations.
>>     Hence, Splunk fails to recognize proper JSON, and won't index the
>>     log file.
>>     - Using Syslog with JSON output is not an option, on FreeBSD
>>     syslogd can't handle lines longer than 1000 Bytes.
>>
>>     Any help is greatly appreciated.
>>
>>     I'm registered to digest, feel free to {B}Cc me.
>>
>>     Patrick PRONIEWSKI
>>     --
>>     Responsable pôle Opérations - DSI - Université Lumière Lyon 2
>>     Responsable Sécurité des Systèmes d'Information
>>
>>
>
>
> Patrick PRONIEWSKI
> --
> Responsable pôle Opérations - DSI - Université Lumière Lyon 2
> Responsable Sécurité des Systèmes d'Information
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: logfeeder-redis2stdout.pl
Type: text/x-perl-script
Size: 8403 bytes
Desc: not available
URL: <http://lists.amavis.org/pipermail/amavis-users/attachments/20141005/02fe78ca/attachment.bin>