Elasticsearch and JSON

[Mark Martinec]

On 2015-08-03 08:40, Benning, Markus wrote:
> Am 2015-07-26 16:52, schrieb Phil Daws:
>> what would be the most suitable way of generating JSON from
>> amavisd-new for injecting into ES using td-agent ?
> 
> Hello Phil,
> 
> i wrote some words about it on my blog:
> 
> https://markusbenning.de/blog/?p=10
> 
> The logging output is very verbose. I posted a patch to filter
> report_json output last year:
> 
> http://lists.amavis.org/pipermail/amavis-users/2014-December/003371.html
> 
> I dont know td-agent, but i used logstash and now saftpresse to get
> data from log to ES.
> 
> Markus


Don't know about td-agent either, but the most reliable and
efficient way to get JSON logging from amavisd is through
a redis server. Redis is used as a queue, so it can smooth out
any inrush of events, or weather over outages of a log service.

Trying to do the same through a regular text log (syslog or
stderr) is IMO a wrong tool for the job. Decoupling an event
produced from a logger seems beneficial, whereas synchronously
logging can bog down the event producer.

Collecting entries from a redis server is fairly trivial,
so it can be implemented in any language with little work.

Even logstash is perfectly capable of reading JSON entries
from a redis server, and feeding them to Elasticsearch :

amavisd.conf:
   @storage_redis_dsn = ( { server => '[::1]:6379', db_id => 1 } );
   $redis_logging_key = 'logstash-amavis';
   $redis_logging_queue_size_limit = 300000;
     # takes about 250 MB of redis memory per 100000 log entries

logstash:

input {
   redis {
     type => "amavis"
     host => "::1"
     db => 1
     data_type => "list"
     key => "logstash-amavis"
     codec => json {}
   }
}

output {
   elasticsearch {
     host => "127.0.0.1"
     port => "9200"
     protocol => "http"
     template_overwrite => true
     index_type => "%{type}"
     codec => json {}
     idle_flush_time => 5
     flush_size => 1000
   }
}


If logstash is not an option, I have a sample perl program
to pull JSON log entries from a queue on a Redis server, and
write them to stdout (so it can feed a system like Splunk,
which may not have a plugin for reading from a Redis server).
Available on request.

   Mark