Elasticsearch and JSON

[Alex]

Hi,


> If logstash is not an option, I have a sample perl program
> to pull JSON log entries from a queue on a Redis server, and
> write them to stdout (so it can feed a system like Splunk,
> which may not have a plugin for reading from a Redis server).
> Available on request.

We're currently using the default JSON capabilities in amavisd,
sending it to syslog.

I'm not very familiar with logstash/ES, but I think I'd be interested
in a better way to store and manage amavis log data.

Do you have instructions for setting up a redis server for this? Does
it work with ES, or in place of it?

Are there functions available for producing reports based on the
amavis info stored in ES or redis already?

Thanks,
Alex

On Mon, Feb 15, 2016 at 9:13 PM, Mark Martinec
<Mark.Martinec+amavis at ijs.si> wrote:
> On 2015-08-03 08:40, Benning, Markus wrote:
>>
>> Am 2015-07-26 16:52, schrieb Phil Daws:
>>>
>>> what would be the most suitable way of generating JSON from
>>> amavisd-new for injecting into ES using td-agent ?
>>
>>
>> Hello Phil,
>>
>> i wrote some words about it on my blog:
>>
>> https://markusbenning.de/blog/?p=10
>>
>> The logging output is very verbose. I posted a patch to filter
>> report_json output last year:
>>
>> http://lists.amavis.org/pipermail/amavis-users/2014-December/003371.html
>>
>> I dont know td-agent, but i used logstash and now saftpresse to get
>> data from log to ES.
>>
>> Markus
>
>
>
> Don't know about td-agent either, but the most reliable and
> efficient way to get JSON logging from amavisd is through
> a redis server. Redis is used as a queue, so it can smooth out
> any inrush of events, or weather over outages of a log service.
>
> Trying to do the same through a regular text log (syslog or
> stderr) is IMO a wrong tool for the job. Decoupling an event
> produced from a logger seems beneficial, whereas synchronously
> logging can bog down the event producer.
>
> Collecting entries from a redis server is fairly trivial,
> so it can be implemented in any language with little work.
>
> Even logstash is perfectly capable of reading JSON entries
> from a redis server, and feeding them to Elasticsearch :
>
> amavisd.conf:
>   @storage_redis_dsn = ( { server => '[::1]:6379', db_id => 1 } );
>   $redis_logging_key = 'logstash-amavis';
>   $redis_logging_queue_size_limit = 300000;
>     # takes about 250 MB of redis memory per 100000 log entries
>
> logstash:
>
> input {
>   redis {
>     type => "amavis"
>     host => "::1"
>     db => 1
>     data_type => "list"
>     key => "logstash-amavis"
>     codec => json {}
>   }
> }
>
> output {
>   elasticsearch {
>     host => "127.0.0.1"
>     port => "9200"
>     protocol => "http"
>     template_overwrite => true
>     index_type => "%{type}"
>     codec => json {}
>     idle_flush_time => 5
>     flush_size => 1000
>   }
> }
>
>
> If logstash is not an option, I have a sample perl program
> to pull JSON log entries from a queue on a Redis server, and
> write them to stdout (so it can feed a system like Splunk,
> which may not have a plugin for reading from a Redis server).
> Available on request.
>
>   Mark