Custom Log Template Question

[Clayton Keller]

On 03/03/2011 09:43 AM, Clayton Keller wrote:
> Mark,
>
>>> I have been struggling with handling some pesky encoded characters in
>>> mail logs as of late.
>>>
>>> My issue involves passing the message on to a remote rsyslog server
>>> which then processes the messages into a database. From time to time I
>>> see messages with something similar to the following:
>>>
>>> "Subject:<RE: VIAGRA \256 Official Site ID031831740>"
>>>
>>> In my case, messages are passed to processing mechanisms via triggers
>>> prior to a final insertion but I am getting DB errors based on the
>>> handling of these "invalid" characters.
>>
>> I'm not sure how you are getting an 8-bit character out of the log.
>> Amavisd logging encodes all nonprintable characters as \nnn or \{xxxx},
>> so syslog should not be seeing any. If your syslog daemon decodes that,
>> it must not assume the decoded string will be a valid string in some
>> encoding such as UTF-8, as the sender can put any junk in his subject
>> or display name, and need not play by the rules.
>>
>
> Good points here as well. This is stock syslog on CentOS 5. I will look
> into this area as well.
>
>>> I don't think the use of mime2utf8 will help me in these types of
>>> instances.
>>
>> I think the mime2utf8 comes closest to what can be achieved with
>> decoding of text in Subject and From header fields. Anyway, the
>> mime2utf8 should be able to always produce a valid UTF-8 -encoded text
>> (although not necessarily printable), which is further protected by turning
>> nonprintable characters to \nnn by the log writing code.
>>
>> If logging get further processing, you may prefer [:b64encode[:mime2utf8...
>> instead of [:dquote|[:mime2utf8...  in the log template.
>>
>
> This idea came to my mind last night too. I am going to do some testing
> of live data with it and see what results it provides as well.
>
>>> Also, after looking through previous mailing list messages
>>> I've seen references to somewhat similar types of issues with the use of
>>> setting the LANG= value with the invocation of amavisd-new. I may be not
>>> thinking this through clearly, but I don't think that would help much in
>>> this case since everything is happening after the fact on a completely
>>> different system.
>>
>> That's unrelated. Setting locale to "C" on a mailer is still a sensible
>> thing to do in principle.
>>
>>> Would I be better off in the log template doing some type of find and
>>> replace regex on the fields to help escape these characters? Is that
>>> even possible? Typically I'd do this in the code that is called by the
>>> trigger but I'm not quite getting that far along in the processing yet
>>> to do that. My thinking at this point is to escape any backslashes
>>> before I write the log and then hopefully I can handle the escaped
>>> characters elsewhere.
>>
>> Are you replacing the write_log() with your own code?
>>
>
> Not at this time no.
>
>>> I see the following noted in the README.customize file:
>>>
>>> "If assigning to variables, care must be taken to properly quote certain
>>> special characters (like backslash), as required by Perl quoting rules.
>>> Text read from amavisd file or from external files is not subject
>>> to Perl quoting rules."
>>>
>>> But what is the best practice to do so within the templates and the
>>> available macros? Or is it more raw perl within the template that I need
>>> to consider?
>>
>> No interpretation/decoding of characters in templates is done by
>> amavisd - it uses the text as provided in a variable. How you put it
>> into a variable depends on your config file: assigning a "..." or '...'
>> enclosed text to a variable is subject to Perl's interpretation of
>> qq() and q() or a 'here-document'. Or you may be reading the
>> text from a file, in which no interpretation occurs.
>>
>>> After additional troubleshooting and trial and error of my issue, it is
>>> looking more like I need to take a more simple approach of a find and
>>> replace mentality to the content passed to amavisd-new.
>>>
>>> Looking through README.customize I don't see any predefined macros which
>>> do such a thing thus far, and am considering a custom macro to do just
>>> this.
>>>
>>> I wanted to throw this out to the list to make sure I'm not overlooking
>>> something obvious or already available.
>>
>> I can adapt the macro mime2utf8 or similar if you can explain
>> what is it that it is supposed to do in place of its current function.
>>
>
> I'm not sure that is necessary at this time, but was something I was
> toying around with possibly doing here as well. Again, I am going to
> allow for more time to test on my end of things. Unfortunately, I can't
> quite answer for you what exactly tell you what I am wanting it to do in
> addition just yet. Hopefully I can better answer that as I get more data.
>
> Thanks again for your response. After I do more troubleshooting/testing
> on my side I will report any findings that may be beneficial to the
> discussion.
>
> Clay
>
>

Just a follow-up on this thread. Mark, utilizing mime2utf8 along with 
some better and "earlier" decoding to unicode and better handling 
escaping of backslashes on my end, my testing is showing that I am able 
to handle the logging oddities that we have been seeing much better than 
before and writing to PostgreSQL with far less issues.

Thanks again for your time and ideas.

Clay