Penpals and Envelope From != Header From

[Mark Martinec]

Stef,

> I just registered on this list, so hello to all of you :-)
> I use Amavis and I love the "penpals" feature.
> 
> Penpals works great to catch "bounces", because undeliverable e-mail will
> always be sent to the "Return-Path" / Envelope From. It also works great
> in normal sending/replying (because the "Envelope From" is equal to the
> "Header From").
> 
> However, penpals doesn't work when the "Envelope From" differs from the
> "From" or the "Reply-To" e-mail headers. Currently, Amavis only stores the
> "Envelope From" (as msgs.sid) and the recipients of a message (as
> msgsrcpt.rid). The "Header From"/"Header Reply-To" is not stored. A (good)
> e-mail client should use the Reply-To (if present) or the From when
> hitting the "Reply" button; and Amavis will not find any penpal link
> because the "recipient" of the reply (recipient id/rid of the incoming
> e-mail) isn't equal to the "envelope from" (sid of the original outgoing
> e-mail) and the current SQL query checks if the recipient of the incoming
> reply equals the sender of the original outgoing e-mail. I'm not sure if
> this is intentional, or just "not yet developed".
> 
> I wrote a small patch that adds the check if the sender of the reply is the
> recipient of the outgoing e-mail. I know that this might not be the case
> in case of forwarding, but it's just to help the penpals algorithm a bit
> more. Tested with my local set-up.
> The SQL impact should be limited, as both sid and rid are indexed fields.
> 
> --- /usr/sbin/amavisd.orig	2012-06-04 00:17:50.000000000 +0200
> +++ /usr/sbin/amavisd	2012-06-12 21:43:58.380743445 +0200
> @@ -1324,7 +1324,7 @@
>      'sel_penpals_msgid' =>  # with a nonempty list of message-id references
>        "SELECT msgs.time_num, msgs.mail_id, subject, message_id, rid".
>        " FROM msgs JOIN msgrcpt USING (partition_tag,mail_id)".
> -      " WHERE sid=? AND msgs.content!='V' AND ds='P' AND message_id IN (%m)".
> +      " WHERE (sid=? OR rid=?) AND msgs.content!='V' AND ds='P' AND message_id IN (%m)".
>          " AND rid!=sid".
>        " ORDER BY rid=? DESC, msgs.time_num DESC",  # LIMIT 1
>    );
> @@ -21784,7 +21784,7 @@
>    if (defined($sel_penpals_msgid) && @$message_id_list && defined($sid)) {
>      # list of refs to Message-ID is nonempty, try reference or recipient match
>      my($n) = scalar(@$message_id_list);  # number of keys
> -    my(@args) = ($sid,$rid);  my(@pos_args);  local($1);
> +    my(@args) = ($sid,$rid,$rid);  my(@pos_args);  local($1);
>      my($sel_taint) = substr($sel_penpals_msgid,0,0);   # taintedness
>      $sel_penpals_msgid =~
>             s{ ( %m | \? ) }  # substitute %m for keys and ? for next arg
>  
> A better would probably be to have Amavis store the (calculated)
> From/Reply-To like it stores the envelope-from and the recipients.
> However, I'm not familiar enough with the code for that. I would be
> willing to work on that, if this feature would be appreciated.

The 'OR rid=?' seems too broad, it implies *anyone* from this site
who have mailed to such recipient.

More recent versions of amavisd-new do offer parsed
addresses from certain mail header fields:

sub rfc2822_from #author addresses list (rfc allows one or more), parsed 'From'
sub rfc2822_sender  # sender address (rfc allows none or one), parsed 'Sender'
sub rfc2822_resent_from # resending author addresses list, parsed 'Resent-From'
sub rfc2822_resent_sender  # resending sender addresses, parsed 'Resent-Sender'
sub rfc2822_to  # parsed 'To' header field: a list of recipients
sub rfc2822_cc  # parsed 'Cc' header field: a list of Cc recipients

(Note that there may theoretically be more than one From address)

It would probably be alright to match to envelope-sender OR a From
address.

> However, penpals doesn't work when the "Envelope From" differs from the
> "From" or the "Reply-To" e-mail headers.

Yes, except that if a reply references the original Message-ID,
a match is still found. This covers most correspondence over
mailing lists, as well as direct correspondence.

> Currently, Amavis only stores the "Envelope From" (as msgs.sid)
> and the recipients of a message (as msgsrcpt.rid).
> The "Header From"/"Header Reply-To" is not stored.

Right. Adding an author address (From) would require extending
the SQL schema with another field.

I'm not sure how frequent are cases where matchng a From would help
while matching a referenced Message-ID failed.

> I'm not sure if this is intentional, or just "not yet developed".

At the time when penpals was introduced, addresses of a From field
were not available. Later the parsing was added, which turned out
to be quite tricky to do right. I believe this now works reliably
and according to RFC 5322.

  Mark