Adding DMARC support for Mailman 3

Asked by 1 year ago
Greetings, I am writing on behalf of a group of companies and single persons, who would like to see a limited feature set of the DMARC¹ standard supported by Mailman 3. Since I know we're all eager to get MM3 out as soon as possible and any additional new feature request stands against that I've contacted Barry offlist and asked if he'd agree that the companies involved pay us, sys4², to implement the feature. He did and we also agreed to dedicate a significant part of the payment to mailman's FSF donation account. Before we take out to write code, I would like to ask mailman-developers how it should be done to fit best into Mailman's architecture. Here are the DMARC features that should go into Mailman 3: - don't allow email that comes from a domain with a DMRAC record of p=reject - take ownership of the email and send it with a From: using the domain of the mailing list. (There's a patch for this for Mailman 2.1, which might might be helpful for Mailman 3.) - find the authentication-results header and rewrite it as an Original-Authentication-header: http://tools.ietf.org/html/draft-kucherawy-original-authres-00.html Speaking of an RFC written by Murray Kucherawy. I've contacted Murray in advance and asked him to assist in case we had any questions regarding his RFC(s). He subscribed and ready to help. I hope I was able to bring all parties required together to make a Mailman DMARC implementation come true and I am curious to hear what you have to say. p [ at ] rick ¹) DMARC <http://www.dmarc.org/> adds policies and more to DKIM, which gives digitally signed identity to sender domains and has become a cornerstone to reputation based mail management. Having DMARC support in Mailman 3 is - in my eyes - a reason for postmasters (not end-users) to upgrade their MM 2.x installation as soon as possible. ²) In case you ask yourself, who "sys4" is: We've accompanied Mailman 3 development since Pycon in Chicago (in 2008 ?). Florian Fuchs, whom I don't have to introduce, works with us as well Ralf Hildebrandt and I. We, Ralf and I, are on the python.org postmaster team. Also with sys4 are Antoine Nguyen and Christian Roesser - both are exerienced Python programmers and both have spent significant time developing email applications. http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

Your Answer

Name:
Reply:

All Answers

Answer by 1 year ago
Greetings, Patrick asked me to introduce a bit why DMARC and mailman. In one year DMARC has gained good support (60% of worldwide mailboxes are protected with DMARC http://www.dmarc.org/news/press_release_20130206.html), but like others I'm worried about the long tail. This is the reason some of the people working with DMARC.org have been sponsoring the openDMARC implementation to make it available on a large set of mail servers (cf http://www.trusteddomain.org/opendmarc/ for a list of sponsors). Some openDMARC packages are now available and I expect to see them as part of GNU/Linux distros anytime soon. Similarly, I'm interested to offer the option to list administrators to transition to a behavior that makes the lists safe/working/compatible with DMARC. As Patrick explained, there are about 3 possibilities, while I'm interested more in some than others (I personally experimented with the patch to mailman 2.1), it is only fair to offer the 3 options and let the list administrator choose the one more suitable for his/her needs. Once Patrick has a better understanding on how to best implement these 3 options, it will be easy, like for openDMARC, to sponsor the work to make it as part of mailman3. I know that several DMARC.org members have shown interest to do so. In an other year, with the help of the mailman community, we can progress more in the fight against fake emails. While this may sound like a sales pitch, there has not been so much excitement in email for a long time. Franck Martin https://www.linkedin.com/in/franckmartin From: "Patrick Ben Koetter" p [ at ] sys4.de To: "Mailman Developers" Mailman-Developers [ at ] python.org Sent: Monday, July 1, 2013 3:44:15 PM Subject: [Mailman-Developers] Adding DMARC support for Mailman 3 Greetings, I am writing on behalf of a group of companies and single persons, who would like to see a limited feature set of the DMARC¹ standard supported by Mailman 3. Since I know we're all eager to get MM3 out as soon as possible and any additional new feature request stands against that I've contacted Barry offlist and asked if he'd agree that the companies involved pay us, sys4², to implement the feature. He did and we also agreed to dedicate a significant part of the payment to mailman's FSF donation account. Before we take out to write code, I would like to ask mailman-developers how it should be done to fit best into Mailman's architecture. Here are the DMARC features that should go into Mailman 3: - don't allow email that comes from a domain with a DMRAC record of p=reject - take ownership of the email and send it with a From: using the domain of the mailing list. (There's a patch for this for Mailman 2.1, which might might be helpful for Mailman 3.) - find the authentication-results header and rewrite it as an Original-Authentication-header: http://tools.ietf.org/html/draft-kucherawy-original-authres-00.html Speaking of an RFC written by Murray Kucherawy. I've contacted Murray in advance and asked him to assist in case we had any questions regarding his RFC(s). He subscribed and ready to help. I hope I was able to bring all parties required together to make a Mailman DMARC implementation come true and I am curious to hear what you have to say. p [ at ] rick
Answer by 1 year ago
I'm all for experimentation and being adaptive as new things come along, and I'm obviously supportive of the DMARC effort. That said, I hope that these are going to be configurable options, defaulted "off" for backward compatibility. This is because: (a) the second bullet above is a significant departure from current use (as I understand it), and fails the test of least surprise if we were going to suddenly see that MM3 does things quite differently than previous versions or, indeed, other packages; and (b) I'm uneasy about Original-Authentication-Results. As far as I'm aware there's only a single, proprietary implementation. Its proponents have explained the logic to me several times, but I remain unconvinced. I'm all for experimentation in order to provide data for future efforts, so I don't really object, but this shouldn't be taken as a well-vetted proposal just because there's an (expired) draft about it. -MSK
Answer by 1 year ago
Hey, I've been trying to find the superuser at Gmail, and it turns out I've known him all along! Would you please fix the breakage where copies of one's posts aren't delivered to yourself, which is a real PITA when trying to diagnose Gmail problems?<wink/> (BTW, did you try to sign up as root? or Administrator? ;-) I second your concerns in principle, but I don't think you need to worry about these features being anything but optional. I suspect that a majority of our user don't actually have the fine control over their DNS needed to implement DMARC. Violating the RFC by munging From (for heaven's sake! have we learned nothing from the sad history of Reply-To munging?) is simply not acceptable for most purposes, although perfectly acceptable for advertising. ;-) Steve
Answer by 1 year ago
Quoted message by Stephen J. Turnbull 1 year ago
Hey, I've been trying to find the superuser at Gmail, and it turns out I've known him all along! Would you please fix the breakage where copies of one's posts aren't delivered to yourself, which is a real PITA when trying to diagnose Gmail problems?<wink/> (BTW, did you try to sign up as root? or Administrator? ;-) I second your concerns in principle, but I don't think you need to worry about these features being anything but optional. I suspect that a majority of our user don't actually have the fine control over their DNS needed to implement DMARC. Violating the RFC by munging From (for heaven's sake! have we learned nothing from the sad history of Reply-To munging?) is simply not acceptable for most purposes, although perfectly acceptable for advertising. ;-) Steve
Most users do not display Reply-To; many cannot (at least not at their level of technical skill). This means that they get no indication of the author of a message unless the author signs the body of the mail, which often isn't done, and is impossible to enforce. So this setting is simply unacceptable except on announce/advertising lists (where Reply-To is usually set to some other address anyway) and on anonymous lists (which as far as I know are relatively rare). If this option becomes a popular filter on large mail hosts, discussion lists (ie, the kind of mailing list that Mailman was originally intended to serve) will take a severe, perhaps fatal, blow. Unfortunately, it doesn't. Imposition of authentication designed for personal and other direct mail on aggregate-and-forward services like Mailman is a knife at our throat in principle. Despite obvious goodwill on the part of IETF and others involved in the discussions, I have seen no proposal that works well for discussion lists as currently constituted. And, even more unfortunate, there is ample evidence that the large freemail services (including AOL), not to mention many inexpensive hosting services, consider the possibility that even one spam gets through a knife at *their* throats, while non-delivery of legitimate mail is no problem because it can always be blamed on somebody else.[1] I fear that any halfway plausible plan (even one as flawed as Gmail's handling of "own posts") could get quick and widespread adoption if Mailman offers such features, and operators of discussion lists will be blamed for their poor attitude toward spam-fighting when they resist using them. One might also speculate that the big portal operators are strategicly hostile to independently operated mailing lists, as they would like people to use their forum services instead. But that's purely speculation. None of this is your fault or Murray's, or DMARC's or DKIM's (despite its yahoo roots). It is, however, a reality that *we* face. I'm aware of that. I was ribbing him for his choice of mailbox at Gamil. He obviously thinks it's funny; nobody imposed it on him! Footnotes: [1] For example, my University's Information and Media Center (which intercepts all SMTP connections, and likes to think of itself as in the same class as the MIT Media Lab) labelled Murray's mail "[Suspected Spam]" in the Subject field (which I removed in the interest of remaining friends). I can't even rely on the "[Spam]" label when applied by their incompetent filters. :-(
Answer by 1 year ago
Quoted message by Stephen J. Turnbull 1 year ago
Hey, I've been trying to find the superuser at Gmail, and it turns out I've known him all along! Would you please fix the breakage where copies of one's posts aren't delivered to yourself, which is a real PITA when trying to diagnose Gmail problems?<wink/> (BTW, did you try to sign up as root? or Administrator? ;-) I second your concerns in principle, but I don't think you need to worry about these features being anything but optional. I suspect that a majority of our user don't actually have the fine control over their DNS needed to implement DMARC. Violating the RFC by munging From (for heaven's sake! have we learned nothing from the sad history of Reply-To munging?) is simply not acceptable for most purposes, although perfectly acceptable for advertising. ;-) Steve
I don't think the idea of telling people to include or go look for a particular substring in the SMTP reply text will ultimately work in a standards document, which relegates this logic to the realm of heuristics. We've already seen resistance to that effect on the IETF lists. We'd be better off trying to register some enhanced status codes and asking the community to begin using those. Well, one (or two) parties have experience with OAR. It would be nice if this was broader, but that there is not after all this time is something I take to mean it's not a pain point others are feeling. It might work great for MM3, to be sure, but they'd effectively be the broad-scale pioneers here. -MSK
Answer by 1 year ago
Quoted message by Stephen J. Turnbull 1 year ago
Hey, I've been trying to find the superuser at Gmail, and it turns out I've known him all along! Would you please fix the breakage where copies of one's posts aren't delivered to yourself, which is a real PITA when trying to diagnose Gmail problems?<wink/> (BTW, did you try to sign up as root? or Administrator? ;-) I second your concerns in principle, but I don't think you need to worry about these features being anything but optional. I suspect that a majority of our user don't actually have the fine control over their DNS needed to implement DMARC. Violating the RFC by munging From (for heaven's sake! have we learned nothing from the sad history of Reply-To munging?) is simply not acceptable for most purposes, although perfectly acceptable for advertising. ;-) Steve
I'm amazed I was able to get it. It's resulted in a lot of kudos, but also some very interesting spam. -MSK
Answer by 1 year ago
Quoted message by Stephen J. Turnbull 1 year ago
Most users do not display Reply-To; many cannot (at least not at their level of technical skill). This means that they get no indication of the author of a message unless the author signs the body of the mail, which often isn't done, and is impossible to enforce. So this setting is simply unacceptable except on announce/advertising lists (where Reply-To is usually set to some other address anyway) and on anonymous lists (which as far as I know are relatively rare). If this option becomes a popular filter on large mail hosts, discussion lists (ie, the kind of mailing list that Mailman was originally intended to serve) will take a severe, perhaps fatal, blow. Unfortunately, it doesn't. Imposition of authentication designed for personal and other direct mail on aggregate-and-forward services like Mailman is a knife at our throat in principle. Despite obvious goodwill on the part of IETF and others involved in the discussions, I have seen no proposal that works well for discussion lists as currently constituted. And, even more unfortunate, there is ample evidence that the large freemail services (including AOL), not to mention many inexpensive hosting services, consider the possibility that even one spam gets through a knife at *their* throats, while non-delivery of legitimate mail is no problem because it can always be blamed on somebody else.[1] I fear that any halfway plausible plan (even one as flawed as Gmail's handling of "own posts") could get quick and widespread adoption if Mailman offers such features, and operators of discussion lists will be blamed for their poor attitude toward spam-fighting when they resist using them. One might also speculate that the big portal operators are strategicly hostile to independently operated mailing lists, as they would like people to use their forum services instead. But that's purely speculation. None of this is your fault or Murray's, or DMARC's or DKIM's (despite its yahoo roots). It is, however, a reality that *we* face. I'm aware of that. I was ribbing him for his choice of mailbox at Gamil. He obviously thinks it's funny; nobody imposed it on him! Footnotes: [1] For example, my University's Information and Media Center (which intercepts all SMTP connections, and likes to think of itself as in the same class as the MIT Media Lab) labelled Murray's mail "[Suspected Spam]" in the Subject field (which I removed in the interest of remaining friends). I can't even rely on the "[Spam]" label when applied by their incompetent filters. :-(
This is speculation, and broad fear of the future... The current practice for a postmaster is to trust (or not) emails from specific mailing lists, not who post them to the list. Adding DKIM to the list and taking ownership will only improve it. You should really read the code of the patch for MM2 and try it.
Answer by 1 year ago
Quoted message by Murray S. Kucherawy 1 year ago
I don't think the idea of telling people to include or go look for a particular substring in the SMTP reply text will ultimately work in a standards document, which relegates this logic to the realm of heuristics. We've already seen resistance to that effect on the IETF lists. We'd be better off trying to register some enhanced status codes and asking the community to begin using those. Well, one (or two) parties have experience with OAR. It would be nice if this was broader, but that there is not after all this time is something I take to mean it's not a pain point others are feeling. It might work great for MM3, to be sure, but they'd effectively be the broad-scale pioneers here. -MSK
Mailman 2 read already the substring, and it is common practices amongst ESP to do that to better classify bounces, at least between soft and hard bounces ( http://www.boogietools.com/Products/Windows/BounceStudioEnterprise/Email-Bounce-Categories.asp ) I suspect this process has been improved in MM3, but I have not had a look at the code. The purpose is for mailman to recognize 5.7.x bounces and not count them against the recipient (or as soft bounce). Hard bounce is the default for any non recognized bounce. Finally, yes, DMARC should register X.7.17 on http://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xml
Answer by 1 year ago
Quoted message by Murray S. Kucherawy 1 year ago
I don't think the idea of telling people to include or go look for a particular substring in the SMTP reply text will ultimately work in a standards document, which relegates this logic to the realm of heuristics. We've already seen resistance to that effect on the IETF lists. We'd be better off trying to register some enhanced status codes and asking the community to begin using those. Well, one (or two) parties have experience with OAR. It would be nice if this was broader, but that there is not after all this time is something I take to mean it's not a pain point others are feeling. It might work great for MM3, to be sure, but they'd effectively be the broad-scale pioneers here. -MSK
MM3 uses flufl.bounce <https://launchpad.net/flufl.bounce> for bounce processing. The process, algorithms and heuristics are essentially identical to MM2.1 In the case of an RFC 3464 compliant DSN, MM does not look at the Status field at all. It only looks at the Action field. Actions beginning with 'fail' result in a 'permanent failure' determination. Actions beginning with 'delayed' result in a 'temporary failure' determination. All other actions result in the bounce being ignored.
Answer by 1 year ago
Quoted message by Franck Martin 1 year ago
Mailman 2 read already the substring, and it is common practices amongst ESP to do that to better classify bounces, at least between soft and hard bounces ( http://www.boogietools.com/Products/Windows/BounceStudioEnterprise/Email-Bounce-Categories.asp ) I suspect this process has been improved in MM3, but I have not had a look at the code. The purpose is for mailman to recognize 5.7.x bounces and not count them against the recipient (or as soft bounce). Hard bounce is the default for any non recognized bounce. Finally, yes, DMARC should register X.7.17 on http://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xml
I see in MM2 that regex are used to classify the bounces: http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/files/head:/Mailman/Bouncers/ which is described in broad terms here: http://www.esosoft.com/support/mailinglist/mailman/bounce.html It seems flufl has the same structure http://bazaar.launchpad.net/~mailman-coders/flufl.bounce/trunk/files/head:/flufl/bounce/_detectors/ but I just read the code quickly...
Answer by 1 year ago
Quoted message by Franck Martin 1 year ago
Mailman 2 read already the substring, and it is common practices amongst ESP to do that to better classify bounces, at least between soft and hard bounces ( http://www.boogietools.com/Products/Windows/BounceStudioEnterprise/Email-Bounce-Categories.asp ) I suspect this process has been improved in MM3, but I have not had a look at the code. The purpose is for mailman to recognize 5.7.x bounces and not count them against the recipient (or as soft bounce). Hard bounce is the default for any non recognized bounce. Finally, yes, DMARC should register X.7.17 on http://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xml
Yes and no. Bounce processing in the non-VERP case works by processing the DSN through a series of heuristic recognizers. A recognizer returns a possibly empty list of bouncing addresses or a special STOP signal. If the STOP signal is returned, it means "I recognize this, but ignore it" and processing of this bounce stops. If a non-empty list is returned, bounces are scored for those addresses and processing stops. If an empty list is returned, processing continues to the next recognizer. Several of the heuristic recognizers use regexps in the recognition process, but ideally, all DSNs would be RFC 3464 compliant and the recognizer for these looks only at the Action field to determine what to do. Of course in the real world, not all DSNs are RFC 3464 compliant, so we have a bunch of other recognizers. This is different in the VERP case in that the bouncing address is always obtained from the envelope recipient and the recognozers are only called to determine if the bounce is fatal or not. MM3 is different in that its recognizers return two sets of addresses, one for permanent and one for temporary failures. And which is not strictly correct for MM2.1 at least because in MM2.1, contrary to the documentation, there are no 'soft bounces'. Either delivery failed, even for a full mailbox, in which case we score a 'hard bounce', or it didn't (yet) in which case we score nothing. In essence, the structure and process are the same, but it does add the temporary failure notion. It would certainly be possible to add recognition of a DMARC policy rejection and not report it as a bounce for a recipient (or fix the bugs one by one as they are reported), although I suspect it would be some while before the dust settles and most all the non-compliant MTAs bounces are properly classified.
Answer by 1 year ago
Quoted message by Stephen J. Turnbull 1 year ago
Most users do not display Reply-To; many cannot (at least not at their level of technical skill). This means that they get no indication of the author of a message unless the author signs the body of the mail, which often isn't done, and is impossible to enforce. So this setting is simply unacceptable except on announce/advertising lists (where Reply-To is usually set to some other address anyway) and on anonymous lists (which as far as I know are relatively rare). If this option becomes a popular filter on large mail hosts, discussion lists (ie, the kind of mailing list that Mailman was originally intended to serve) will take a severe, perhaps fatal, blow. Unfortunately, it doesn't. Imposition of authentication designed for personal and other direct mail on aggregate-and-forward services like Mailman is a knife at our throat in principle. Despite obvious goodwill on the part of IETF and others involved in the discussions, I have seen no proposal that works well for discussion lists as currently constituted. And, even more unfortunate, there is ample evidence that the large freemail services (including AOL), not to mention many inexpensive hosting services, consider the possibility that even one spam gets through a knife at *their* throats, while non-delivery of legitimate mail is no problem because it can always be blamed on somebody else.[1] I fear that any halfway plausible plan (even one as flawed as Gmail's handling of "own posts") could get quick and widespread adoption if Mailman offers such features, and operators of discussion lists will be blamed for their poor attitude toward spam-fighting when they resist using them. One might also speculate that the big portal operators are strategicly hostile to independently operated mailing lists, as they would like people to use their forum services instead. But that's purely speculation. None of this is your fault or Murray's, or DMARC's or DKIM's (despite its yahoo roots). It is, however, a reality that *we* face. I'm aware of that. I was ribbing him for his choice of mailbox at Gamil. He obviously thinks it's funny; nobody imposed it on him! Footnotes: [1] For example, my University's Information and Media Center (which intercepts all SMTP connections, and likes to think of itself as in the same class as the MIT Media Lab) labelled Murray's mail "[Suspected Spam]" in the Subject field (which I removed in the interest of remaining friends). I can't even rely on the "[Spam]" label when applied by their incompetent filters. :-(
I said so myself. The fact that I'm paranoid just means I've read Bellovin and Cheswick. No, it's an extrapolation of my own occasional experience, and the frequent pain of others that I see on this list, and the observed behavior of sysadmins, some of whom I respect but who are under extreme pressure to stop spam, and others who are less competent. It's a very specific fear of a broken standard that may be imposed on us by powerful third parties. It's possible that mailing lists as we know them today are an anachronism, that they themselves are fundamentally broken in a world where spam constitutes 90% of all email traffic, and we should let them go rest in peace. I can accept that. There may be no solution that allows both existing mailing list customs to continue and provides socially acceptable levels of spam prevention. If so, so be it. I don't need to when you suggest violating basic RFCs to make DMARC work better. Sometimes it's appropriate to "take ownership of From". DMARC is not a valid reason to do so, and I'm not going to try that. And what good does trying a patch do? I fear a *social* problem, not that your patch will make Mailman technically unable to receive or send mail. If the latter happens, we debug the patch. Minor irritation, no more than that.
Answer by 1 year ago
Quoted message by Franck Martin 1 year ago
This is speculation, and broad fear of the future... The current practice for a postmaster is to trust (or not) emails from specific mailing lists, not who post them to the list. Adding DKIM to the list and taking ownership will only improve it. You should really read the code of the patch for MM2 and try it.
Hi Stephen, There is a case where the mailing list administrator configured the list to take ownership of the "From". Telling people that it was not a good idea never works. It's easier to wait for the denial of service (which happened) and watch the complaints to pour in. Regards, -sm
Answer by 1 year ago
In this message, I'll talk about implementation strategies. In very general terms, what I think you're looking at is writing an extension/plugin that provides additional rules and handlers, and then modifying or extending the built-in processing chains and pipelines to consult those new rules and handlers. In this way, DMARC support (or some subset of it) can start out as an unofficial extension, which then based on experience may eventually make it into the standard distribution. I've spoken many times about the architecture for moderation (rules and chains) and modification (pipeline of handlers), so I won't go into more detail here. Please start with my Pycon 2012 talk and feel free to ask any questions you may still have. In some sense there is overlap with how I would integrate any other anti-spam tool in MM3. Most of us agree that such things are better done in the MTA, but it's just not always possible to do that. List admins have more control over how their lists are configured than over their MTA, so I do see value in providing these services in MM3, as plugins. For #1 you would have a rule that can answer the question of DMARC disposition. Rules output binary results, and if this rule hits, it would run an action, probably to discard the message, although it could also hold it or reject/bounce it. I doubt you'd want to accept the message. If the rule misses then processing continues along as normal. Munging the From header is done by a processing handler, after the message has already been accepted for posting, and is being prepared for delivery to the list membership. Similarly, the Authentication-Results rewrite would be done in the same or a different handler. With respect to both headers, I'm assuming that the munging is the same for every recipient; there's a different hook point for personalization of outgoing messages. -Barry
Answer by 1 year ago
Current MM3 doesn't have a great way to expose plugin configuration options to the REST API, which is what a web ui would access to control this stuff. I've thought about it a lot, so I think I know how I'd do it, but there's details to be worked out and code to be written around this. Mailman has support for anonymous lists, which can rewrite the From header to substitute the mailing list's posting address for the original author's address. I frankly don't know how much that feature is used in MM2.1, if at all. -Barry
Answer by 1 year ago
Quoted message by Stephen J. Turnbull 1 year ago
Hey, I've been trying to find the superuser at Gmail, and it turns out I've known him all along! Would you please fix the breakage where copies of one's posts aren't delivered to yourself, which is a real PITA when trying to diagnose Gmail problems?<wink/> (BTW, did you try to sign up as root? or Administrator? ;-) I second your concerns in principle, but I don't think you need to worry about these features being anything but optional. I suspect that a majority of our user don't actually have the fine control over their DNS needed to implement DMARC. Violating the RFC by munging From (for heaven's sake! have we learned nothing from the sad history of Reply-To munging?) is simply not acceptable for most purposes, although perfectly acceptable for advertising. ;-) Steve
Actually, Reply-To munging is a bit more complicated. There is a boolean flag that indicates whether any existing Reply-To headers should first be stripped, the default being not to strip. Then there is a ternary option to specify what kind of munging should happen. The default is not to munge Reply-To. The other two options are to put the mailing list's posting address in the Reply-To, or to put an explicit header (which the list admin can enter) into the Reply-To. I am strongly in the Don't Munge camp, so the default is to leave all these headers alone. Mostly, the munge options were added to silence the vocal Munge advocates. Using an explicit header has a valid use case: redirecting replies to an announce-only list posting to a discussion mailing list. Of course, the effects of any of these headers is heavily dependent on the end user's MUA and the user's own habits. I think peoples' opinion on these contentious issues is almost always driven by the MUA they use. Mine has very nice buttons to do the appropriate kind of reply (although often Gmail conspires against doing the right thing... so what's new there? ;). -Barry
Answer by 1 year ago
One other thing to keep in mind. Most list administrators have no clue how to configure their lists. Just like most technologies, they'll use whatever defaults get shipped. Postmaster are more clueful for sure (our own python.org ones being tip top :) and some of them do get involved in list configurations. MM3 supports "list styles" which are essentially composable settings applied when a mailing list is created. If there could even be such a thing as a "DMARC style", it would only need to touch the DMARC related options, and this style could be shipped in the previously mentioned plugin. -Barry
Answer by 1 year ago
Quoted message by Franck Martin 1 year ago
This is speculation, and broad fear of the future... The current practice for a postmaster is to trust (or not) emails from specific mailing lists, not who post them to the list. Adding DKIM to the list and taking ownership will only improve it. You should really read the code of the patch for MM2 and try it.
It doesn't work on people who convince themselves it's the only way to solve their problems. It often does work on people who are grasping at straws, and know it. You can often convince the latter that they'll only make their problems worse.
Answer by 1 year ago
Quoted message by Barry Warsaw 1 year ago
In this message, I'll talk about implementation strategies. In very general terms, what I think you're looking at is writing an extension/plugin that provides additional rules and handlers, and then modifying or extending the built-in processing chains and pipelines to consult those new rules and handlers. In this way, DMARC support (or some subset of it) can start out as an unofficial extension, which then based on experience may eventually make it into the standard distribution. I've spoken many times about the architecture for moderation (rules and chains) and modification (pipeline of handlers), so I won't go into more detail here. Please start with my Pycon 2012 talk and feel free to ask any questions you may still have. In some sense there is overlap with how I would integrate any other anti-spam tool in MM3. Most of us agree that such things are better done in the MTA, but it's just not always possible to do that. List admins have more control over how their lists are configured than over their MTA, so I do see value in providing these services in MM3, as plugins. For #1 you would have a rule that can answer the question of DMARC disposition. Rules output binary results, and if this rule hits, it would run an action, probably to discard the message, although it could also hold it or reject/bounce it. I doubt you'd want to accept the message. If the rule misses then processing continues along as normal. Munging the From header is done by a processing handler, after the message has already been accepted for posting, and is being prepared for delivery to the list membership. Similarly, the Authentication-Results rewrite would be done in the same or a different handler. With respect to both headers, I'm assuming that the munging is the same for every recipient; there's a different hook point for personalization of outgoing messages. -Barry
Good idea, to held the message for approval. It is harder to know what you don't know... Holding the message for approval would give a chance for the list admin to guide the subscriber. I thought about munging the From Header on a sender basis (if it has a DMARC policy) but that makes the list oscillating between two states therefore making life difficult for the recipient (rules to sort messages).
Answer by 1 year ago
Quoted message by Barry Warsaw 1 year ago
Current MM3 doesn't have a great way to expose plugin configuration options to the REST API, which is what a web ui would access to control this stuff. I've thought about it a lot, so I think I know how I'd do it, but there's details to be worked out and code to be written around this. Mailman has support for anonymous lists, which can rewrite the From header to substitute the mailing list's posting address for the original author's address. I frankly don't know how much that feature is used in MM2.1, if at all. -Barry
This code was useful to do the patch for MM2.1. I too doubt it is much in use, after all we gave away our privacy long time ago on all the social networks and cloud services... but I guess it answered a need otherwise it would not have been there.
Answer by 1 year ago
Quoted message by Barry Warsaw 1 year ago
In this message, I'll talk about implementation strategies. In very general terms, what I think you're looking at is writing an extension/plugin that provides additional rules and handlers, and then modifying or extending the built-in processing chains and pipelines to consult those new rules and handlers. In this way, DMARC support (or some subset of it) can start out as an unofficial extension, which then based on experience may eventually make it into the standard distribution. I've spoken many times about the architecture for moderation (rules and chains) and modification (pipeline of handlers), so I won't go into more detail here. Please start with my Pycon 2012 talk and feel free to ask any questions you may still have. In some sense there is overlap with how I would integrate any other anti-spam tool in MM3. Most of us agree that such things are better done in the MTA, but it's just not always possible to do that. List admins have more control over how their lists are configured than over their MTA, so I do see value in providing these services in MM3, as plugins. For #1 you would have a rule that can answer the question of DMARC disposition. Rules output binary results, and if this rule hits, it would run an action, probably to discard the message, although it could also hold it or reject/bounce it. I doubt you'd want to accept the message. If the rule misses then processing continues along as normal. Munging the From header is done by a processing handler, after the message has already been accepted for posting, and is being prepared for delivery to the list membership. Similarly, the Authentication-Results rewrite would be done in the same or a different handler. With respect to both headers, I'm assuming that the munging is the same for every recipient; there's a different hook point for personalization of outgoing messages. -Barry
We are not asking mailman to do the work of DMARC here. There is openDMARC for that.
Answer by 1 year ago
Quoted message by Barry Warsaw 1 year ago
In this message, I'll talk about implementation strategies. In very general terms, what I think you're looking at is writing an extension/plugin that provides additional rules and handlers, and then modifying or extending the built-in processing chains and pipelines to consult those new rules and handlers. In this way, DMARC support (or some subset of it) can start out as an unofficial extension, which then based on experience may eventually make it into the standard distribution. I've spoken many times about the architecture for moderation (rules and chains) and modification (pipeline of handlers), so I won't go into more detail here. Please start with my Pycon 2012 talk and feel free to ask any questions you may still have. In some sense there is overlap with how I would integrate any other anti-spam tool in MM3. Most of us agree that such things are better done in the MTA, but it's just not always possible to do that. List admins have more control over how their lists are configured than over their MTA, so I do see value in providing these services in MM3, as plugins. For #1 you would have a rule that can answer the question of DMARC disposition. Rules output binary results, and if this rule hits, it would run an action, probably to discard the message, although it could also hold it or reject/bounce it. I doubt you'd want to accept the message. If the rule misses then processing continues along as normal. Munging the From header is done by a processing handler, after the message has already been accepted for posting, and is being prepared for delivery to the list membership. Similarly, the Authentication-Results rewrite would be done in the same or a different handler. With respect to both headers, I'm assuming that the munging is the same for every recipient; there's a different hook point for personalization of outgoing messages. -Barry
Of course you are, in feature #1. Unless you take it literally (reject all email that comes from such a domain, *including* email that would authenticate correctly in a full DMARC implementation). I think that the appropriate interpretation of that feature request is that "in some cases, Mailman needs to play the role of MTA in the DMARC protocol." Anything less than a full implementation is a denial of service and screws everybody: the domain owners, the recipients, and the Mailman site admins, list admins, and moderators. And us, who will take the lion's share of PRs for it even if it's a third-party module. Steve
Answer by 1 year ago
Quoted message by Barry Warsaw 1 year ago
In this message, I'll talk about implementation strategies. In very general terms, what I think you're looking at is writing an extension/plugin that provides additional rules and handlers, and then modifying or extending the built-in processing chains and pipelines to consult those new rules and handlers. In this way, DMARC support (or some subset of it) can start out as an unofficial extension, which then based on experience may eventually make it into the standard distribution. I've spoken many times about the architecture for moderation (rules and chains) and modification (pipeline of handlers), so I won't go into more detail here. Please start with my Pycon 2012 talk and feel free to ask any questions you may still have. In some sense there is overlap with how I would integrate any other anti-spam tool in MM3. Most of us agree that such things are better done in the MTA, but it's just not always possible to do that. List admins have more control over how their lists are configured than over their MTA, so I do see value in providing these services in MM3, as plugins. For #1 you would have a rule that can answer the question of DMARC disposition. Rules output binary results, and if this rule hits, it would run an action, probably to discard the message, although it could also hold it or reject/bounce it. I doubt you'd want to accept the message. If the rule misses then processing continues along as normal. Munging the From header is done by a processing handler, after the message has already been accepted for posting, and is being prepared for delivery to the list membership. Similarly, the Authentication-Results rewrite would be done in the same or a different handler. With respect to both headers, I'm assuming that the munging is the same for every recipient; there's a different hook point for personalization of outgoing messages. -Barry
While ugly, that might be the best we can do for now. I have thought about adding an action to links for when the rule misses, the default being 'Defer' (i.e the next link in the chain executes as normal). That would at least give you more control over each step in the chain. But handling more than two cases quickly gets into ugliness. Another possibility is to collapse the reject/quarantine "hit" into a single boolean result. Rules can add key/values to the metadata dictionary, so you could imagine that a hit wouldn't jump directly to the Reject or Hold chains. Instead it would jump to a custom (terminal) chain that made the more specific determination of whether to reject or hold the message. Of course, we'd likely log and fire an event, so at least it wouldn't happen completely silently. Yep. There is some limited ability to do additional checking at LMTP time, but this isn't pluggable currently. Currently there's only one moderation queue, but it can be set up to auto-discard held requests after a period of time. -Barry
Answer by 1 year ago
Quoted message by Barry Warsaw 1 year ago
In this message, I'll talk about implementation strategies. In very general terms, what I think you're looking at is writing an extension/plugin that provides additional rules and handlers, and then modifying or extending the built-in processing chains and pipelines to consult those new rules and handlers. In this way, DMARC support (or some subset of it) can start out as an unofficial extension, which then based on experience may eventually make it into the standard distribution. I've spoken many times about the architecture for moderation (rules and chains) and modification (pipeline of handlers), so I won't go into more detail here. Please start with my Pycon 2012 talk and feel free to ask any questions you may still have. In some sense there is overlap with how I would integrate any other anti-spam tool in MM3. Most of us agree that such things are better done in the MTA, but it's just not always possible to do that. List admins have more control over how their lists are configured than over their MTA, so I do see value in providing these services in MM3, as plugins. For #1 you would have a rule that can answer the question of DMARC disposition. Rules output binary results, and if this rule hits, it would run an action, probably to discard the message, although it could also hold it or reject/bounce it. I doubt you'd want to accept the message. If the rule misses then processing continues along as normal. Munging the From header is done by a processing handler, after the message has already been accepted for posting, and is being prepared for delivery to the list membership. Similarly, the Authentication-Results rewrite would be done in the same or a different handler. With respect to both headers, I'm assuming that the munging is the same for every recipient; there's a different hook point for personalization of outgoing messages. -Barry
Verbose, yes. Is it really ugly, though? I don't know how much you were directly influenced by iptables and SIEVE, but the idea of rule chains as a way to very flexibly configure filters has been implemented many times. The model is very simple and completely flexible. This is pretty much what I was suggesting. No, but it might be many days before the originator gets around to asking why their message hasn't appeared. Does LMTP provide the necessary ability to reject? Steve
Answer by 1 year ago
Quoted message by Stephen J. Turnbull 1 year ago
Verbose, yes. Is it really ugly, though? I don't know how much you were directly influenced by iptables and SIEVE, but the idea of rule chains as a way to very flexibly configure filters has been implemented many times. The model is very simple and completely flexible. This is pretty much what I was suggesting. No, but it might be many days before the originator gets around to asking why their message hasn't appeared. Does LMTP provide the necessary ability to reject? Steve
That's what I meant. With luck, then, the LMTP rejection could happen during the MTA's SMTP transaction.