...making Linux just a little more fun!
Rick Moen [rick at linuxmafia.com]
Wed, 30 Aug 2006 19:22:58 -0700
Ben suggested I forward this mailing list post here, as it's relevant to my recent article submission. [http://linuxgazette.net/131/moen.html]
----- Forwarded message from rick -----
Date: Wed, 30 Aug 2006 19:00:21 -0700 To: luv-main at luv.asn.au Cc: ben at linuxgazette.net Subject: Re: SRS development Quoting Daniel Pittman (daniel at rimspace.net):
> I wish you wouldn't. SPF is great, in theory, but not really so hot in > practice. Using it will only encourage others to adopt it.
There's a lot of confusion about this. Let's dissect it into parts:
o SPF reference records within a domain's DNS: These allow the domain administrator to specify which IPs are those of the domain's (and optionally, various subdomains') sole intended MX hosts -- along with suggestions as to the degree of confidence that the admin feels receiving SMTP hosts should grant to that data. o SPF-checking routines executed by a receiving SMTP host's MTA during the incoming SMTP session, prior to saying yes or no to the delivery attempt. (It's also possible to check SPF RRs at later points, such as during MDA execution, but less useful.)Speaking from a domain administrator's perspective, one has _no downside_ from publishing SPF RRs for one's domain. It's good and useful data, providing the public with a definitive means of detecting and rejecting mail forged to dishonestly claim your domain as its origin, even mail competently forging the 'envelope From' and Return-Path headers (Joe Jobs). It's categorically useful even to domain admins who favour alternate approaches. I.e., is you think Hadmut Danisch's Reverse-MX (RMX) proposal was better designed, or Yahoo's DomainKeys, or SPF plus Meng/Microsoft's Purported Responsible Address (PRA) header, or S/MIME, or Email Postmarks, or a full gpg web-of-trust architecture, or BATV, or Jim Fenton's Identified Internet Mail -- then nothing stops you from participating in all of them, if you wish.
It would be a Good Thing even if nothing like SRS (Sender Rewriting Scheme) had even been attempted, and every forwarding mechanism on the globe other than mailing lists (which have rewritten 'envelope From' and Return-Path all along) were to instantly and permanently break.
Why? _Because I own domains._ I'm therefore tired of spammers and malware authors being able to Joe Job them believably. Why would I not spend five minutes writing a TXT RR that makes crystal clear that mx4.pinkmeatproducts.com isn't a legitimate source for my domains' mail?
'No so hot', you say? OK, compared to what? The relevant comparison for a domain owner is: extant SPF record versus none. Do you honestly assert that my domains would be better off without them? If so, please do explain.
Please don't say 'It's not so hot because DomainKeys might be better.' Please don't say 'It's not so hot because it doesn't cure the spam problem.' Please don't say 'It's not so hot because it'll make it more difficult to forward mail from my roving laptop through arbitrary smarthosts, and make me rewrite my /etc/alias and ~/.forward entries.' Please don't tell me there are edge cases where the Right Thing for an MTA might not be obvious, such as when an SPF lookup returns Fail.
Those are all true, but irrelevant to the question.
Thank you.
-- Cheers, Founding member of the Hyphenation Society, a grassroots-based, Rick Moen not-for-profit, locally-owned-and-operated, cooperatively-managed, rick at linuxmafia.com modern-American-English-usage-improvement association.
Rick Moen [rick at linuxmafia.com]
Wed, 30 Aug 2006 20:49:40 -0700
OK, I should provide some context: When Meng Wong drafted the SPF proposal, he knew there would be a variety of bullshit objections:
o It doesn't {solve the spam problem | bring about world peace | etc.}
o It might interfere with my favourite way of relaying mail through
multiple SMTP servers (/etc/aliases, ~/.forward, commercial mail
forwarding services, etc.) -- because accurate designation in the
DNS of which IPs are authorised to handle my domain's outbound
mail will tend to make my mail that I riccochet through those
mechanisms unauthorised, when examined by SPF-checking software
that receives it.
o It might interfere with my ability to lob off outbound SMTP
mail directly from any arbitrary IP address at will.
o There are alternate proposals, that might be better.
One relevant (and tempting) possible reply would be 'So what?' Which,
as you'll see below, tends to be my response.
1. SPF never aimed to cure the spam problem, or any other grandiose
objective. It's solely designed to validate mail-sending domains,
giving existing legitimate mail-sending domains a tool to make
clear which mail purporting to be from those domains is real,
and which is forged.
2. Yes, people who send mail out in dodgy fashions run the risk of
looking like domain-forgers. Deal with it, guys! A friend of mine,
here in Silicon Valley, had a classic t-shirt printed up, just after the
dot-com bust: 'Sorry to hear about your business model.' In the
same spirit: 'Sorry to hear about your problematic mail-forwarding
method. Hope you get better, soon.'
Suffice it to say, there are workarounds to fix all of those problematic
mail-forwarding and sending-SMTP-from-arbitrary-IPs models so that they
don't end up issuing mail that looks to an SPF-aware world as if it's
forged. However, I'd be delighted with SPF even if those didn't exist
or were hopeless, because it fixes _a serious problem_ very well!
Meng Wong, however, is a nicer guy than I am, and so has worked hard on those workarounds. Many involve his 'Sender Rewriting Scheme' (SRS), cited in the Subject header and moaned over endlessly by my correspondent. SRS is a means for doing forwarding (including the /etc/aliases and ~/.forward kind) in an SPF-friendly manner -- usually using the services of a Perl library, libsrs.
People like my correspondent keep trying to invent outlandish scenarios in which they allege that SPF-checking SMTP services could be a problem for them, because something they might want to do could break, either because libsrs isn't yet good enough, or it isn't locally available, or hasn't yet been implemented, or for some other wacky reason. He asserts that progress must not be allowed to occur, because he or somebody else might have a problem!
My point to him is: Sorry, but the world is likely to move forward regardless of your problem, and so it really doesn't matter whether your problem is real (and non-trivial) or not. Better concentrate on coping, sir.
To read more, the single best source is Meng Wong's 'whitepaper': http://www.openspf.org/whitepaper.pdf
----- Forwarded message from Daniel Pittman <daniel at rimspace.net> -----
From: Daniel Pittman <daniel@rimspace.net> To: TAG <tag@lists.linuxgazette.net> To: luv-main at luv.asn.au Date: Thu, 31 Aug 2006 12:34:09 +1000 Subject: Re: SRS developmentRick Moen <rick at linuxmafia.com> writes:
> Quoting Daniel Pittman (daniel at rimspace.net): > >> I wish you wouldn't. SPF is great, in theory, but not really so hot in >> practice. Using it will only encourage others to adopt it. > > There's a lot of confusion about this. Let's dissect it into parts: > > o SPF reference records within a domain's DNS: These allow the domain > administrator to specify which IPs are those of the domain's > (and optionally, various subdomains') sole intended MX hosts -- > along with suggestions as to the degree of confidence that the > admin feels receiving SMTP hosts should grant to that data. > > o SPF-checking routines executed by a receiving SMTP host's MTA > during the incoming SMTP session, prior to saying yes or no to > the delivery attempt. (It's also possible to check SPF RRs at later > points, such as during MDA execution, but less useful.) > > Speaking from a domain administrator's perspective, one has _no downside_ > from publishing SPF RRs for one's domain. It's good and useful data, > providing the public with a definitive means of detecting and rejecting > mail forged to dishonestly claim your domain as its origin,
As long as there are no legitimate outbound emails using your From address, which is not the case in a wide range of circumstances.
[...]
> 'No so hot', you say? OK, compared to what?
Not compared to anything, just in and of itself. Perhaps "not such a great idea" would have been a better way to phrase that.
"Not so hot" does suggest that I had a better solution to offer which, at this stage, I don't. All I have to offer is the "not worse" alternative of not using a tool I believe to be broken.
> The relevant comparison for a domain owner is: extent SPF record > versus none. Do you honestly assert that my domains would be better > off without them? If so, please do explain.
Yes, I think so, because when you deploy SPF records you encourage others to also deploy SPF records. However...
> Please don't say 'It's not so hot because DomainKeys might be better.' > Please don't say 'It's not so hot because it doesn't cure the spam > problem.'
... I don't assert either of these.
> Please don't say 'It's not so hot because it'll make it more difficult > to forward mail from my roving laptop through arbitrary smarthosts, and > make me rewrite my /etc/alias and ~/.forward entries.' > Please don't tell me there are edge cases where the Right Thing for an > MTA might not be obvious, such as when an SPF lookup returns Fail, > > Those are all true, but irrelevant to the question.
You just asked me not to use the reasons why I do believe that. The first two are not relevant, yes.
The final points are relevant, in my opinion, because they are the core of the argument that SPF is not a good tool -- it has gaping holes in the initial design where existing services were not considered.
The addition of SRS to the protocol does address those cases, but by consequence moved SPF from "no changes to email" into the "replace SMTP with something similar but different" class of solutions.
You feel that is not relevant, though, which is your prerogative. The net result that we disagree on this topic in such a manner that no agreement will be possible.
I have reasons I believe sound, you categorically reject them, so we must simply agree to disagree on the topic.
Regards, Daniel
--
Digital Infrastructure Solutions -- making IT simple, stable and secure
Phone: 0401 155 707 email: contact at digital-infrastructure.com.au
http://digital-infrastructure.com.au/
----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Wed, 30 Aug 2006 20:19:27 -0700 To: luv-main at luv.asn.au From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentQuoting Daniel Pittman (daniel at rimspace.net):
> > Speaking from a domain administrator's perspective, one has _no downside_ > > from publishing SPF RRs for one's domain. It's good and useful data, > > providing the public with a definitive means of detecting and rejecting > > mail forged to dishonestly claim your domain as its origin. > > As long as there are no legitimate outbound emails using your From > address, which is not the case in a wide range of circumstances.
Huh? Either I'm too tired, or that sentence doesn't parse.
Consider domain linuxmafia.com, which has as its sole MX the machine uncle-enzo.linuxmafia.com (aka linuxmafia.com), IP 198.144.195.186.
$ dig -t txt linuxmafia.com +short "v=spf1 a mx -all" $ dig -t mx linuxmafia.com +short 10 linuxmafia.com. $ dig -t a linuxmafia.com +short 198.144.195.186Each and every day of the year, there is a tonne of legitimate outbound e-mails using From addresses (envelope aka MAIL FROM) citing that domain -- not to mention its legitimate use in Return-Path headers, Received headers, internal "From:" headers, etc.
Some of that outbound mail originates in roving laptops on arbitrary dynamic IPs -- sent through uncle-enzo over a variety of methods such as SMTPS on 587/tcp, SSL-accessed webmail, SSH shell access, and SSH tunneling. Some of it's locally generated. But there's a huge lot of it. Every day.
So, 'as long as there is no legitimate outbound e-mails using my From address'? To the contrary, sir: All of the large amount of legitimate outbound e-mail is of that exact nature.
> > 'No so hot', you say? OK, compared to what? > > Not compared to anything, just in and of itself.
Which in effect means: compared to _doing nothing_ about domain forgeries (Joe Jobs). OK, show me.
But then, you did not.
With my publication of SPF RRs in my DNS, nobody can plausibly pull Joe Jobs on my domains, any more. Without them, they could and did.
Thus my question, which you conspicuously declined to address.
> > The relevant comparison for a domain owner is: extent SPF record > > versus none. Do you honestly assert that my domains would be better > > off without them? If so, please do explain. > > Yes, I think so, because when you deploy SPF records you encourage > others to also deploy SPF records.
Thus, you are now avoiding two questions:
o Do you assert that my domains would be better off (and how)? o Do you assert that those others' domains would be better off (and how)?I remain interested in your answer -- but tentatively conclude that you probably completely lack one.
> The final points are relevant, in my opinion, because they are the core > of the argument that SPF is not a good tool [...]
One more time: Why is it not in MY INTEREST AS A DOMAIN OWNER to publish SPF RRs in my DNS, given that it immediately and permanently prevents credible Joe Job forgery of those domains' mail? You have not addressed my question.
Because you almost certainly can't, sir. It's a no-brainer boon to any domain admin.
All you can do is say 'Some applications of it by some other MTAs, somewhere, could break $MY_FAVOURITE_FORWARDING'. Which is _irrelevant to the question_.
> The addition of SRS to the protocol does address those cases, but by > consequence moved SPF from "no changes to email" into the "replace SMTP > with something similar but different" class of solutions.
I'm having flashbacks to all the people a decade ago who said 'But I rely on open mail relays. Pressuring people to close them isn't a good solution! There are gaping holes in that plan!'
Sorry to hear about your problems with forwarding, sending SMTP from arbitrary IPs, etc. -- but I fail to see why I should do nothing about spammers and malware authors forging my domains, just because you have a possible mail-forwarding problem elsewhere entirely that has not nothing to do with my domains at all.
On that matter, basically, when you get tired of saying 'No, wait! This emergence of warm-bloodedness and live birth is bad, and could make life inconvenient for many dinosaurs', you might consider the merits of evolving. ;->
> I have reasons I believe sound, you categorically reject them, so we > must simply agree to disagree on the topic.
I await your actually _addressing my question_ with interest.
-- Cheers, "Heedless of grammar, they all cried 'It's him!'" Rick Moen -- R.H. Barham, _Misadventure at Margate_ rick at linuxmafia.com
Rick Moen [rick at linuxmafia.com]
Wed, 30 Aug 2006 22:07:51 -0700
----- Forwarded message from Daniel Pittman <daniel at rimspace.net> -----
From: Daniel Pittman <daniel@rimspace.net> To: TAG <tag@lists.linuxgazette.net> To: luv-main at luv.asn.au Date: Thu, 31 Aug 2006 14:20:35 +1000 X-Spam-Status: No, score=-2.4 required=4.0 tests=AWL,BAYES_00, FORGED_RCVD_HELO autolearn=ham version=3.1.1 Subject: Re: SRS development Rick Moen <rick at linuxmafia.com> writes:
> Quoting Daniel Pittman (daniel at rimspace.net): > >> > Speaking from a domain administrator's perspective, one has _no downside_ >> > from publishing SPF RRs for one's domain. It's good and useful data, >> > providing the public with a definitive means of detecting and rejecting >> > mail forged to dishonestly claim your domain as its origin. >> >> As long as there are no legitimate outbound emails using your From >> address, which is not the case in a wide range of circumstances. > > Huh? Either I'm too tired, or that sentence doesn't parse.
You are right: I missed a key clause, and it doesn't make sense as it stands. The clause in question was "from servers that you don't run."
[...]
> One more time: Why is it not in MY INTEREST AS A DOMAIN OWNER to > publish SPF RRs in my DNS, given that it immediately and permanently > prevents credible Joe Job forgery of those domains' mail? You have > not addressed my question. > > Because you almost certainly can't, sir. It's a no-brainer boon to > any domain admin.
It may be beneficial to you, with your userbase and use of email as it stands. That depends almost entirely on how you interact with sending email; if it all comes from a set of servers that you run then, yes, SPF can be of some benefit.
It isn't a boon for all users: anyone who uses a wide range of other services that may result in email exiting a server that you don't control, but with your details, is faced with a range of unpleasant alternatives.
> All you can do is say "Some applications of it by some other MTAs, > somewhere, could break $MY_FAVOURITE_FORWARDING". Which is > _irrelevant to the question_.
Fair enough. I still don't agree with that assertion, but I don't think you will change your mind on that front.
[...]
> Sorry to hear about your problems with forwarding, sending SMTP from > arbitrary IPs, etc. -- but I fail to see why I should do nothing about > spammers and malware authors forging my domains, just because you have > a possible mail-forwarding problem elsewhere entirely that has not > nothing to do with my domains at all.
I wouldn't suggest doing nothing. I just wouldn't suggest SPF, as it represents a solution to the easy part of the "Joe Job" problem, and hasn't really addressed the hard parts.
> On that matter, basically, when you get tired of saying "No, wait! > This emergence of warm-bloodedness and live birth is bad, and could make > life inconvenient for many dinosaurs", you might consider the merits of > evolving. ;->
I might well consider those merits. Just give me a few million years to see how it works out for y'all. ;)
Seriously: I am sorry you feel that my objection that SPF and SRS break real world use of email is not a good objection to the use of SPF.
You are right, though: I don't have any further argument to oppose it.
So, be my guest. Go ahead and advocate the deployment of SPF more broadly, and the widespread use of it. ;)
Perhaps I am wrong, and the existing and valuable services are, in fact, not as valuable -- or as existing -- as I believe.
Perhaps SPF does, in fact, address these issues and I can't see that, or they are irrelevancies, a legacy of times gone by when the Interweb was a friendlier place or something.
I am sorry, though, if you feel that is avoiding your questions. I don't believe it is, and it certainly isn't an intentional avoidance of them.
I think we simply have different values here: you obviously value the benefits that SPF records bring you justify the cost, which is fine.
I see SPF as a bad solution, because it cannot work in a number of environments that I deal with. Without that being solved[1] SPF is a loss for us.
Regards, Daniel
Footnotes: [1] ...through SRS being deployed by every service provider we use, or something similar.
--
Digital Infrastructure Solutions -- making IT simple, stable and secure
Phone: 0401 155 707 email: contact at digital-infrastructure.com.au
http://digital-infrastructure.com.au/
----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Wed, 30 Aug 2006 22:01:13 -0700 To: luv-main at luv.asn.au From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentQuoting Daniel Pittman (daniel at rimspace.net):
> You are right: I missed a key clause, and it doesn't make sense as it > stands. The clause in question was "from servers that you don't run."
OK, imagine I'm right now sitting at a machine I don't run. I want to send outbound SMTP mail with an originating domain of "linuxmafia.com". I see several obvious ways, all of which work, and many of which my users already use routinely:
1. Configure a MUA to send using SMTP AUTH on 587/tcp to linuxmafia.com.
Note that this is a bog-standard solution, pretty much universally
available. Just to make double-sure of that, I just checked on a
copy of _Microsoft Outlook_. Even it does it! It's right there
in "Add a new e-mail account."
And, before you say it, your time-honoured ridiculous objection about
the hotel blocking access to outbound ports or a mandatory
transparent SMTP proxy isn't likely to apply: 587/tcp doesn't
invite blocking measures, as its IETF-mandated standard of being
used only for authenticated SMTP tends to prevent it from becoming
the public-menacing sewer that 25/tcp is.
2. Fire up a Web browser. Connect to Squirrelmail.
3. SSH to my shell on linuxmafia.com. Use mutt. (If necessary, and
on someone's Crippled OS machine, copy putty.exe yet again from
the USB flash drive in my pocket.) This option is available only to
users who have shell access and like console mailers -- which I do
-- but that limitation doesn't apply to any of the other answers.
4. SSH forwarding and a nullmailer. (And, if they're blocking outbound
22/tcp, that's perfectly OK: sshd is also listening on 23/tcp and a
couple of others.)
Note: I've mentioned all of these before.
I'm aware you said "from servers that you don't run". Sorry, I don't route my mail (or my users' mail) through servers I don't run. However, if I did, and relied on either /etc/aliases entries or ~/.forward entries on it to redirect my domain's mail, and if that server were so antique and decrepit that I weren't allowed to copy libsrs into ~/bin and use it for my redirects, then I'd just add the machine's IP to my domain's SPF record, making it an additional authorised MX. No SRS required.
Don't tell me you didn't think of that?
> It may be beneficial to you, with your userbase and use of email as it > stands. That depends almost entirely on how you interact with sending > email; if it all comes from a set of servers that you run then, yes, SPF > can be of some benefit.
See rejoinder, above.
> It isn't a boon for all users: anyone who uses a wide range of other > services that may result in email exiting a server that you don't > control, but with your details, is faced with a range of unpleasant > alternatives.
See rejoinder above, and tell me which part of it doesn't work. (Hint: It all works.)
> I wouldn't suggest doing nothing. I just wouldn't suggest SPF, as it > represents a solution to the easy part of the "Joe Job" problem, and > hasn't really addressed the hard parts.
Maybe you're not getting the fact that having rendered Joe Job forgeries no longer feasible matters. And my having done that didn't damage anything for anyone -- not me, not my users, and not anyone else.
If I relied on .forward and /etc/aliases forwarding between SMTP hosts for my in-domain outbound mail, I'd need to revamp to use libsrs. Which does work, by the way -- and it's not like it's difficult to deploy, just a small Perl library, for Ghu's sake. But I don't need those things, in part because I long ago classed those as part of the problem, and something to phase out -- just as open mail relays were, a decade ago.
> So, be my guest. Go ahead and advocate the deployment of SPF more > broadly, and the widespread use of it. ;)
As chance would have it, that'll be in the next issue of _Linux Gazette_. ;->
> I see SPF as a bad solution, because it cannot work in a number of > environments that I deal with.
1992: "I see shutting down open relays as a bad solution, because it breaks a number of the usage models I deal with."
> ...through SRS being deployed by every service provider we use, or > something similar.
I'll have to admit I'm curious about what you do that's so dependent on /etc/aliases-style mail forwarding. Maybe you're solving the wrong problem. Personally, I got away from all that stuff, years and years ago!
-- Cheers, * Contributing Editor, Linux Gazette * Rick Moen -*- See the Linux Gazette in its new home: -*- rick at linuxmafia.com <http://linuxgazette.net/>
Rick Moen [rick at linuxmafia.com]
Wed, 30 Aug 2006 22:27:11 -0700
A new poster to the thread, with something substantive to say.
----- Forwarded message from Matthew Wallis <mattw at madmonks.org> -----
Date: Thu, 31 Aug 2006 15:16:19 +1000 To: luv-main at luv.asn.au From: Matthew Wallis <mattw@madmonks.org> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentOn Thu, Aug 31, 2006 at 02:20:35PM +1000, Daniel Pittman wrote:
> Rick Moen <rick at linuxmafia.com> writes: > > Quoting Daniel Pittman (daniel at rimspace.net): > > > >> > Speaking from a domain administrator's perspective, one has _no downside_ > >> > from publishing SPF RRs for one's domain. It's good and useful data, > >> > providing the public with a definitive means of detecting and rejecting > >> > mail forged to dishonestly claim your domain as its origin. > >> > >> As long as there are no legitimate outbound emails using your From > >> address, which is not the case in a wide range of circumstances. > > > > Huh? Either I'm too tired, or that sentence doesn't parse. > > You are right: I missed a key clause, and it doesn't make sense as it > stands. The clause in question was "from servers that you don't run." > > [...] > > > One more time: Why is it not in MY INTEREST AS A DOMAIN OWNER to > > publish SPF RRs in my DNS, given that it immediately and permanently > > prevents credible Joe Job forgery of those domains' mail? You have > > not addressed my question.
As an example of a legitimate[1] case where a server not controlled by you sends mail on your behalf.
Any one of a number of job advertising sites will allow you to use a form on their website to submit a job application.
You fill it out with your email address, and upload your resume as an attachment, and the site sends it on to the potential employer, using your address as the from address, so that any correspondence from said employer goes back to the applicant.
You then of course find that mail from that website is handled by a completely seperate mail server provided by the hosting company, so just whitelisting servers from that domain doesn't work, you have to check log files to see where the message really came from.
It's this sort of "legitimate" abuse that made me give up on stopping spam, and switch to tag and bag with spamassassin. Show the users how to setup filters based on score, and let them deal with it.
Greylisting would handle this reasonably effectively, but that assumes that the sending mailserver handles 450's appropriately, but last I checked places like Telstra had a buisness product based on a broken virus-scanning MTA product, so I don't hold out much hope of that.
Matt.
[1] For certain broad definitions of legitimate.
----- End forwarded message -----
----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Wed, 30 Aug 2006 22:25:05 -0700 To: luv-main at luv.asn.au From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentQuoting Matthew Wallis (mattw at madmonks.org):
> As an example of a legitimate[1] case where a server not controlled by > you sends mail on your behalf. > > Any one of a number of job advertising sites will allow you to use a > form on their website to submit a job application. > > You fill it out with your email address, and upload your resume as an > attachment, and the site sends it on to the potential employer, using > your address as the from address, so that any correspondence from said > employer goes back to the applicant. > > You then of course find that mail from that website is handled by a > completely seperate mail server provided by the hosting company, so just > whitelisting servers from that domain doesn't work, you have to check > log files to see where the message really came from.
This sort of scenario is why standard implementations of SPF-checking in MTAs includes a list of known "legitimate" but non-compliant forwarders of exactly this sort -- and permits mail routed through them by default.
Of course, if one doesn't trust that, one can use (e.g.) a GMail account on one's resume, thus exporting the dealing-with-clueless-forwarders problem to Google, Inc.
Alternatively, you can decide (as I did) that you'd rather -not- have outside entities of unknown competence handling any of your domain's outbound mail on your behalf. ;->
----- End forwarded message -----
Neil Youngman [ny at youngman.org.uk]
Thu, 31 Aug 2006 09:16:44 +0100
On or around Thursday 31 August 2006 06:27, Rick Moen reorganised a bunch of electrons to form the message:
> A new poster to the thread, with something substantive to say. > > ----- Forwarded message from Matthew Wallis <mattw at madmonks.org> ----- ><Quoted messages SNIPPED>
> As an example of a legitimate[1] case where a server not controlled by > you sends mail on your behalf. > > Any one of a number of job advertising sites will allow you to use a > form on their website to submit a job application. > > You fill it out with your email address, and upload your resume as an > attachment, and the site sends it on to the potential employer, using > your address as the from address, so that any correspondence from said > employer goes back to the applicant. > > You then of course find that mail from that website is handled by a > completely seperate mail server provided by the hosting company, so just > whitelisting servers from that domain doesn't work, you have to check > log files to see where the message really came from. > > It's this sort of "legitimate" abuse that made me give up on stopping > spam, and switch to tag and bag with spamassassin. Show the users how to > setup filters based on score, and let them deal with it. > > Greylisting would handle this reasonably effectively, but that assumes > that the sending mailserver handles 450's appropriately, but last I > checked places like Telstra had a buisness product based on a broken > virus-scanning MTA product, so I don't hold out much hope of that.
The solution to this is in the hands of the site that does the emailing for you and it's for them to provide some sort of forwarding.
One possibility is the old "percent hack", obviously with sufficient authentication to ensure that they are only relaying to addresses for which they have sent messages .
Another possibility is a temporary alias that they forward to the senders address, possibly with the real email address as the familiar name if they want that to be available to the recipient, e.g. From: "pete.marsh at client.example.com" <temp112345@server.example.com> To: TAG <tag@lists.linuxgazette.net>
And of course there's always the Reply-To: Header.
There are solutions for this sort of application that don't require SPF to be ditched.
Neil Youngman
Rick Moen [rick at linuxmafia.com]
Thu, 31 Aug 2006 14:48:28 -0700
Quoting Neil Youngman (ny at youngman.org.uk):
> On or around Thursday 31 August 2006 06:27, Rick Moen reorganised a bunch of > electrons to form the message: > > A new poster to the thread, with something substantive to say. > > > > ----- Forwarded message from Matthew Wallis <mattw at madmonks.org> ----- > > > <Quoted messages SNIPPED> > > > As an example of a legitimate[1] case where a server not controlled by > > you sends mail on your behalf. > > > > Any one of a number of job advertising sites will allow you to use a > > form on their website to submit a job application. > > > > You fill it out with your email address, and upload your resume as an > > attachment, and the site sends it on to the potential employer, using > > your address as the from address, so that any correspondence from said > > employer goes back to the applicant. > > > > You then of course find that mail from that website is handled by a > > completely seperate mail server provided by the hosting company, so just > > whitelisting servers from that domain doesn't work, you have to check > > log files to see where the message really came from. > > > > It's this sort of "legitimate" abuse that made me give up on stopping > > spam, and switch to tag and bag with spamassassin. Show the users how to > > setup filters based on score, and let them deal with it. > > > > Greylisting would handle this reasonably effectively, but that assumes > > that the sending mailserver handles 450's appropriately, but last I > > checked places like Telstra had a buisness product based on a broken > > virus-scanning MTA product, so I don't hold out much hope of that. > > The solution to this is in the hands of the site that does the emailing for > you and it's for them to provide some sort of forwarding.
Hi, Neil.
Poster Matthew Wallis's point is not that 'the site that does the emailing for you' (e.g., a site that e-mails out your curriculum vita from its own Web servers, but bearing your claimed sending address in its headers) can't do forwarding in such a fashion that replies reach you correctly. Mr Wallis's point is that widespread deployment and checking of SPF records in sending domains' DNS would tend to make such mailings appear to be forged mail, because they will originate from what the SPF records assert is an 'unauthorised MX' for that domain.
Imagine that I signed up with www.resumes-are-us.com, provided them a copy of my curriculum vita, and then sat back and let them mail out that resume to various prospective employers. Each e-mail would bear headers claiming that it was from 'rick at linuxmafia.com'. Possibly even the 'envelope From', Return-Path, and Received headers might be forged by resumes-are-us.com's sending mail exchanger (outbound SMTP host).
The SPF record for domain linuxmafia.com says....
$ dig -t txt linuxmafia.com +short
"v=spf1 a mx -all"
That means: 'Please regard any domain from either the linuxmafia.com
"A" record host or any "MX" record host as originating from one of the
domain's authorised mail exchangers. Please consider this information
definitive, and thus please reject mail from any other mail exchangers.'
Thus, if some employer's receiving MTA is configured to check claimed-sender domains' SPF records (if any) during attempted incoming deliveries, and if the MTA is also willing to honour those records' recommendations, then the copy of my resume arriving from, say, mx1.resumes-are-us.com will get automatically 550-rejected.
Basically, Mr Wallis has conjured up an rather contrived scenario where he can say: 'See? You claim it's a good idea to specify which are the legitimate senders of your domain's outgoing mail, but here's an example of where doing so will tend to cause legitimate mail to get rejected, because it's coming from an unauthorised location.'
To phrase it another way: 'Making your domain's mail be sendable only from authorised servers is bad because it doesn't let you originate your domain's mail from unauthorised servers.'
Wallis is correct, except I don't want my domain's mail to be deliverable from unauthorised servers. I don't buy his fundamental assumption that that was ever a good idea at all. If a job-search site offers to send 'rick at linuxmafia.com' mail on my behalf from its own mail servers and not mine, my automatic response is 'Please don't'. If it does that without asking me first, and isn't able to succeed, then good! Saves me the trouble of LARTing them for forging my domain's mail.
Or, to be more concise, 'Stop the madness, Mr Wallis. What you claim that SPF "breaks" should _never be done_ in the first place.'
If you really intend for some particular third-party MX to send some of your domain's outbound mail, you would want, as should be obvious, to add it to your SPF record, saying 'Oh, and mx1.resumes-are-us.com is legitimate, too.' Wallis's rejoinder to that is: 'Suppose you don't know where Resumes Are Us sends its mail from? Or suppose it keeps changing?'
That just might be a clue that having random mail servers forge your outbound mail is just a _bad idea_? Not to Mr Wallis, but I certainly don't share his view on that. I think he's trying to perpetuate one of the very problems we'd like to utterly eliminate. Speaking of which:
> There are solutions for this sort of application that don't require > SPF to be ditched.
I have, if possible, somewhat less than zero patience for people attempting to halt progress because it might break some ill-conceived and problematic computing practice they've gotten away with until now.
Here's what I said off-list to Ben, the other day, as prefatory comments to the original Daniel Pittman post.
Forwarded because Daniel articulates the classic bullshit argument against SPF: It's 'Hey, I enjoy wandering the world with my laptop and blithely issuing my corporate domain's outbound mail from arbitrary IPs at each of my thousands of stayovers. Any attempt to convince my company to publish SPF RRs would mean that my latest transmissions from the beach on Phuket could get rejected as arriving from an unauthorised MX IP. Obviously, anything that inconveniences me even temporarily is Super Bad; ergo, I oppose this entire idea.'And, from a follow-up:Basically, even if universal publication of, and checking of SPF records took hold, and in the process, every mechanism for forwarding SMTP and issuing outbound SMTP from arbitrary, unplanned IPs instantly broke, I'd still say: 'Screw that. Sorry to hear about your collateral damage, bucko. I'm just happy that nobody's going to be able to convincingly forge my domain's mail, any more. What, you want to be able to keep outbound mail from .forward files, /etc/aliases files, and randomly meandering laptops working _too_? Cool. Good luck with that, chum. Let me know how it works out.'
There are of course other remedies other than SMTP AUTH, but that's the obvious answer to the Daniel Pittmans of the world. However, I'm disinclined to get into that with him, because it really misses the larger point, that he is the guy with a problem, and really should concentrate on coping rather than expecting the SMTP server admins of the world to eschew something that works for them just because it could conceivably inconvenience him. Besides, the whining is endless: If I say 'Don't be an idiot. Use SMTP AUTH on 587/tcp, or SSH tunnels, or plain old SSH, or webmail, or any of a dozen obvious remedies', he'd just come back with 'But what if that's not offered? What if it's inconvenient? What if I don't have the right software?' At a certain point, you just have to say 'It's a cruel world, son. Buy your own damned hanky.'
-- Cheers, Founding member of the Hyphenation Society, a grassroots-based, Rick Moen not-for-profit, locally-owned-and-operated, cooperatively-managed, rick at linuxmafia.com modern-American-English-usage-improvement association.
Benjamin A. Okopnik [ben at linuxgazette.net]
Thu, 31 Aug 2006 21:13:57 -0400
On Thu, Aug 31, 2006 at 02:48:28PM -0700, Rick Moen wrote:
> > Imagine that I signed up with www.resumes-are-us.com, provided them a > copy of my curriculum vita, and then sat back and let them mail out > that resume to various prospective employers. Each e-mail would bear > headers claiming that it was from 'rick at linuxmafia.com'. Possibly even > the 'envelope From', Return-Path, and Received headers might be forged > by resumes-are-us.com's sending mail exchanger (outbound SMTP host).
In my opinion, people who do that - i.e., propagate methods and systems that are screamingly-obvious spammer-enablers - should indeed fail in their attempt to make a buck and go out of business, leaving the way clear for those who use, and cooperate in using, more intelligent methods which limit or prevent the damage.
> Wallis is correct, except I don't want my domain's mail to be > deliverable from unauthorised servers. I don't buy his fundamental > assumption that that was ever a good idea at all. If a job-search site > offers to send 'rick at linuxmafia.com' mail on my behalf from its own > mail servers and not mine, my automatic response is 'Please don't'. > If it does that without asking me first, and isn't able to succeed, > then good! Saves me the trouble of LARTing them for forging my domain's > mail.
It seems to me that if I was going to run that sort of a site, I'd have a whole lot more leverage - both with employers and job-seekers - if I simply relayed the mail from one to the other (not open relay, obviously - only those who had signed up with me.) Why in the world would I ever pass up a chance to stick my business name in my customers' faces? The employer would send his mail to client54321 at TheBestJobSiteInTheWorld.com, which would alias out to 'rick at linuxmafia.com'; Rick would send his mail to employer12345 at TheBestJobSiteInTheWorld.com, which would get flipped to whatever the employer's real address is. Either that, or they can come to my site (whooppee, a chance to hit them with ads!) and read their mail there. No forgery in any of the cases, SPF-compliance all the way - the employer's server authenticates itself to me, mine authenticates to Rick's, and vice-versa for mail going the other way - and I get to do some free advertising along the way.
So, no - there's no reason to support a more-broken-than-usual business model. Better ones do exist, all the way from what I've described (call it the eBay method) to much better, slicker methods that don't require violating the chain of even nominal trust (if that word can be applied to SMTP at all.)
> I have, if possible, somewhat less than zero patience for people > attempting to halt progress because it might break some ill-conceived > and problematic computing practice they've gotten away with until now.
To quote myself responding to Rick's cited email,
What I see in that exchange, again, is "why can't we keep doing it the old way?" And the answer is, "because the world has changed while you weren't watching". Adding ", shithead" to the end of it is purely optional, and depends on your annoyance level from having to tell them something that is their own responsibility to know.The "--with-extra-hot-peppers" option is not one I would normally use - but it's available, and listed in the man page. I do get mightily annoyed at people pining and whining about how Things Aren't Like They Used To Be.
* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Kapil Hari Paranjape [kapil at imsc.res.in]
Fri, 1 Sep 2006 09:05:33 +0530
Hello,
On Thu, 31 Aug 2006, Rick Moen wrote:
> Quoting Neil Youngman (ny at youngman.org.uk): > > The solution to this is in the hands of the site that does the emailing for > > you and it's for them to provide some sort of forwarding. > > Hi, Neil. > > Poster Matthew Wallis's point is not that 'the site that does the > emailing for you' (e.g., a site that e-mails out your curriculum vita > from its own Web servers, but bearing your claimed sending address > in its headers) can't do forwarding in such a fashion that replies > reach you correctly. Mr Wallis's point is that widespread deployment and > checking of SPF records in sending domains' DNS would tend to make such > mailings appear to be forged mail, because they will originate from what > the SPF records assert is an 'unauthorised MX' for that domain.
I still don't see this as an objection to SPF. Let's look at a post-office scenario as a comparison.
<example> Lets say that a company that handles job applications on my behalf is at address A.
The Post Office insists that the company put its own address A on the Envelope in which it sends out these applications. This way the PO ensures that wrongly addressed mail has a legitimate "bounce" address.
The company is free to put my address B in the letter that it encloses inside the envelope on my behalf. Someone who wishes to appoint me based on my resume should respond to that address B.
All that the SPF record does is to tell the Post Office which persons are authorised to send mail out on behalf of the company. </example>
Web sites that prompt users to complete a form and use it to generate e-mail must take responsibility for the e-mail so generated---and the way for them to take responsibility is to put their own address in the Envelope From. I do not see any reason why such web sites consider it legitimate to use the address provided by the user as an Envelope From address.
The way I see it SPF says "If server X generates mail then the Envelope From address in that mail must be an address that X is authorised to send mail for".
The major problem that users will face as SPF becomes more prevalent is the "forwarding mechanism". SRS is proposed as one alternative but I think that using the forwarder's address in the Envelope From is also quite reasonable. This extends the principle that if you generate mail traffic then you take responsibility for it.
Regards,
Kapil. --
Neil Youngman [ny at youngman.org.uk]
Fri, 1 Sep 2006 08:18:02 +0100
On or around Thursday 31 August 2006 22:48, Rick Moen reorganised a bunch of electrons to form the message:
> Quoting Neil Youngman (ny at youngman.org.uk): > > > > The solution to this is in the hands of the site that does the emailing > > for you and it's for them to provide some sort of forwarding. > > Hi, Neil. > > Poster Matthew Wallis's point is not that 'the site that does the > emailing for you' (e.g., a site that e-mails out your curriculum vita > from its own Web servers, but bearing your claimed sending address > in its headers) can't do forwarding in such a fashion that replies > reach you correctly. Mr Wallis's point is that widespread deployment and > checking of SPF records in sending domains' DNS would tend to make such > mailings appear to be forged mail, because they will originate from what > the SPF records assert is an 'unauthorised MX' for that domain.
I guess my point is that I don't see a real problem, or at least not a significant problem. Their inability to forge email headers is not the underlying problem AFAIAC.
It is at most a presentational problem and I'm not sure that it's even that much. Kapil has pointed out that SPF only affects the SMTP envelope, so that the RFC 2822 From header, which most users see, can still appear to come from you. Anybody that looks deeper will be able to see that the message originated somewhere else, with or without SPF.
It seems to me that a suitable analogy for Matthew Wallis's argument would be "The forgery laws should be repealed, so that I can sign documents in my wife's/children's/client's name". Put like that, well it's so obviously sensible, isn't it?
<SNIP>
> Or, to be more concise, 'Stop the madness, Mr Wallis. What you claim > that SPF "breaks" should _never be done_ in the first place.'
I think we agree on that.
Neil
Rick Moen [rick at linuxmafia.com]
Fri, 1 Sep 2006 01:36:35 -0700
Quoting Neil Youngman (ny at youngman.org.uk):
> I guess my point is that I don't see a real problem, or at least not a > significant problem. Their inability to forge email headers is not the > underlying problem AFAIAC.
At the risk of neepery: Forgery isn't actually the issue, here, but rather apparent forgery. Matthew Wallis's concern is that programmatic SPF-checking will tend to classify as forged any mail originating from third-party, unofficially-outsourced MXes such as resume-forwarding companies. (I acknowledge that Wallis's scenario creates a problem for -him-, but maintain that his mail-handling mechanism is a bad one, and doomed anyway.)
> It is at most a presentational problem and I'm not sure that it's even > that much. Kapil has pointed out that SPF only affects the SMTP > envelope, so that the RFC 2822 From header, which most users see, can > still appear to come from you. Anybody that looks deeper will be able > to see that the message originated somewhere else, with or without > SPF.
Unfortunately, the 'envelope From' header (and its matching Return-Path header) is the vital part: Unlike almost all data within the SMTP envelope, it (or, more specifically, its IP information) cannot be forged, because it is created by the receiving SMTP host as an inherent part of the SMTP conversation. That is why SPF-checking at the MTA level uses that information (and ignores the RFC 2822 internal-to-envelope information) to determine whether the delivering MX is 'authorised' or not for the claimed sender domain.
(Spammers and malware authors tend to forge all possible SMTP tracking data, leaving only the 'envelope From' IP and certain of the Received header information, if you're very careful, as reliable data.)
> It seems to me that a suitable analogy for Matthew Wallis's argument would > be "The forgery laws should be repealed, so that I can sign documents in my > wife's/children's/client's name". Put like that, well it's so obviously > sensible, isn't it?
I would say it's more like: 'We can't afford to have the forgery laws enforced, because at random intervals I need to have other people legitimately sign documents in my/my wife's/children's/client's name, and my business model is such that I'll never possess an accurate list of who those authorised signers are.'
Which is not reasonable either.
Rick Moen [rick at linuxmafia.com]
Thu, 31 Aug 2006 03:59:01 -0700
Two more exchanges.
----- Forwarded message from Matthew Wallis <mattw at madmonks.org> -----
Date: Thu, 31 Aug 2006 15:51:58 +1000 To: luv-main at luv.asn.au From: Matthew Wallis <mattw@madmonks.org> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentOn Wed, Aug 30, 2006 at 10:25:05PM -0700, Rick Moen wrote:
> Quoting Matthew Wallis (mattw at madmonks.org): > > > As an example of a legitimate[1] case where a server not controlled by > > you sends mail on your behalf. > > > > Any one of a number of job advertising sites will allow you to use a > > form on their website to submit a job application. > > > > Alternatively, you can decide (as I did) that you'd rather -not- have > outside entities of unknown competence handling any of your domain's > outbound mail on your behalf. ;-> >
This is all wonderful for people sending _from your_ domain. The trick is for people sending _to you_ when you do a look up on their domain and find out that SPF for their domain doesn't allow this host, or spamassassin gives it a score of 900 because it's 100% html, and the very nature of the content just makes it all the more spammy.
This is the problem I ran into at a couple of my clients sites who were in the market for new employees, they advertised on a number of sites dedicated to finding people in the markets they were involved in, and more often than not, the applications sent in were rejected.
Now, as much as I might like, I can't fix other peoples mailservers, and on my own personal mailserver, where I'm not expecting to get a lot of job applications, I'm fine with this, but as far as my clients were concerned, this was stopping legitimate email from coming to them.
We can whitelist lots of sites, or add them to our SPF, but you're just diluting the effectiveness of SPF then, as you can't guarantee that those sites will always be legitimate.
Matt.
----- End forwarded message -----
----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Thu, 31 Aug 2006 03:31:40 -0700 To: luv-main at luv.asn.au From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentQuoting Matthew Wallis (mattw at madmonks.org):
> This is all wonderful for people sending _from your_ domain. The trick > is for people sending _to you_ when you do a look up on their domain > and find out that SPF for their domain doesn't allow this host, or > spamassassin gives it a score of 900 because it's 100% html, and the > very nature of the content just makes it all the more spammy.
I fear you may have greatly misread my post: I was saying that, if (like me) you decide you'd rather not have outside entities of unknown competence handling any of your domain's outbound mail on your behalf -- such as the mentioned "job advertising sites" -- then you would avoid using "a form on their Web site to submit a job application", and instead send all mail (e.g., curricum vitae copies) that purports to be from your domain _from your own MXes_.
> Now, as much as I might like, I can't fix other peoples mailservers....
Quite so. That's why I don't use them to send my domains' outbound mail. Thank you for making my point for me. ;->
> We can whitelist lots of sites, or add them to our SPF, but you're just > diluting the effectiveness of SPF then, as you can't guarantee that > those sites will always be legitimate.
No, you don't add "lots of sites" to your SPF records -- only those that will serve as your MXes. That's what the records are for. Nicht wahr?
-- Cheers, "Heedless of grammar, they all cried 'It's him!'" Rick Moen -- R.H. Barham, _Misadventure at Margate_ rick at linuxmafia.com----- Forwarded message from Daniel Pittman <daniel at rimspace.net> -----
From: Daniel Pittman <daniel@rimspace.net> To: TAG <tag@lists.linuxgazette.net> To: luv-main at luv.asn.au Date: Thu, 31 Aug 2006 15:33:53 +1000 Subject: Re: SRS developmentRick Moen <rick at linuxmafia.com> writes:
> Quoting Daniel Pittman (daniel at rimspace.net): > >> You are right: I missed a key clause, and it doesn't make sense as it >> stands. The clause in question was "from servers that you don't run." > > OK, imagine I'm right now sitting at a machine I don't run. I want to > send outbound SMTP mail with an originating domain of "linuxmafia.com". > I see several obvious ways, all of which work, and many of which my > users already use routinely:
[... various outbound methods ...]
> Note: I've mentioned all of these before.
Yup. Indeed, none of those are new to me.
> I'm aware you said "from servers that you don't run". Sorry, I > don't route my mail (or my users' mail) through servers I don't run.
Lucky you. Not everyone has that much control over their the routing of their users mail. At least, not today, and not until the SRS or SPF compatible policies are in place all over the world.
[...]
>> I wouldn't suggest doing nothing. I just wouldn't suggest SPF, as it >> represents a solution to the easy part of the "Joe Job" problem, and >> hasn't really addressed the hard parts. > > Maybe you're not getting the fact that having rendered Joe Job > forgeries no longer feasible matters. And my having done that > didn't damage anything for anyone -- not me, not my users, and not > anyone else.
Maybe not. I am happy to agree that I could be wrong about all this, or that my priorities may be misplaced. That is, in part, which I am carrying on this conversation -- if I never talk about it I will never learn if things have changed.
[...]
>> ...through SRS being deployed by every service provider we use, or >> something similar. > > I'll have to admit I'm curious about what you do that's so dependent > on /etc/aliases-style mail forwarding. Maybe you're solving the wrong > problem. Personally, I got away from all that stuff, years and years > ago!
Specifically: I have had a number of clients with delivery problems because they had SPF records and transparent SMTP proxies get in the way.
That can be solved by deploying some or all of the solutions you list above, but causes nasty, hidden problems -- since it isn't obvious until the bounce gets back to you (if it ever does) that the message was, in fact, not delivered correctly.[1]
I have also dealt with situations where staff from one organization work, possibly for extended periods, for another -- who have a security policy that (correctly, in my eyes) forbids client machines direct access to the Internet.
In which case they need to either send email originating from the host organization, relay through the host organization SMTP servers, or convince the MTA admins to add a specific mail routing rule to forward their messages back to our systems before they hit the Internet.
Well, those, and a few sites that block anything that lacks SPF records, but those are more an annoyance than anything -- you can't outlaw stupid people, and they would find some other stupidity if it wasn't misconfiguration SPF checks. (and no, I don't blame SPF for that ;)
Regards, Daniel
Footnotes: [1] Also, since this looks like a faked sender would /you/ send a bounce back to the (presumably) innocent third party telling them about it?
--
Digital Infrastructure Solutions -- making IT simple, stable and secure
Phone: 0401 155 707 email: contact at digital-infrastructure.com.au
http://digital-infrastructure.com.au/
----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Thu, 31 Aug 2006 03:54:42 -0700 To: luv-main at luv.asn.au From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentQuoting Daniel Pittman (daniel at rimspace.net):
> > I'll have to admit I'm curious about what you do that's so dependent > > on /etc/aliases-style mail forwarding. Maybe you're solving the wrong > > problem. Personally, I got away from all that stuff, years and years > > ago! > > Specifically: I have had a number of clients with delivery problems > because they had SPF records and transparent SMTP proxies get in the > way.
Well, they shouldn't be exposing their outbound mail to other people's SMTP proxies. ;-> What's happened to port 25 -- exactly the sort of thing you describe -- is predictable given decades of abuse, e.g., the day in 1997 when a certain spammer dumping tens of thousands of copies of the world's first "revenge spam" into ibm.net's PPP dial-up POPs in Chicago, going out completely without authentication or any other control to anti-spam activists' MXes (including yr. humble servant). This was the world's very first "Joe Job", named for ethical Web-hosting entrepreneur Joe Doll. Read the fascinating story here: http://web.archive.org/web/20001013162208/http://www.markwelch.com/yuri.htm
The most immediately obvious way for your clients to avoid transparent SMTP proxies and similar pitfalls would be sending such mail directly to the _real MX's_ 587/tcp port in the context of an SMTP AUTH session -- not attempting to use the much-abused, much-blocked, much-proxied, totally unauthenticated 25/tcp destination.
> That can be solved by deploying some or all of the solutions you list > above, but causes nasty, hidden problems -- since it isn't obvious until > the bounce gets back to you (if it ever does) that the message was, in > fact, not delivered correctly.[1]
Well, no, that's not the case. The SMTP AUTH / 587/tcp alternative, for example, is a direct handoff to the real MX. You can be much, much more certain of that than with delivery to 25/tcp, because specifically of the SMTP AUTH authentication step. Even if someone attempts to transparent-proxy port 587 (sort of a man-in-the-middle thing, I guess), which isn't very likely, you do know that your mail actually reached the real MX on the other side, because you completed that login handshake. I'd have to re-read the documentation on crypto options in SMTP AUTH: My vague recollection is that there is also provision for strong authentication of the destination host itself (some aspect of the TLS implementation), but I'd have to verify that.
It also should be obvious that the same high level of assurance is possible with the other non-25/tcp means I mentioned of moving mail to the real MX host, e.g., SSH forwarding, or webmail reached over SSL.
> I have also dealt with situations where staff from one organization > work, possibly for extended periods, for another -- who have a security > policy that (correctly, in my eyes) forbids client machines direct > access to the Internet.
There are indeed situations where someone just absolutely shoots you in the foot, no doubt about it. Hey, you might just have to say "Sorry, we can't do this." But I'm not sure why that prevents SPF from being useful, even at that. E.g., if your contractual entanglements require that a bunch of your domain's outbound mail issue from your customer's Internet gateway box, then add that box's IP to your domain's SPF record, right? Is there something I'm missing, here? That seems to me like a pretty simple problem, even though you pose it as a difficult (and infuriating) edge-case -- and I agree that it is that.
> In which case they need to either send email originating from the host > organization, relay through the host organization SMTP servers, or > convince the MTA admins to add a specific mail routing rule to forward > their messages back to our systems before they hit the Internet.
So? See above.
> Well, those, and a few sites that block anything that lacks SPF records, > but those are more an annoyance than anything -- you can't outlaw stupid > people, and they would find some other stupidity if it wasn't > misconfiguration SPF checks. (and no, I don't blame SPF for that ;)
Well said. ;->
----- End forwarded message -----
Rick Moen [rick at linuxmafia.com]
Thu, 31 Aug 2006 23:53:26 -0700
Matthew Wallis is very much the best of the "But progress breaks my business model!" people in this thread, so I'm trying to treat him with some respect.
----- Forwarded message from Matthew Wallis <mattw at madmonks.org> -----
Date: Fri, 1 Sep 2006 10:01:37 +1000 To: luv-main at luv.asn.au From: Matthew Wallis <mattw@madmonks.org> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentOn Thu, Aug 31, 2006 at 09:49:50PM +1000, Russell Coker wrote:
> On Thursday 31 August 2006 20:31, Rick Moen <rick at linuxmafia.com> wrote: > > > This is all wonderful for people sending _from your_ domain. The trick > > > is for people sending _to you_ when you do a look up on their domain > > > and find out that SPF for their domain doesn't allow this host, or > > > spamassassin gives it a score of 900 because it's 100% html, and the > > > very nature of the content just makes it all the more spammy. > > > > I fear you may have greatly misread my post: ?I was saying that, if > > (like me) you decide you'd rather not have outside entities of unknown > > competence handling any of your domain's outbound mail on your behalf -- > > such as the mentioned "job advertising sites" -- then you would avoid > > using "a form on their Web site to submit a job application", and > > instead send all mail (e.g., curricum vitae copies) that purports to be > > from your domain _from your own MXes_. > > It's best to just avoid using such services. Smart employers won't advertise > with agencies that run such broken systems and smart candidates are best > advised to avoid them too.
I think you're both possibly mis-interpreting my emails slightly, and as geeks sometimes will, focusing on too narrow a population group.
In the instances described, these job sites, were run by companies specialising in the pharmaceuticals industry, not IT. They don't know who handles their mail, nor do they particularly care, they do not understand that their system is flawed "What do you mean it doesn't work? It sends mail doesn't it?"
Also the people who use these sites, similarly do not care how their mail is sent, just as long as it is recieved. These are all professionals within their fields of endevour, and these sites are specifically setup so that people in the pharmaceuticals industry can find eachother.
Not use these sites? That would be like farmers not trying to sell their produce to the citys. Sure they'll get a few sales off eachother, but the citys will buy the whole damned lot and come back for more.
For me to ask the company not to use them, would be like asking them not to take crops from farmers who have blue heelers. Rejecting mail from there would not improve the situation, it would just reduce the number of applications recieved.
"How are we meant to hire anyone if you're rejecting their mail?"
Now if we can get the people who run the sites to care, we might get somewhere. I'm gathering that the reason for all this hackery is due to the site getting paid per referral, if they weren't, then they'd just put a mailto link up with a "Send your resume here."
Matt.
----- End forwarded message -----
----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Thu, 31 Aug 2006 23:49:33 -0700 To: luv-main at luv.asn.au From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentQuoting Matthew Wallis (mattw at madmonks.org):
> I think you're both possibly mis-interpreting my emails slightly, and as > geeks sometimes will, focusing on too narrow a population group.
Mea culpa, sir.
> In the instances described, these job sites, were run by companies > specialising in the pharmaceuticals industry, not IT. They don't know > who handles their mail, nor do they particularly care, they do not > understand that their system is flawed "What do you mean it doesn't > work? It sends mail doesn't it?" > > Also the people who use these sites, similarly do not care how their > mail is sent, just as long as it is recieved. These are all > professionals within their fields of endevour, and these sites are > specifically setup so that people in the pharmaceuticals industry can > find eachother.
To try to summarise: You've described a situation where your domain has business reasons to offload some significant fraction of its outbound SMTP to third-party firms' mail exchangers that themselves may be outsourced, or at least be on MXes that are non-identifiable within reason by you. Therefore, you're saying that publishing SPF records (or, by implication, any of the other schemes to undermine mail-forgery by validating "envelope From" and Return-Path) is against your interests because the third-party-sourced mail will appear to be forged.
I agree. Obviously, if (for any reason) you can't specify which machines are your authorised MXes, then SPF et al. don't work -- because those fixes to SMTP and DNS are _based on_ specifying your authorised MXes.
(IMPORTANT NOTE: As a reminder, MTA implementations of SPF-checking come equipped with whitelists of known legitimate third-party forwarders that are not yet SPF-friendly. I've said this before. I don't know if you heeded my saying it the first time, so I'm repeating myself.)
In the short term, I'd concentrate on confining that problem situation to a subdomain -- or use, e.g., GMail accounts, thereby making it Somebody Else's Problem. In the longer term, I'd try hard to eliminate that entire problem domain.
Why? Because I think it's going the way of the dodo. I was fed up with joe-jobbing within about a week of its invention as a spammer's revenge ploy against Joe Doll at the beginning of 1997. So are basically the entire technically-aware fraction of the world's mail-admin community. Increasingly, if your mail comes across as empiracally dodgy for any of numerous reasons including gross failures of RFC-compliance or coming from an implausible MX, you're going to find it rejected or (worse) silently dropped on the floor.
That sir, is becoming a business issue likely to make the "How are we meant to hire anyone if you're rejecting their mail?" issue you speak of look really small by comparison.
In the early 1990s, there were entire segments of business going around saying "Eliminate open relays? We rely on them!" Well, they did. But then, they adapted, or they died.
I'd not advise domain admins to publish SPF records (_or_ any or all of the alternatives) in the few edge cases where inability to specify one's MXes makes that infeasible and the external forwarders are not covered by the MX whitelists or easily add-able. Instead, I'd tell them "Sucks to be you. Good luck fixing your situation over the longer term."
----- End forwarded message -----
Neil Youngman [ny at youngman.org.uk]
Fri, 1 Sep 2006 08:47:19 +0100
On or around Friday 01 September 2006 07:53, Rick Moen reorganised a bunch of electrons to form the message:
> Matthew Wallis is very much the best of the "But progress breaks my > business model!" people in this thread, so I'm trying to treat him with > some respect. > > ----- Forwarded message from Matthew Wallis <mattw at madmonks.org> ----- > > Date: Fri, 1 Sep 2006 10:01:37 +1000 > To: luv-main at luv.asn.au > From: Matthew Wallis <mattw at madmonks.org> > Subject: Re: SRS development > > In the instances described, these job sites, were run by companies > specialising in the pharmaceuticals industry, not IT. They don't know > who handles their mail, nor do they particularly care, they do not > understand that their system is flawed "What do you mean it doesn't > work? It sends mail doesn't it?"
Another analogy "They do not understand that their health and safety system is flawed. What do you mean there's a problem? The work gets done doesn't it"
They don't tell the government that they won't implement new laws, because they have to change their working practices, do they
> Also the people who use these sites, similarly do not care how their > mail is sent, just as long as it is recieved. These are all > professionals within their fields of endevour, and these sites are > specifically setup so that people in the pharmaceuticals industry can > find eachother.
If they're "professionals within their fields of endevour", then they should understand the value of hiring somebody competent within our field and getting the job done right.
Also, if they care that their mail is received, then stopping receiving it is an incentive to fix it.
> Not use these sites? That would be like farmers not trying to sell their > produce to the citys. Sure they'll get a few sales off eachother, but > the citys will buy the whole damned lot and come back for more. > > For me to ask the company not to use them, would be like asking them not > to take crops from farmers who have blue heelers. Rejecting mail from > there would not improve the situation, it would just reduce the number > of applications recieved.
If the company really must receive mail from these people, then set up a rule exempting certain servers form the SPF checks. Don't expect the rest of the world to give up SPF because they want to be a special case. Come to that, if you turn off SPF checking entirely, it doesn't hurt anyone else, but saying we shouldn't use it because you don't like it, doesn't cut any ice with me.
> "How are we meant to hire anyone if you're rejecting their mail?" > > Now if we can get the people who run the sites to care, we might get > somewhere. I'm gathering that the reason for all this hackery is due to > the site getting paid per referral, if they weren't, then they'd just > put a mailto link up with a "Send your resume here."
And how exactly does forging the return address make it easier for them to get paid per referral?
Neil
Rick Moen [rick at linuxmafia.com]
Fri, 1 Sep 2006 03:16:25 -0700
As we see here, technological dinosaurs are sometimes quite insistent that their inability to cope with change is somehow Somebody Else's Problem.
----- Forwarded message from Matthew Wallis <mattw at madmonks.org> -----
Date: Fri, 1 Sep 2006 17:15:03 +1000 To: luv-main at luv.asn.au From: Matthew Wallis <mattw@madmonks.org> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentOn Thu, Aug 31, 2006 at 11:49:33PM -0700, Rick Moen wrote:
> Quoting Matthew Wallis (mattw at madmonks.org): > > > I think you're both possibly mis-interpreting my emails slightly, and as > > geeks sometimes will, focusing on too narrow a population group. > > Mea culpa, sir. > > > In the instances described, these job sites, were run by companies > > specialising in the pharmaceuticals industry, not IT. They don't know > > who handles their mail, nor do they particularly care, they do not > > understand that their system is flawed "What do you mean it doesn't > > work? It sends mail doesn't it?" > > > > Also the people who use these sites, similarly do not care how their > > mail is sent, just as long as it is recieved. These are all > > professionals within their fields of endevour, and these sites are > > specifically setup so that people in the pharmaceuticals industry can > > find eachother. > > To try to summarise: You've described a situation where your domain > has business reasons to offload some significant fraction of its > outbound SMTP to third-party firms' mail exchangers that themselves may > be outsourced, or at least be on MXes that are non-identifiable within > reason by you. Therefore, you're saying that publishing SPF records > (or, by implication, any of the other schemes to undermine mail-forgery > by validating "envelope From" and Return-Path) is against your interests > because the third-party-sourced mail will appear to be forged.
Sorry, but you've still got this somewhat arse about. My domain was not offloading anything. We were simply trying to make use of someone elses service.
10,000 independant people with email addresses ranging from ISP addresses to free webmail based addresses to god knows what, have one thing in common, they desire to work in the pharmaceuticals industry.
Pharmaceuticaljobs.com lists jobs for these people, my company puts up an advertisement on pharmaceuticaljobs.com.
Pharmaceuticalsjobs.com get's paid for it's referrals, 10,000 people log into their site, find our job, upload their resume, and have it forwarded on to us, with the users email address as the From: address. That email could be from anywhere, hotmail, aol, internode, mac.com.
We look up the from address, find out that their ISP has SPF, and that pharmaceuticalsjobs.com is not listed in their SPF, so we ditch it.
Using gmail would not solve this issue, unless google put pharmaceuticalsjobs.com in thier SPF. Or are you saying that we should get the applications forwarded to gmail? And if gmail starts using SPF?
Matt.
----- End forwarded message -----
----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Fri, 1 Sep 2006 03:13:25 -0700 To: luv-main at luv.asn.au From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentQuoting Matthew Wallis (mattw at madmonks.org):
> Sorry, but you've still got this somewhat arse about. My domain was not > offloading anything. We were simply trying to make use of someone elses > service.
Not intending to sound belligerent, but that is exactly what I referred to as "offloading": You're outsourcing (a portion of) your outbound SMTP traffic to third-parties' MX hosts whose identity (per your account, earlier) you don't even know.
In the long term, that is going to prove an extremely bad idea, for reasons cited, among others.
I'm sorry you are likely to suffer adverse consequences in the medium term from your reliance on an ill-conceived model of transmitting mail. People who relied on open relays had a similar problem. And yet we moved past that. Thus my point.
> Using gmail would not solve this issue, unless google put > pharmaceuticalsjobs.com in thier SPF.
Er, you are mistaken.
$ dig -t txt google.com +short "v=spf1 ptr ?all"
That "?" means that google.com's SPF RR (which covers GMail) is just messing around with the idea of genuinely implementing it, and that its information should not be relied upon.
----- End forwarded message -----
Rick Moen [rick at linuxmafia.com]
Fri, 1 Sep 2006 03:45:24 -0700
Chris points out that there may have been an understanding about whose mail was being handed off to third parties. Either way, it's a bad idea, and an accident waiting to happen.
----- Forwarded message from Chris Samuel <csamuel at vpac.org> -----
From: Chris Samuel <csamuel@vpac.org> To: TAG <tag@lists.linuxgazette.net>Organization: VPAC - The Victorian Partnership for Advanced Computing
To: luv-main at luv.asn.au Date: Fri, 1 Sep 2006 20:19:58 +1000 Subject: Re: SRS developmentOn Friday 01 September 2006 8:13 pm, Rick Moen wrote:
> Not intending to sound belligerent, but that is exactly what I > referred to as "offloading": ?You're outsourcing (a portion of) your > outbound SMTP traffic to third-parties' MX hosts whose identity (per > your account, earlier) you don't even know. ?
No, the people who are submitting their resumes to the company he works for are (unknowingly) outsourcing their outgoing email.
His system is just the receiver.
My guess is that this is one of the reasons that an SPF_FAIL in SpamAssassin carries a relatively low score (around 1.3 or 1.4) and SPF_PASS carries virtually none (-0.01).
cheers! Chris
-- Christopher Samuel - (03)9925 4751 - VPAC Deputy Systems Manager Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Fri, 1 Sep 2006 03:39:00 -0700 To: luv-main at luv.asn.au From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentQuoting Chris Samuel (csamuel at vpac.org):
> No, the people who are submitting their resumes to the company he works for > are (unknowingly) outsourcing their outgoing email. > > His system is just the receiver.
{scraches head}
In that case, I really have to wonder why he's saying (or at least implying) that publishing SPF RRs is a bad idea in his usage model, since that would seem not especially relevant to his problem.
Resume submitters (people other than Matthew Wallis) who both publish SPF RRs for their domains saying "Don't accept my domain's mail unless it comes from this list of IPs" and simultaneously also use resume-forwarding companies to send a bunch of their domains' mail from different IPs entirely -- well, those folks I'd call somewhat unclear on what the frell they're doing. ;->
(Of course, I suppose the intended scenarios is: 1) Domain admin for company example.com implements SPF on the basis of the known set of authorised mail exchangers. 2) User, unaware of the actions of company.com's domain admin, then attempts to use a resume-forwarding service -- finding out the hard way that his resumes are getting rejected by recipient MTAs on grounds of delivery from an "unauthorised MX".)
I will suggest -- yet again -- that that rather wild mode of mail-handling is destined to go the way of the dodo. However, one of my colleagues at _Linux Gazette_, editor-in-chief Ben Okopnik, had some interesting observations that may also be relevant:
In my opinion, people who do that - i.e., propagate methods and systems that are screamingly-obvious spammer-enablers - should indeed fail in their attempt to make a buck and go out of business, leaving the way clear for those who use, and cooperate in using, more intelligent methods which limit or prevent the damage. [...] It seems to me that if I was going to run that sort of a site, I'd have a whole lot more leverage - both with employers and job-seekers - if I simply relayed the mail from one to the other (not open relay, obviously - only those who had signed up with me.) Why in the world would I ever pass up a chance to stick my business name in my customers' faces? The employer would send his mail to client54321 at TheBestJobSiteInTheWorld.com, which would alias out to 'rick at linuxmafia.com'; Rick would send his mail to employer12345 at TheBestJobSiteInTheWorld.com, which would get flipped to whatever the employer's real address is. Either that, or they can come to my site (whooppee, a chance to hit them with ads!) and read their mail there. No forgery in any of the cases, SPF-compliance all the way - the employer's server authenticates itself to me, mine authenticates to Rick's, and vice-versa for mail going the other way - and I get to do some free advertising along the way. So, no - there's no reason to support a more-broken-than-usual business model. Better ones do exist, all the way from what I've described (call it the eBay method) to much better, slicker methods that don't require violating the chain of even nominal trust (if that word can be applied to SMTP at all.)
> My guess is that this is one of the reasons that an SPF_FAIL in SpamAssassin > carries a relatively low score (around 1.3 or 1.4) and SPF_PASS carries > virtually none (-0.01).
A better reason is that the correct place to do SPF-checking is in MTA ACLs, not SpamAssassin.
----- End forwarded message -----
Rick Moen [rick at linuxmafia.com]
Fri, 1 Sep 2006 15:56:15 -0700
Three more.
----- Forwarded message from Chris Samuel <csamuel at vpac.org> -----
From: Chris Samuel <csamuel@vpac.org> To: TAG <tag@lists.linuxgazette.net>Organization: VPAC - The Victorian Partnership for Advanced Computing
To: luv-main at luv.asn.au Date: Fri, 1 Sep 2006 21:03:04 +1000 Subject: Re: SRS developmentOn Friday 01 September 2006 8:39 pm, Rick Moen wrote:
> Quoting Chris Samuel (csamuel at vpac.org):
> > No, the people who are submitting their resumes to the company he works
> > for are (unknowingly) outsourcing their outgoing email.
> >
> > His system is just the receiver.
>
> {scraches head}
>
> In that case, I really have to wonder why he's saying (or at least
> implying) that publishing SPF RRs is a bad idea in his usage model,
> since that would seem not especially relevant to his problem.
Because his boss is making it his problem would be my guess. 
Probably what's happening is that his boss sees that their mail server is rejecting the email when other peoples don't and so of course it must be theirs that is broken otherwise it wouldn't be rejecting them. They need to get the resumes so they must "fix" their mail server not to reject them.
We all know that doesn't necessarily make logical sense from our techie point of view, but when the people who are paying you are screaming at you to fix it or else then you fix it (if you're attached to getting paid).
> > My guess is that this is one of the reasons that an SPF_FAIL in > > SpamAssassin carries a relatively low score (around 1.3 or 1.4) and > > SPF_PASS carries virtually none (-0.01). > > A better reason is that the correct place to do SPF-checking is in MTA > ACLs, not SpamAssassin.
Same thing with amavisd-new and Postfix's content_filter which is what I use
everywhere.. 
-- Christopher Samuel - (03)9925 4751 - VPAC Deputy Systems Manager Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia----- Forwarded message from Russell Coker <russell at coker.com.au> -----
From: Russell Coker <russell@coker.com.au> To: TAG <tag@lists.linuxgazette.net>Reply-To: russell at coker.com.au
To: luv-main at luv.asn.au Date: Fri, 1 Sep 2006 23:02:37 +1000Cc: Matthew Wallis <mattw at madmonks.org>
Subject: Re: SRS developmentOn Friday 01 September 2006 17:15, Matthew Wallis <mattw at madmonks.org> wrote:
> Sorry, but you've still got this somewhat arse about. My domain was not > offloading anything. We were simply trying to make use of someone elses > service. > > 10,000 independant people with email addresses ranging from ISP > addresses to free webmail based addresses to god knows what, have one > thing in common, they desire to work in the pharmaceuticals industry. > > Pharmaceuticaljobs.com lists jobs for these people, my company puts up > an advertisement on pharmaceuticaljobs.com. > > Pharmaceuticalsjobs.com get's paid for it's referrals, 10,000 people log > into their site, find our job, upload their resume, and have it > forwarded on to us, with the users email address as the From: address. > That email could be from anywhere, hotmail, aol, internode, mac.com.
Pharmaceuticalsjobs.com presumably has a small number of IP address for it's mail servers. It should not be difficult to discover all of them and have them all excluded from SPF checks. If the Pharmaceuticalsjobs.com people have even half a brain they will have added a header that can be used to uniquely identify mail from them which they will add to all future email, this may be something as simple as "Pharmaceuticalsjobs.com CV" in the message subject. This can also be used as a key to turn off SPF.
Another option is to hassle Pharmaceuticalsjobs.com about this every time you do business with them. They should listen to the people who pay them. Tell them that if they don't you will offer some friendly advice to the seek.com and jobserve.com people about how to best meet the needs of the pharmaceutical industry.
-- http://etbe.blogspot.com/ My Blog http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages----- End forwarded message -----
----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Fri, 1 Sep 2006 15:51:47 -0700 To: luv-main at luv.asn.au From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentQuoting Russell Coker (russell at coker.com.au):
> Pharmaceuticalsjobs.com presumably has a small number of IP address for it's > mail servers. It should not be difficult to discover all of them and have > them all excluded from SPF checks.
In hopes of being useful: I think this is the forwarders whitelist Meng Wong mentioned in one of the pieces I read at some point: http://trusted-forwarder.org/ Please note the item "How do I request a mail server to be listed on the T-FWL?"
(This is the compensating-for-noncompliant-forwarders mechanism that I've mentioned twice in this thread, but posters seem to keep ignoring.)
----- End forwarded message -----
Rick Moen [rick at linuxmafia.com]
Fri, 1 Sep 2006 18:14:16 -0700
Can you say 'special pleading'?
----- Forwarded message from Daniel Pittman <daniel at rimspace.net> -----
From: Daniel Pittman <daniel@rimspace.net> To: TAG <tag@lists.linuxgazette.net> To: luv-main at luv.asn.au Date: Sat, 02 Sep 2006 10:43:28 +1000 Subject: Re: SRS developmentRick Moen <rick at linuxmafia.com> writes:
> Quoting Chris Samuel (csamuel at vpac.org):
[...]
>> My guess is that this is one of the reasons that an SPF_FAIL in >> SpamAssassin carries a relatively low score (around 1.3 or 1.4) and >> SPF_PASS carries virtually none (-0.01).
SPF_PASS carries almost no score because it is manually set low to avoid giving senders of SPAM (who were also enthusiastic early adopters of the technology) a boost in the score.
This is correct, as an SPF pass doesn't indicate anything about the desirability of the message that SpamAssassin is considering, only the authenticity of the sender.
Spammy senders who are legitimate should be caught my the other checks, especially as the sender authentication should (in theory) make blacklisting effective.
> A better reason is that the correct place to do SPF-checking is in MTA > ACLs, not SpamAssassin.
Unless this is an (unstated) argument that the implementation of SPF checking in SpamAssassin in poorly implemented, this is a very weak argument.
(Casting aside the argument that, as in my systems, SpamAssassin is part of the MTA, not a distinct non-MTA entity.)
The only advantage I can see of using an MTA ACL would be that it is a black/white decision with no shades of grey[1] which SpamAssassin considers SPF FAIL an indicator (and a poor one at that) of anything untoward.
Have I missed some key reason that the MTA is any better, or different, to SpamAssassin? Is this an unstated claim that SpamAssassin has a faulty implementation of SPF?
Personally, I see the low ranking of SPF by the machine learning system SpamAssassin uses as reflective of reality: publishing SPF records doesn't hurt anything[2] but testing them does, because the world isn't "SPF compliant" yet.
I also consider the SpamAssassin model superior to an MTA ACL implementation, because it reflects reality: in some cases other considerations should override the technical issues and pass the message through even in the face of "non compliant senders."
Regards, Daniel
Footnotes: [1] Well, except that the SPF implementation includes a list of forwarders who paid enough to protect their "broken" business model ^W^W^W^W^W^W^W^W are important enough to whitelist.
[2] Immediately, or other than by encouraging SPF deployment.
--
Digital Infrastructure Solutions -- making IT simple, stable and secure
Phone: 0401 155 707 email: contact at digital-infrastructure.com.au
http://digital-infrastructure.com.au/
----- Forwarded message from Rick Moen <rick at linuxmafia.com> -----
Date: Fri, 1 Sep 2006 18:11:23 -0700 To: luv-main at luv.asn.au From: Rick Moen <rick@linuxmafia.com> To: TAG <tag@lists.linuxgazette.net> Subject: Re: SRS developmentQuoting Daniel Pittman (daniel at rimspace.net):
> > A better reason is that the correct place to do SPF-checking is in MTA > > ACLs, not SpamAssassin. > > Unless this is an (unstated) argument that the implementation of SPF > checking in SpamAssassin in poorly implemented, this is a very weak > argument. > > (Casting aside the argument that, as in my systems, SpamAssassin is > part of the MTA, not a distinct non-MTA entity.)
Apologies for being unclear. Most people, when they speak of SpamAssassin, are contemplating its use post-SMTP-acceptance, e.g., at the level of the MDA/LDA. I have sympathy for people who need to do spam-cleanup post-MTA for many reasons, including the fact that it doesn't work very well. One is lacking in some of the data and capabilities available uniquely during the SMTP delivery attempt.
E.g., you know exactly which IP address you're speaking to, because you're exchanging packets on a live TCP socket in real-time, and thus don't have to worry about whether you'll be sending back your DSN (delivery status notification, the result code and error text from your side of the SMTP conversation) to the correct host. You can rest assured that your '250 Accepted' or '550 Die, spammer, die' DSN will be conveyed to the originating host rather than some innocent host whose headers are being forged: Your system will not be guilty of generating backscatter scam with those 550s.
Of course, as you note, it is eminently possible with many MTAs to integrate SA into this step as part of the process of deciding whether to say (e.g.) 250 or 550. My own system has done this for years, courtesy of my friend Marc Merlin's 'sa-exim' modification to Exim4. And of course I've tweaked for local standards what SA does with the results it gets when it queries the running spfd process.
As long as the implementation of SPF-checking is done at SMTP time, before the receiving MTA says 'Sure, I'll take that', it really doesn't make a huge amount of difference what specific modules does it.
> publishing SPF records doesn't hurt anything[2] but testing them does, > because the world isn't "SPF compliant" yet.
Well, that depends critically on what you mean by 'testing them', doesn't it?
Sorry, but I'm not going to allow that vague handwave slip by. Testing them in a dumb fashion hurts things; testing them intelligently hurts, in my considered view, only what _really badly needs_ to be hurt.
Just to be clear, I'll say that again: Some modes of handling SMTP mail are very bad ideas, and I cheer wildly when somebody hurts them. Forwarding through unplanned, unknown third-party MXes without rewriting the envelopes is, in particular, just bad and must die. As a friend of Papa Darwin, I'm delighted to assist in killing it.
Did I mention my friend Steven "dillo" Okay's t-shirt, that he got printed up here in Silicon Valley at the worst part of the dot-dom economic depression? 'Sorry to Hear About Your Business Model.' A similar sentiment applies here.
> I also consider the SpamAssassin model superior to an MTA ACL > implementation, because it reflects reality: in some cases other > considerations should override the technical issues and pass the > message through even in the face of "non compliant senders."
I've now mentioned the whitelist for trusted forwarders three times, now. Are you just not getting, or having selective vision problems, or what?
(Although, frankly, I'd personally rather see them killed off, as noted.)
----- End forwarded message -----