Friday, December 9th, 2005

postfix+clamscan is eating my mail

When I went to check my mail this morning, there was a lot less than I thought there should be. That is, there was no new email, not even spam. This never happens, because I get so much spam and list mail that there's always something there.

I did some investigation and found that postfix was running everything through clamav as configured, but clamav was rejecting everything as having a virus. Even text messages with no attachment!

When I integrated clamscan into postfix, I used a postfix after-queue content filter script. That example shows filter processing as:

filter <in.$$

That redirects the temporary file in.$$ to the filter's stdin for processing. So my script reads:

/usr/local/bin/clamscan -d /var/lib/clamav <in.$$

Looks ok, right? Well, clamscan doesn't support redirecting content into its stdin. Instead, when run without arguments it ignores stdin and processes everything in the current directory. Since the current directory is the temporary filter queue directory, no problem. (Well almost; there is also a race condition if two scans are running simultaneously and one is a virus - the other will be thought to be a virus too.)

Upon further investigation, I found that sometime last evening clamscan had crashed and left a clamscan.core file in the filter working directory. The problem here is that the clamscan core file is itself identified as a virus! While clamscan is running, it contains virus signatures in memory which are then written to disk if it dumps core.

I fixed this problem by changing my clamscan line to:

/usr/local/bin/clamscan -d /var/lib/clamav in.$$

A subtle change but it makes all the difference.

(I don't know why clamscan dumped core. At least if it happens again, it won't cause me to lose 11 hours of email.)
(4 comments | Leave a comment)

Thursday, November 10th, 2005

interesting email surprise

I received the following message yesterday, and it almost had me fooled:

From: "Jamie" <jamie.andrews@totalbusiness.co.uk>
To: <greg@hewgill.com>
Subject: Photo Approval
Date: Thu, 10 Nov 2005 03:31:07 -0600

Hello,
Your photograph was forwarded to us as part of an article we are publishing for
our December edition of Total Business Monthly.
Can you check over the format and get back to us with your approval or any
changes?
If the picture is not to your liking then please send a preferred one.
We have attached the photo with the article here.
Kind regards,
Jamie Andrews
Editor
www.TotalBusiness.co.uk
**********************************************
The Professional Development Institute
**********************************************

I had received a legitimate message a few days ago from somebody in Germany asking to use one of my pictures in an architectural publication of some kind (I gave them permission). Initially I thought this might be related. Then I noticed the attachment:

[-- Attachment #2: Photo+Article.exe --]
[-- Type: application/octet-stream, Encoding: base64, Size: 13K --]

It's an email virus! Luckily I handle my mail using mutt on freebsd, so it would have been several more steps for me to "open" the attachment, so I didn't run it. It's amazing the lengths to which these virus authors are going to try to trap people.

In related news, I'm still getting fake-rolex spam which spamassassin is still marking as BAYES_50 (which means spamassassin thinks it's middle of the road, no sign of spam). Every one of those damn things I run through sa-learn to teach it, but the people who write those have found that the english language is extremely flexible and there are a million ways to package their sales pitch.

Spam is starting to become overwhelming again. Whatever happened to SPF, anyway?

(14 comments | Leave a comment)

Friday, November 5th, 2004

gmail thinks I'm a spammer

I've been using gmail for a while now. I have set up procmail rules that forward all my email to my gmail account for easier reading (I like the way gmail lets me read mailing lists and such). What I've noticed for a while now, is that whenever I make a post to a mailing list, my own message ends up in my gmail Spam folder. Every time this happens, I mark it as "Not spam" and move it back out. It doesn't help. I tried some test messages today to my gmail account and other gmail accounts, and each time my email was marked as spam.

Unfortunately, gmail does not provide a way for me to determine why a given message is marked as spam. But I have my own theory...

Recent generations of email worms (such as Netsky and Bagle) send worm messages using email addresses found on the infected computer. These email engines scan through the computer's hard drive looking for various types of files that contain email addresses, and send email worm messages using those addresses in the To and From field. I get a lot of these messages, usually about a thousand per day (slightly less on weekends). Assuming the worms randomly select addresses, I can assume that at least an equivalent number of worm messages are also sent "From" my email address.

The next question is why does my email address appear on so many computers around the world? I believe the answer lies with VNC. Many years ago I contributed some code to the VNC project, and the VNC authors acknowledged my contribution by including my email address in the VNC "history.txt" file. Since VNC is a very popular program, my email address appears in a .txt file on an unknown number of computers on the Internet, where it can be easily picked up and used by email worms.

I have done some analysis of email worm messages I receive, and a significant number of them (around 20%, I don't remember the exact number) have addresses in the From field that also come from VNC-related files. This lends support to my theory.

Connecting the dots, it seems that gmail's systemwide spam filter has identified <greg@hewgill.com> as a sender whose messages should always be marked as spam. It's quite disappointing for me to reach this conclusion.

I have submitted a problem report to gmail, asking whether they can shed any light on this issue. The ideal solution would be to remove my name from their global address filter list, and instead let the filtering happen by content (gmail rejects worm-infected email). The worst case solution is they ignore the problem or just tell me to change my email address (which I don't consider an acceptable solution).

Another disheartening implication here is that other email providers, large or small, might also automatically consider my email messages spam for the same reasons. This would not make me very happy.

(2 comments | Leave a comment)

Wednesday, June 2nd, 2004

poor man's srs

I've been getting an incredible amount of junk bounce messages from email worms. These usually come from the worm sending a message with my email in the From field and a bogus address in the To field. The receiving mail server dutifully returns the message to me, as if I had sent it. I quickly got tired of manually inspecting and deleting these, so I set up a procmail rule to route them to a 'bounce' folder:

* ^Return-Path: (<>|MAILER-DAEMON@)

After setting this up I never really looked in that folder, so I would miss the possibility of receiving a legitimate bounce message (for example, if a message I actually sent was undeliverable for some reason). After reading about Sender Rewriting Scheme, I figured I could rig up something similar that would help (in a simple way, without implementing full SRS).

I changed my .muttrc such that the Envelope-From on outgoing mail is now greg-foo@hewgill.com:

set sendmail="/usr/lib/sendmail -fgreg-foo@hewgill.com"

I actually used a different word than foo above but I'm not going to mention it here. If an email worm were to pick up that address off this page, this whole scheme would be defeated. When setting this up you can use any word you want.

Then I changed my procmail rule to also check bounce messages addressed to my normal address:

* ^Return-Path: (<>|MAILER-DAEMON@)
* ^TO_greg@hewgill.com

In this way, only fake bounce messages that are addressed to my normal address (and are thus worm-generated) get shuffled off to the 'bounce' folder. Any bounce message that I legitimately receive as a result of a message I actually sent, will be addressed to greg-foo@hewgill.com instead, and will not be matched by the above rule. So, they will end up in my normal inbox as they should.

Initial tests show that this is working great. If for some reason a worm discovers the SRS-style Envelope-From address I am using and starts sending me fake bounce messages to that address, I can easily change it to something else.

Fighting spam and worm mail is taking up a nontrivial amount of my time these days, hopefully this will help reduce that time.

(7 comments | Leave a comment)

Monday, December 1st, 2003

qmail may be eating your email

The other day I placed an order online, and entered my email address to get a confirmation message. It was supposed to be sent right away after placing the order, but it didn't arrive. However, since I have been watching my mail server logs more closely recently (due to greylisting), I noticed that the company's mail server had attempted to deliver my confirmation message several times unsuccessfully. I found that this had nothing to do with greylisting rejecting the message, it was being passed straight through to qmail.

The qmail logs were showing an incoming connection but no message was logged as being delivered. Some sleuthing around with the help of davehart, including writing a SMTP shim that logged the precise contents of the mail delivery transaction, revealed the problem. Qmail was rejecting the message with an error message: "451 See http://pobox.com/~djb/docs/smtplf.html."

It turns out that the sending software was trying to send a message that was composed with only LF characters at the end of each line (normally email is sent with both CR and LF line ending characters). Qmail considers this illegal, and when it detects such characters it responds with the above error message and immediately drops the connection. This causes the sending server to save the undelivered message and try again later.

The qmail document refers to a document it calls "822bis", which apparently was an older name for RFC 2822. Section 2.3 of this document does indeed prohibit the use of bare LF characters within the message body. However, this document is not considered a standard; to find a standard relating to email transfer we must look at RFC 822, also known as STD 11. RFC 822 does not specifically prohibit the use of bare LF characters in message bodies.

RFC 821 also requires, in section 4.1.1, that "The receiver should not close the transmission channel until it receives and replies to a QUIT command (even if there was an error)." Based on this, I have concluded that the qmail behaviour is in error and should not be used as is.

The culprit in qmail is the function blast() in the qmail-smtpd.c file. This function scans the message body and if it detects a bare LF character, calls the straynewline() function. This function prints the above 451 message, and abruptly exits:
void straynewline() { out("451 See http://pobox.com/~djb/docs/smtplf.html.\r\n"); flush(); _exit(1); }

There is no evidence that anything bad will happen if the bare LF characters are permitted in the message body. So, in order to fix this problem in qmail, I modified qmail-smtpd.c so that the straynewline() function takes no action:
void straynewline() { }

This modification finally let the order confirmation I was expecting arrive successfully.

I recommend that everybody who runs qmail apply this modification. You may be missing mail and not know it.
(11 comments | Leave a comment)

Sunday, November 30th, 2003

greylisting, day 14

About two weeks ago I implemented the anti-spam technique known as greylisting on my mail server. The results have been astounding in their success.

Greylisting relies on passively verifying the behaviour of the sending SMTP server. The first time an incoming connection is made from an unknown server, the delivery is rejected with a temporary failure error message. This temporary failure message causes a normal sending server to try again later, but it seems that a typical high-volume message sender used to send bulk mail will not bother trying again. If the message delivery is attempted a second time, the greylist filter lets the message through, concluding that the sender is legitimate. Subsequent messages from the same sender are then let through on the first try in order to avoid unnecessarily delaying mail. Messages that are sent once and never retried never make it through the filter.

I have been keeping a history of messages caught by various spam filters. The "spam" line is spamassassin, the black line is the spamcop DNS-based blacklist, and the others are custom filters for specific kinds of email worms. Since implementing greylisting, the amount of spam that enters my mail system has been reduced by about 95%.

I also have some graphs of greylisting status, which report the counts of senders in the greylisting database. The green indicates the current number of legitimate senders seen within the last 10 days. The blue line indicates the current number of senders that have only tried once within the last six hours. Multiplying this value by 4 gives approximately the number of message blocked in the last 24 hours, which matches well with the 200 to 300 spam messages previously caught by my existing filters.

I have been watching mail logs pretty carefully and have not noticed any messages that should have been accepted but weren't. Some others who have experimented with this technique have noticed some messages were blocked that shouldn't have been.

For reference, I'm using the qgreylist implementation along with qmail. This implementation differs from the original greylisting paper by only considering the sending server IP address (instead of the ip/sender/recipient triplet) when making the decision to accept or reject an incoming message. This is much simpler and appears to be sufficient to block the vast majority of bulk mail.
(2 comments | Leave a comment)