Greylisting
In short, a method for reducing spam at the MTA (mail server) outlined at http://projects.puremagic.com/greylisting/whitepaper.html .
From the whitepaper:
"Greylisting has been designed from the start to satisfy certain criteria:
- Have minimal impact on users
- Limit spammers ability to circumvent the blocking
- Require minimal maintenance at both the user and administrator level
User-level spam blocking, while somewhat effective has a few key drawbacks that make its use in the continuing spam war undesirable. A few of these are:
- It provides no notice to the senders of legitimate email that is falsely identified as spam.
- It places most of the costs of processing the spam on the receivers side rather than the spammers side.
- It provides no real disincentive to spammers to stop wasting our time and resources.
As a result, Greylisting is designed to be implemented at the MTA level, where we can cause the spammers the most amount of grief."
2004-06-09 SRIV A tcl implementation using Sqlite as the database. Follow the directions on the Sqlite page to set up the Sqlite extension for use in your tcl installation. I have yet to place this on my mail server, its on my todolist for this week. I just wanted to get the code out for now. I'll post statistics on its effectiveness after its been in use for a month or so.
2004-06-09 SRIV Results of its first full week in use, compared to the previous week.
#! /bin/sh
# tcl \
exec tclsh "$0" ${1+"$@"}
# Created 2004-06-09 by Steve Redler IV
# greylisting engine based on the principles of
# http://projects.puremagic.com/greylisting/whitepaper.html
lappend auto_path lib
package require sqlite
#main test proc, returns PASS or FAIL to caller
proc grey_test {ipadress sender recipient} {
# Test if weve seen this triplet before
# For all checks, we ignore records whose lifetime has expired
# If we have not seen it, create a record describing it and
# return a tempfail to the sending MTA.
set curtriple "ipadress = '$ipadress' AND sender = '$sender' AND recipient = '$recipient'"
greydb eval "SELECT * FROM triples WHERE $curtriple" query {}
if {! [info exists query(ipadress)]} {
greydb eval "INSERT INTO triples VALUES('$ipadress', '$sender', \
'$recipient', [clock seconds], [expr [clock seconds] + ($::blockmins * 60)], \
[expr [clock seconds] + ($::expiremins * 60)], 1, 0)"
return "FAIL new record "
}
#update records whose lifetime has expired ie: inactive records
if {$query(recexptime) < [clock seconds]} {
incr query(blockedattempts)
set recexptime [expr [clock seconds] + ($::expiremins * 60)]
set blockexptime [expr [clock seconds] + ($::blockmins * 60)]
greydb eval "UPDATE triples SET blockexptime = $blockexptime, blockedattempts = \
$query(blockedattempts), recexptime = $recexptime WHERE $curtriple"
return "FAIL expired record"
}
# If we have seen it, and the block is not expired ,
# return a tempfail to the sending MTA.
# Increment the failed count on the matching row.
# If the sender is the special case of the null sender,
# do not return a failure after RCPT, instead wait until
# after the DATA phase.
if {$query(blockexptime) > [clock seconds]} {
incr query(blockedattempts)
greydb eval "UPDATE triples SET blockedattempts = $query(blockedattempts) WHERE $curtriple"
return "FAIL timed block "
}
# If we have seen it, and the block has expired and lifetime
# has expired, then pass the email.
# Increment the passed count on the matching row.
# Reset the expiration time of the record to be the standard
# lifetime past the current time.
incr query(msgspassed)
set recexptime [expr [clock seconds] + ($::expiremins * 60)]
greydb eval "UPDATE triples SET msgspassed = $query(msgspassed), recexptime = $recexptime WHERE $curtriple"
return "PASS email passed "
}
catch {source [file join $::confdir smtpd.conf]}
catch {source smtpd.conf}
if {! [info exists ::confdir]} {set ::confdir "."}
#blockmins is suggested to be 60 minutes
set ::blockmins 60
#expiremins is suggested to be 240 minutes
set ::expiremins 240
#open and/or initialize the database
sqlite greydb [file join $::confdir greylist.db]
greydb timeout 60000
if {[greydb eval {Select name from SQLITE_MASTER}] == ""} {
puts "Initializing databse [file join $::confdir greylist.db]"
greydb eval {CREATE TABLE triples(ipadress text, sender text, recipient text, createtime int, blockexptime int, recexptime int, blockedattempts int, msgspassed int)}
}
puts "greylisting module loaded"
#end of module
# append this test code to simulate some usage, delete the greylist.db file in between test runs
if 1 {
# set some short time, we dont got all day!
set ::blockmins 1
set ::expiremins 2
set run 0
#here are some initial messages
while {$run < 5} {
puts [clock format [clock seconds] -format %T]
puts " [grey_test 192.168.0.1 [email protected] [email protected]] 192.168.0.1 [email protected] [email protected]"
puts " [grey_test 192.168.0.1 [email protected] [email protected]] 192.168.0.1 [email protected] [email protected]"
puts " [grey_test 192.168.0.1 [email protected] [email protected]] 192.168.0.1 [email protected] [email protected]"
after 20000
incr run
}
#wait for records to expire, see if they reset
puts "pausing for 120 seconds"
after 130000
puts [clock format [clock seconds] -format %T]
puts " [grey_test 192.168.0.1 [email protected] [email protected]] 192.168.0.1 [email protected] [email protected]"
puts " [grey_test 192.168.0.1 [email protected] [email protected]] 192.168.0.1 [email protected] [email protected]"
puts " [grey_test 192.168.0.1 [email protected] [email protected]] 192.168.0.1 [email protected] [email protected]"
#test again
set run 0
while {$run < 5} {
puts [clock format [clock seconds] -format %T]
puts " [grey_test 192.168.0.1 [email protected] [email protected]] 192.168.0.1 [email protected] [email protected]"
puts " [grey_test 192.168.0.1 [email protected] [email protected]] 192.168.0.1 [email protected] [email protected]"
puts " [grey_test 192.168.0.1 [email protected] [email protected]] 192.168.0.1 [email protected] [email protected]"
after 20000
incr run
}
#done
}2004-06-29 SRIV Added busy timeout to the database. Yesterday I shoehorned this module into my existing email proxy. It spawns multiple copies of itself on incomming mail, so several processes could try to have a lock on the database at one time. The default timeout is 0 ms, which caused an instant error if the db was locked.
After one day of operation I can see that this technique for blocking spam is highly effective. A side effect is that it appears to be blocking all the mail sent from virus infected Windows computers as well! I'll post some detailed results after I collect more data.
Mini-Update Not one virus got passed the greylist to trigger the AntiVirus scanner in the last 24 hours. My server intercepts a minimum of 10-30 per day. Awesome!
Could this be combined with a Tcl-based SMTP server? I've got this little hardware box sitting here which could be quite useful as a self-contained email filter <wink>. It could collect emails from the big bad net, and send the ones that get through onwards to a regular SMTP server on a local network, for example... -jcw
AK IIRC 'stever' (== SRIV) was working on a combination smtp/pop/... server which could do this. He mentioned this on the chat.
SRIV Indeed I am. I've named it "emaild", a combo smtpd/pop3d server including black, white and greylisting controls on incomming email, and pop-before-smtp authentication to allow remote users to send mail. It is packaged as a single file cross platform starkit. The goal is to simply drop this file and a tclkit onto a host and have an instant email server. A sort of complement to tclhttpd. Long term goals are to finish the tcl dns server and enhance my tcl webmail server to round out the feature set. Emaild is being tested now and I plan to release it after it performs well on a production server for at least a month. I dont have the means to handle the bug reports if I were to release it now :) Anyone interested in playing, testing or collaborating on emaild can contact me. Otherwise, I hope to have it out by the end of July 2004.
My current email proxy is "DasProxy" - http://dasproxy.sourceforge.net , which is a pure tcl front end to an smtpd like sendmail. Its based on vers .80 of MailStripper. But after successfully running this on a production server for 1 year, I got the urge to eliminate the sendmail (and its configuration hell) and go to a pure tcl solution.