Skip to content

Test my auto-generated ruleset

(I posted this to the SA users and dev lists, too.)

I've been working on a new way to auto-generate body rules recently (see previous posts). The results are checked into SVN trunk daily in the "rulesrc/sandbox/jm/20_sought.cf" file.

We haven't had much time to figure out how to produce auto-generated 3.2.x rule updates for our entire ruleset at updates.SpamAssassin.org, so instead of dealing with that, I've taken a shortcut around it ;) I'm now making just the "20_sought.cf" ruleset available as a standalone, unofficial sa-update ruleset at sought.rules.yerp.org.

Before using it, you'll need the GPG key:

  wget http://yerp.org/rules/GPG.KEY
  sudo sa-update --import GPG.KEY                

then use this to update:

  sudo sa-update \
        --gpgkey 6C6191E3 --channel sought.rules.yerp.org \
        [...other channels...] \
        --channel updates.spamassassin.org

(similar to how you'd use Daryl's sa-update version of the SARE rulesets.)

Feel free to run sa-update as frequently as you like.

Please consider it alpha; I may take it down in a few months depending on how it goes, or if we can get it working as part of the core updates. In the meantime though, I'm curious to hear how you get on with it. (In particular, copies of false positives would be very welcome.)

Update: it's been very successful, so I'd now consider it in production.

rules rulesets sa-update spamassassin

60 Comments

  1. James
    August 21, 2007

    Hi Justin,

    Thanks for all the work you do on SpamAssassin!

    I've been very interested in reading this approach to generating rules.

    I've been testing these rules on a fairly small company server (not using sa-update: I've been downloading them from SVN and checking them myself before putting them into production). I've been getting SA scores on OEM spam into the 150 range, with no noticed FPs.

    One thing I have noticed: JM_SEEK_UTAGQU is nearly identical to a local rule I added about a year back. (My rule only fires if the text is in the subject, as it always is in my experience). It's a very good spam sign (and very unlikely to FP), so between your rule and mine, an e-mail can get 5.5 points just by having that one spam sign. In this case, it isn't a problem.

    It's just that daily automatically-updated rules are likely to clash with other rules, and an e-mail get scored twice for the same spam-sign. (Revision 567659 has two copies of JM_SEEK_7HOHFW -- since the name is identical, SA knows what to do, of course).

    Given the potential complexity of Perl regexps, I'm not sure what can be done about this.

  2. August 21, 2007

    Yes, I noticed the duplicate rule -- I need to look into that. Thanks for the feedback!

  3. Slava
    February 25, 2008

    Hello Justin,

    Will this rule (sought.cf) work with SA v.3.1.9?
    Thank you,
    Slava.

  4. February 25, 2008

    hi Slava --

    it should. It would be more efficient on 3.2.x, since you can use "sa-compile" to compile the ruleset there, but should work fine on 3.1.x as well.

  5. Bubuk
    February 27, 2008

    Hi,
    Excuse my nubiness in all these. I have my root cron like so:

    41 * * * * /usr/local/bin/freshclam --quiet

    00 * * * * /usr/bin/sa-update --gpgkey D1C035168C1EBC08464946DA258CDB3ABDE9DC10 --channel saupdates.openprotect.com --channel updates.spamassassin.org --allowplugins

    and I would like to add sought.cf into my rulesets.

    1) So I login as root. And I ran

    wget http://yerp.org/rules/GPG.KEY

    sa-update --import GPG.KEY

    2) Now how do I incorporate these lines (from your instruction above)
    sa-update \
    --gpgkey 6C6191E3 --channel sought.rules.yerp.org \
    [...other channels...] \
    --channel updates.spamassassin.org

    into my root cron? Would this do the job:

    41 * * * * /usr/local/bin/freshclam --quiet

    00 * * * * /usr/bin/sa-update --gpgkey D1C035168C1EBC08464946DA258CDB3ABDE9DC10 --channel saupdates.openprotect.com --gpgkey 6C6191E3 --channel sought.rules.yerp.org --channel updates.spamassassin.org --allowplugins

    Please help. TIA.

  6. Sujit Choudhury
    February 29, 2008

    Hi Justin,
    I am running SA 3.1.7 with SARE rules as well as the original SA rulesets. I also use sagrey.cf,botnet.cf and imageinfo.cf. Will the addition of this ruleset of yours give more false positives?

  7. February 29, 2008

    Sujit: I would not expect it to do so.

  8. John
    May 20, 2008

    Just seeing if i understand correctly, 20_sought.cf will list all current automatically generate rules in the file when updated, and uses your own spam and ham lists to generate, not ones from the end user... amirite?

  9. John
    May 20, 2008

    well still kinda vague, i mean the spam and ham lists to generate are yours (justin) not the local administrator

  10. May 20, 2008

    hi John -- yep, that's the case.

  11. May 29, 2008

    Justin, is there any chance you could add some 'describe' lines for the meta rules in 20_sought.cf? JM_SOUGHT_[1-3] doesn't add much to the readability of the spam report, but even a generic description would be useful, particularly for users of web GUIs (e.g. Maia Mailguard) that try to present a detailed report to the user.

  12. June 6, 2008

    Justin:

    Can you include a generation timestamp in a comment at the top of the file?

    Thanks!

  13. June 9, 2008

    hi John --

    unfortunately I'd prefer not to, as it makes a lot of noise in the SVN changes mails... I'm trying to minimize the noise in those.

  14. June 15, 2008

    Justin, do you know of anyone using your tools to produce rulesets from corpa focused on a specific type of spam (e.g. 419 and lottery fraud)?

    For example, I'm using the SARE fraud ruleset but certain recent variants (e.g. the ATM Card ones) aren't hitting at all. I suggested to them that using your tools against a fraud-specific corpus might be a good idea to keep the rules current.

    I poked around in your latest ruleset and didn't see much that looked fraud-related. Do you have much fraud spam in your spamtrap corpus?

  15. June 16, 2008

    John -- no, I don't. I agree though it'd be a natural.

    'I poked around in your latest ruleset and didn't see much that looked fraud-related. Do you have much fraud spam in your spamtrap corpus?'

    Unfortunately, relative rates of different types of spam differ between different accounts -- you can see this on our ruleQA site, http://ruleqa.spamassassin.org/ . if you click on a rule's "display hits over time" graph, you can see that different mass-check contributors have entirely different hitrates on spam for various rules! This means that what makes up 10% of one contributor's corpus might make up only 0.3% of another's. it's very odd. I think it's a side effect of where spammers find addresses.

    I've been trying to find a way to compensate for that, for a long time. it's tricky. :(

    I would love to see multiple "sought" rulesets, run against different corpora. I've responded to your mail on the SA users list....

  16. Patrick
    July 3, 2008

    sa-compile did not work, so I checked JM_SOUGHT_1, JM_SOUGHT_2 and JM_SOUGHT_3 one by one. The issue was caused by JM_SOUGHT_3, the other meta-tests work fine.
    I verified on two platforms with different releases of gcc and spamassassin.

  17. July 3, 2008

    @Patrick: please check the SA FAQ -- this problem is frequently caused by using the wrong version of re2c. If that's not it I suggest mailing the SA users mailing list...

  18. coreyva
    July 9, 2008

    I use sa-update with the sare rules and now your sought.cf. I figured my config my be helpful to some so I'll post it here.

    crontab
    11 2 * * * /usr/bin/sa-update --gpgkeyfile /etc/spamassassin/sare-sa-update-gpgkeys.txt --channelfile /etc/spamassassin/sare-sa-update-channels.txt

    sare-sa-update-gpgkeys.txt contains the gpg keys needed

    856AA88A

    6C6191E3

    sare-sa-update-channels.txt contains the channels to update.

    updates.spamassassin.org
    sought.rules.yerp.org
    72_sare_redirect_post3.0.0.cf.sare.sa-update.dostech.net
    70_sare_evilnum0.cf.sare.sa-update.dostech.net
    70_sare_bayes_poison_nxm.cf.sare.sa-update.dostech.net
    70_sare_html0.cf.sare.sa-update.dostech.net
    70_sare_html_eng.cf.sare.sa-update.dostech.net
    70_sare_header0.cf.sare.sa-update.dostech.net
    70_sare_header_eng.cf.sare.sa-update.dostech.net
    70_sare_specific.cf.sare.sa-update.dostech.net
    70_sare_adult.cf.sare.sa-update.dostech.net
    72_sare_bml_post25x.cf.sare.sa-update.dostech.net
    99_sare_fraud_post25x.cf.sare.sa-update.dostech.net
    70_sare_spoof.cf.sare.sa-update.dostech.net
    70_sare_random.cf.sare.sa-update.dostech.net
    70_sare_oem.cf.sare.sa-update.dostech.net
    70_sare_genlsubj0.cf.sare.sa-update.dostech.net
    70_sare_genlsubj_eng.cf.sare.sa-update.dostech.net
    70_sare_unsub.cf.sare.sa-update.dostech.net
    70_sare_uri0.cf.sare.sa-update.dostech.net
    70_sare_obfu0.cf.sare.sa-update.dostech.net
    70_sare_stocks.cf.sare.sa-update.dostech.net
    70_sare_whitelist.cf.sare.sa-update.dostech.net
    70_sare_evilnum1.cf.sare.sa-update.dostech.net
    70_sare_evilnum2.cf.sare.sa-update.dostech.net
    70_sare_whitelist_spf.cf.sare.sa-update.dostech.net
    70_sare_whitelist_rcvd.cf.sare.sa-update.dostech.net
    70_sare_header1.cf.sare.sa-update.dostech.net
    70_sare_header2.cf.sare.sa-update.dostech.net
    70_sare_header_eng.cf.sare.sa-update.dostech.net
    70_sare_uri_eng.cf.sare.sa-update.dostech.net
    70_sare_uri1.cf.sare.sa-update.dostech.net
    70_sare_uri2.cf.sare.sa-update.dostech.net
    70_sare_obfu4.cf.sare.sa-update.dostech.net
    70_sare_obfu3.cf.sare.sa-update.dostech.net
    70_sare_obfu2.cf.sare.sa-update.dostech.net
    70_sare_obfu1.cf.sare.sa-update.dostech.net
    70_sare_genlsubj2.cf.sare.sa-update.dostech.net
    70_sare_genlsubj1.cf.sare.sa-update.dostech.net
    70_sare_genlsubj_eng.cf.sare.sa-update.dostech.net
    70_sare_html2.cf.sare.sa-update.dostech.net
    70_sare_html_eng.cf.sare.sa-update.dostech.net
    chickenpox.cf.sare.sa-update.dostech.net

    This way you can add channels and keys easier. Hope that's helpful to some.

  19. Patrick
    July 11, 2008

    SA FAQ doesn't mention re2c, but when searching the wiki I found SaCompileRefSymbolError which suggests to update to version 0.12.0 or later. Therefore I've been using 0.12.2 for ages, but your rules didn't work on July 3, 2008 anyway.
    However, I tested today's rules which work fine.
    Since nobody else reported problems, I just don't bother.

  20. Jens Schleusener
    July 13, 2008

    The mechanismn behind 20_sought.cf seems a successful idea since it helps me to detect some otherwise unmarked spams.

    Til July 8th the file 20_sought.cf was updated roughly 2-4 times a day but the last update seems now to be made at that day 4 pm UTC. Any reason or a problem of my update script?.

  21. July 14, 2008

    @Jens -- that should be fixed now.

  22. Jens Schleusener
    July 14, 2008

    Many thanks, it works again! Extract of my script's log:

    === Tue Jul 8 08:05:07 2008: 69459 Jul 8 08:05 20_sought.cf

    === Tue Jul 8 18:05:08 2008: 80330 Jul 8 18:05 20_sought.cf

    === Mon Jul 14 19:05:09 2008: 80666 Jul 14 19:05 20_sought.cf

    === Mon Jul 14 22:05:09 2008: 82708 Jul 14 22:05 20_sought.cf

  23. Jens Schleusener
    August 6, 2008

    The last update of "20_sought.cf" seems now be made at August 3rd at 1 am UTC
    (roughly three days ago). Again a small problem or just a delay?

  24. August 6, 2008

    Jens, I'm in the middle of a server move. it should be back again shortly.

  25. Jens Schleusener
    August 6, 2008

    Ok, no hurry. Good things are worth waiting for ...

  26. Rich Wales
    November 16, 2008

    These rules appear to be broken, as of version 320713383 (12 Nov. 2008). Starting with that version, my sa-compile has been failing in the middle of one of the re2c's.

    I tried paring down the affected scanner###.re file and running re2c by hand, and it looks like the offender is __SEEK_CCNRJL, which was introduced in version 320713383. There might be other problems after this point, but hopefully this can serve as a starting point for someone to be able to find and fix the problem(s).

    In the meantime, I'm going to try reinstalling the previous version (320713360), and disabling sa-update until I hear that things are working properly again.

  27. Sebastian
    November 17, 2008

    I can confirm this problem, it's not working for me either. I removed the compiled rules rather then disabling sa-update.

  28. Rich Wales
    November 19, 2008

    The problem I reported the other day went away after I upgraded my SpamAssassin from version 3.2.3 to 3.2.5.

    I am once again successfully using Justin's rules (currently version 320718672).

  29. November 19, 2008

    Sorry -- I should have mentioned that there are indeed some fixes to sa-compile in 3.2.4 and 3.2.5; if you have problems, an upgrade of that is strongly recommended.

  30. Mark
    April 17, 2009

    These rules put a heavy load on my busy servers. The fraud ruleset is over 300k, which Spamassassin says is too large to run. Can this be fixed somehow? If not, just consider this a warning to others who are seeing performance problems.

  31. Marcelo
    May 7, 2009

    Hello Justin,

    For many reasons I don“t use sa-update but I would kindly ask to use your rules. Is it possible I get it using wget one time per week ? I am trying to discover the url to wget :))

    Can you give me it ?

    Thanks and Best Regards,
    Marcelo

  32. May 7, 2009

    Marcelo, you'll have to do a few steps to determine the URL, to emulate sa-update.

    First, "dig mirrors.sought.rules.yerp.org. TXT" to get the mirrors file's URL -- http://yerp.org/rules/MIRRORED.BY. curl that, and you'll see that the base URL for updates is http://yerp.org/rules/stage/ . next, "dig 0.2.3.sought.rules.yerp.org txt" to get the current version: '320772726'. put that together with the base URL, add .tar.gz , and that's the tarball: http://yerp.org/rules/stage/320772726.tar.gz .

    it'd be easier to use sa-update. ;)

  33. May 11, 2009

    Dear Justin,

    Thanks for this. Are there plans to incorporate your rules into SA 3.2.+? I would like to be aware so that if/when that happens, I can stop incorporating these into my SA setup to avoid double-scoring particular email attributes.

  34. May 11, 2009

    hi Nick -- currently, no. it'll stay as an optional add-on ruleset, separately sa-update'd.

  35. Stan
    June 8, 2009

    Dear Justin

    I would like create a unofficial channel as you

    I create my package tar.gz:

    cd /home/rules/updates

    ls -lrt

    *.cf

    tar -cf 2.tar *.cf

    gzip -9v 2.tar

    sha1sum 2.tar.gz

    gpg -bas 2.tar.gz

    I export my public key

    gpg --export -a key > /home/rules/PUB.key

    I create a file MIRRORED.BY in /home/rules/ with url http://myserver.domain.com/rules/updates/

    And after how to configure DNS to use for example --channel updates.myserver.domain.com with sa-update

    regards

  36. SpamZombie
    August 5, 2009

    Heya, Justin. Are these rules still being updated every 4 hours? The timestamps on the files sometimes seem to indicate they're not.

    Also, I did read through the comments (and some of the SA wiki on the SoughtRules, which says they're updated every 4 hours) but didn't see if you said how often it was okay to check for updates? I'm currently connecting 4 times in 24 hours, but would like to get them more often if possible and don't want to overload the server...

    Thanks!

  37. January 27, 2010

    It appears that this list is no longer available...

  38. January 27, 2010

    what gives you that idea?

  39. SpamZombie
    January 27, 2010

    what gives you that idea?

    We were getting 404's for several hours trying to download the rules yesterday via sa-update. Maybe that's what Tony means? I thought maybe you'd discontinued 'em, but didn't see a post here so figured it was a glitch.

    Seems to be working fine today.

  40. Paul Fisher
    March 21, 2010

    Now that SpamAssassin 3.3.1 is out, is "sought" going to be updated to support the sa-update in 3.3.1? DNS for yerp.org needs to be updated.

    dbg: dns: query failed: 1.3.3.sought.rules.yerp.org => NXDOMAIN
    channel: no updates available, skipping channel

    Thanks.

  41. March 21, 2010

    Paul, thanks for noticing that. fixed.

  42. Steve Rawlinson
    April 22, 2010

    Sorry to bother you with what is probably a dumb question but following your intructions for using sa-update to get these rules everything works fine until:

    sa-update --gpgkey 6C6191E3 --channel sought.rules.yerp.org

    http: request failed: 500 Can't connect to yerp.org:8080 (connect: timeout): 500 Can't connect to yerp.org:8080 (connect: timeout)

    Am I doing something stupid?

  43. April 22, 2010

    hmm. it sounds like port 8080 isn't going to work at your site; I recently moved the hosting to a server running on that port, but firewall issues may mean I need to reconsider that.

  44. Steve Rawlinson
    April 23, 2010

    You're right it was a firewall at my end. Fixed now, many thanks.

  45. D. Stussy
    August 16, 2010

    "Feel free to run sa-update as frequently as you like."

    Although you do say this, you (and every other SA channel provider) should indicate a suggested update frequency. Otherwise, one may end up with some a$$ who decides a cron job to check every 5 minutes is adequate (but clearly inappropriate for a service that updates once per hour or per day). Therefore, the frequency in which YOU update the rules you offer should be the disclosed (and be the minimum that is used by everyone to check for updates).

  46. marcin
    January 14, 2011

    I can't use this rules, i'm getting:
    GET http://rules.yerp.org.s3.amazonaws.com/rules/stage/3301058947.tar.gz request failed, retrying: 500 Can't connect to rules.yerp.org.s3.amazonaws.com:80 (Bad hostname 'rules.yerp.org.s3.amazonaws.com'): 500 Can't connect to rules.yerp.org.s3.amazonaws.com:80 (Bad hostname 'rules.yerp.org.s3.amazonaws.com')

    Is this channel still active?

  47. January 14, 2011

    hey Marcin -- that should work ok:


    HEAD http://rules.yerp.org.s3.amazonaws.com/rules/stage/3301058947.tar.gz
    200 OK
    Date: Fri, 14 Jan 2011 15:08:16 GMT
    Accept-Ranges: bytes
    ETag: "043fcddbf086c5b3fb6ab5d4418a23e0"
    Server: AmazonS3
    Content-Length: 27913
    Content-Type: application/x-tar
    Last-Modified: Fri, 14 Jan 2011 11:17:04 GMT
    Client-Date: Fri, 14 Jan 2011 15:08:15 GMT
    Client-Peer: 72.21.207.153:80
    Client-Response-Num: 1
    X-Amz-Id-2: 8+ptVQ8xBS27qv4irHC5HzMvRJKpmiuOdBhxSlaeVDtiNWRXGunVpzxa+hJjYEpz
    X-Amz-Meta-S3cmd-Attrs: uid:1037/gname:jm/uname:jm/gid:1037/mode:33188/mtime:12950[...]

  48. Marcin
    January 15, 2011

    Thanks Justin, now everething works. I've downloaded rules succesfully.
    Regards.

  49. Andrea
    March 1, 2012

    Justin,

    I'm using your rules. Once downloaded and compiled, I got this warning with spamassassin --lint:

    Mar 1 18:46:33.521 [20359] dbg: zoom: skipping rule __SEEK_ZUDUJB, code differs in compiled ruleset
    Mar 1 18:46:33.535 [20359] dbg: zoom: able to use 2647/2648 'body_0' compiled rules (99.962%)

    Is there a problem in the rules or on my side?

    Thanks,
    Andrea

  50. Robert Pelletier
    March 6, 2012

    I didn't get any update in weeks. Usually, I get at least one per day. Is it still active?

  51. Al
    March 8, 2012

    It seems there is no DNS record for sought.rules.yerp.org.

    I wanted to start using these rules, but it looks as if there is a problem.
    Am I missing something, have the rules moved, is this temporary, or are they gone for good ?

    Thanks for any information, and for the spam fighting efforts!

    -Al

  52. James
    May 9, 2012

    Hi Justin,

    __SEEK_0IKD76 seems to be firing on MailChimp standard headers. MailChimp are a known ESP with a fairly good reputation: I think this means you don't have any of their email in your known good corpus.

    Al, if you're still reading, there isn't supposed to be a DNS record for sought.rules.yerp.org. Try

    dig -t txt +short 2.3.3.sought.rules.yerp.org
    or
    nslookup -type=txt 2.3.3.sought.rules.yerp.org

    James

  53. Evan Richardson
    November 7, 2012

    Doesn't seem to work for me. I tried following the directions:

    1.) download the gpg.key
    2.) import gpgkey (sa-update --import GPG.KEY
    3.) type the sa-update --gpgkey 6c6191e3 --channel sought.rules.yerp.org --channel updates.spamassassin.org but I get the following:

    "error: GPG validation failed! The update downloaded successfully, but the GPG signature verification failed. channel: GPG validation failed, channel failed"

  54. mso
    February 17, 2013

    What zq Freddie says: For about a week now it's constantly returning error 403 Forbidden from Amazon's S3 service...

  55. Sudip
    February 18, 2013

    I am also having the same error for last one/two weeks. This is the message :

    http: GET http://rules.yerp.org.s3.amazonaws.com/rules/stage/3302013021221.tar.gz request failed: 403 Forbidden: AccessDeniedAccess DeniedE91FEE743F37AB782jrZhtzJeiwRCD+L8OEsUoN5+HPaqqst2/9TP7oNDainn1sNF52HOXItmHOSs6bD
    channel: could not find working mirror, channel failed

    After a google search it turned out that many people are now having the same error. How do we change the mirror??

    Thanks in advance

    Sudip

  56. Andy
    February 18, 2013

    As a quick fix to stop the error message you can rename the file "/etc/mail/spamassassin/channel.d/sought.conf" to anything that doesn't end in ".conf". This will obviously stop the rules from updating so you'll need to monitor the situation and manually restore the file if/when the problem is resolved.

  57. mso
    February 18, 2013

    @Andy: Thank you, will give it a try!

  58. February 18, 2013

    hey folks -- that should now be fixed. let me know if there are still problems!

  59. mso
    February 19, 2013

    It looks good again. Thanks a lot!