Email defences; additional DNS RBL


#21

Yes, github would be better – I must get round to it – but I have to confess my only attempt to use it left me a bit on the confused side. :flushed:


#22

It’s great to hear that it’s being used. I’m toying with the idea of supporting whitelisting (service-specific, site-wide), mainly for spamcop – it’s doing useful work but I’m seeing false positives for the likes of facebookmail & pinterest.

admin@vm1:~$ /srv/.all-sites/utils/rblinfo

   10 rbl services configured (non-spamhaus might 'tag')
    8 rbl services show log rejection messages

  service                    sites     rejections
---------------------------------------------------
  zen.spamhaus.org              26          1,018
  b.barracudacentral.org        15            108
  truncate.gbudb.net            15             48
  multi.uribl.com               15             28
  all.spamrats.com              12             27
  bl.spamcop.net                10             20
  excommunicado.co.uk            4             13
  dbl.spamhaus.org              15              3
  bl.mailspike.net              13              0
  dnsbl.dronebl.org              6              0
  TOTAL                          -          1,265
---------------------------------------------------
  spamassassin                  26             10
  clamav                         0
---------------------------------------------------
  [Duration: ~0.13s]

#23

The latest version of 76-x-dns-blacklists-extra adds supports for dnsbl.sorbs.net, rhsbl.sorbs.net and all.s5h.net. There’s been a change in the way the non-rhsbl services work: a hit at a service set to “tag” will now prevent lookups at any other services - so, it’s possible that tagging prevents rejection. As “tag” is primarily a test-mode I didn’t think it was worth replicating the old behaviour.

The dnslists consolidation should mean more efficient lookups and the config is easier to maintain but the next version will go further and avoid hardcoding supported services. Instead, expect something like…

/srv/.all-sites/config/blacklists/dnsbl-services.conf
/srv/.all-sites/config/blacklists/rhsbl-services.conf

… which contain the hostnames of all valid services. The files can be updated in realtime with no need to alter the exim4 config or restart the service. When an entry is removed any associated /srv/*/config/blacklists files will be ignored.

Whitelisting may also become a thing…

/srv/.all-sites/config/blacklists/whitelist-by_ip
/srv/.all-sites/config/blacklists/whitelist-by_sender
/srv/.all-sites/config/blacklists/whitelist-by_hostname
# Non-spamhaus dnsbl lookups
# ===========================
#
# 20171002
# - new consolidated dnslists format for non-rhsbl services (rhsbl to follow)
#   lookups will now stop on first hit, therefore, 'tag' might prevent a different
#   service from rejecting.
# - added support for all.s5h.net
# 20170928
# - added support for dnsbl.sorbs.net and rhsbl.sorbs.net
# - removed experimental whitelisting
# 20170916
# - removed support for bad.psky.me
# - added whitelisting by rbl service (site-wide)
#
# https://forum.bytemark.co.uk/t/email-defences-additional-dns-rbl/2508
#
# DNS RBL lookups will occur when any of the following files are found in /srv/my-brilliant-site/config/blacklists
#
#    rbl (ip addresses)
#    b.barracudacentral.org  http://www.barracudacentral.org/
#    truncate.gbudb.net      http://www.gbudb.com (127.0.0.2 only)
#    bl.mailspike.net        http://mailspike.net (127.0.0.2, 127.0.0.10-12)
#    all.spamrats.com        http://www.spamrats.com (127.0.0.36-38 : dyna, noptr, spam)
#    bl.spamcop.net          https://www.spamcop.net/ (false positives likely?, also used by spamassassin)
#    bl.spamcannibal.org     http://www.spamcannibal.org
#    dnsbl.dronebl.org       https://dronebl.org
#    dnsbl.sorbs.net         http://www.sorbs.net
#    all.s5h.net             http://www.usenix.org.uk/content/rbl.html
#
#    rhsbl (domains)
#    excommunicado.co.uk     https://github.com/excommunicado/communicado
#    multi.uribl.com         http://uribl.com/
#    dbl.spamhaus.org        https://www.spamhaus.org/dbl/ (127.0.1.[2,4,5,6,101-106] !127.0.1.255)
#    rhsbl.sorbs.net         http://www.sorbs.net
#
# Messages will be rejected unless the files contain a line beginning "tag" - which will result
# in a log entry and (untested) an 'X-Spam-Blacklisted-at' header.
#
# ------------------------------------------------------------------------------------
# vars:
# - creates; acl_m105076_act (action), acl_m105076_msg (message)
#   (neither is used outside this acl/message/connection)
# ------------------------------------------------------------------------------------
# Consider the bumble bee, grasshopper;
# - can the X-Spam-Blacklisted-at header clobber or be clobbered?
# - is acl_m105076_msg needed for the deny message;
#   can message/log_message cascade across verbs?
# - should $acl_m* be $acl_c* for non-rhsbl lookups?
# - is checking for whitelisted senders/rcpts redundant here?
# - add hosts = !+relay_from_hosts : !+private_addresses ?
# - test for def:acl_m* (& $dnslist_*) in case strict_acl_vars is set
# - efficiency; file lookups (& caching ${lookup) v defining
#   warn and deny lists at start-up?
# Might-Do:
# - standardise support for response-specific actions (127.0.0.3 = tag)
# - further dnslists consolidation
# - use exim directory-search/lookup to avoid hardcoding rbl engines
# - use scoring on multiple warn hits; reject on incidence and/or score
# - add support for spamassassin scoring on warn (probably not)
# ------------------------------------------------------------------------------------


# dnsbl (ip addresses)
# -----------------------------
# multi-service dnslists for non-rhsbl services only (for now)

         warn            domains              = +vhost_domains
                         dnslists             = ${filter{ \
                                                          b.barracudacentral.org : \
                                                          truncate.gbudb.net     : \
                                                          all.spamrats.com       : \
                                                          bl.mailspike.net       : \
                                                          bl.spamcannibal.org    : \
                                                          bl.spamcop.net         : \
                                                          dnsbl.dronebl.org      : \
                                                          dnsbl.sorbs.net        : \
                                                          all.s5h.net              \
                                                        } \
                                                        {exists{VHOST_DIR/$domain/VHOST_CONFIG_DIR/blacklists/$item}} \
                                                }
                         add_header           = X-Spam-Blacklisted-at: $dnslist_domain ($dnslist_value)
                         set acl_m105076_act  = ${lookup{tag}lsearch{VHOST_DIR/$domain/VHOST_CONFIG_DIR/blacklists/$dnslist_domain}{tag}{reject}}
                         set acl_m105076_msg  = $sender_host_address is blacklisted at $dnslist_domain ($dnslist_value) \
                                                ${if def:dnslist_text {\n$dnslist_text}}
                         logwrite             = ACL105076 H=$sender_fullhost IP address blacklisted at $dnslist_domain ($dnslist_value) - $domain set to $acl_m105076_act

         deny            condition            = ${if eq {$acl_m105076_act}{reject}}
                         message              = $acl_m105076_msg

#                        !condition            = ${if and{ \
#                                                   {exists{VHOST_DIR/.all-sites/config/blacklists/whitelist-by_ip}} \
#                                                   {${lookup{$sender_host_address}net-iplsearch{VHOST_DIR/.all-sites/rbl/whitelists/$dnslist_domain/by_ip}}} \
#                                                 }}
#                        !condition            = ${if and{ \
#                                                   {exists{VHOST_DIR/.all-sites/config/blacklists/whitelist-by_sender}} \
#                                                   {${lookup{$sender_address_domain}wildlsearch{VHOST_DIR/.all-sites/rbl/whitelists/$dnslist_domain/by_sender}}} \
#                                                 }}
#                        !condition            = ${if and{ \
#                                                   {exists{VHOST_DIR/.all-sites/config/blacklists/whitelist-by_hostname}} \
#                                                   {${lookup{$sender_host_name}wildlsearch{VHOST_DIR/.all-sites/rbl/whitelists/$dnslist_domain/by_hostname}}} \
#                                                 }}


# rhsbl (reverse-path domain)
# -----------------------------

# excommunicado

         warn            domains              = +vhost_domains
                         condition            = ${if exists{VHOST_DIR/$domain/VHOST_CONFIG_DIR/blacklists/excommunicado.co.uk}}
                         dnslists             = excommunicado.co.uk/$sender_address_domain
                         add_header           = X-Spam-Blacklisted-at: $dnslist_domain ${if def:dnslist_value {($dnslist_value)}}
                         set acl_m105076_act  = ${lookup{tag}lsearch{VHOST_DIR/$domain/VHOST_CONFIG_DIR/blacklists/$dnslist_domain}{tag}{reject}}
                         set acl_m105076_msg  = Communicado Ltd., see http://blog.hinterlands.org/2013/10/unwanted-email-from-communicado-ltd/\n \
                                                $dnslist_text
                         logwrite             = ACL105076 H=$sender_fullhost $sender_address_domain blacklisted at $dnslist_domain - $domain set to $acl_m105076_act

         deny            condition            = ${if eq {$acl_m105076_act}{reject}}
                         message              = $acl_m105076_msg

# multi.uribl.com

         warn            domains              = +vhost_domains
                         condition            = ${if exists{VHOST_DIR/$domain/VHOST_CONFIG_DIR/blacklists/multi.uribl.com}}
                         dnslists             = multi.uribl.com=127.0.0.2,127.0.0.4,127.0.0.8/$sender_address_domain
                         add_header           = X-Spam-Blacklisted-at: $dnslist_domain ${if def:dnslist_value {($dnslist_value)}}
                         set acl_m105076_act  = ${lookup{tag}lsearch{VHOST_DIR/$domain/VHOST_CONFIG_DIR/blacklists/$dnslist_domain}{tag}{reject}}
                         set acl_m105076_msg  = $sender_address_domain blacklisted at $dnslist_domain ($dnslist_value)\n \
                                                $dnslist_text
                         logwrite             = ACL105076 H=$sender_fullhost $sender_address_domain blacklisted at $dnslist_domain - $domain set to $acl_m105076_act

         deny            condition            = ${if eq {$acl_m105076_act}{reject}}
                         message              = $acl_m105076_msg

# dbl.spamhaus.org

         warn            domains              = +vhost_domains
                         condition            = ${if exists{VHOST_DIR/$domain/VHOST_CONFIG_DIR/blacklists/dbl.spamhaus.org}}
                         dnslists             = dbl.spamhaus.org!=127.0.1.255/$sender_address_domain
                         add_header           = X-Spam-Blacklisted-at: $dnslist_domain ${if def:dnslist_value {($dnslist_value)}}
                         set acl_m105076_act  = ${lookup{tag}lsearch{VHOST_DIR/$domain/VHOST_CONFIG_DIR/blacklists/$dnslist_domain}{tag}{reject}}
                         set acl_m105076_msg  = $sender_address_domain blacklisted at $dnslist_domain ($dnslist_value)\n \
                                                $dnslist_text
                         logwrite             = ACL105076 H=$sender_fullhost $sender_address_domain blacklisted at $dnslist_domain - $domain set to $acl_m105076_act

         deny            condition            = ${if eq {$acl_m105076_act}{reject}}
                         message              = $acl_m105076_msg

# rhsbl.sorbs.net

         warn            domains              = +vhost_domains
                         condition            = ${if exists{VHOST_DIR/$domain/VHOST_CONFIG_DIR/blacklists/rhsbl.sorbs.net}}
                         dnslists             = rhsbl.sorbs.net/$sender_address_domain
                         add_header           = X-Spam-Blacklisted-at: $dnslist_domain ${if def:dnslist_value {($dnslist_value)}}
                         set acl_m105076_act  = ${lookup{tag}lsearch{VHOST_DIR/$domain/VHOST_CONFIG_DIR/blacklists/$dnslist_domain}{tag}{reject}}
                         set acl_m105076_msg  = $sender_address_domain blacklisted at $dnslist_domain ($dnslist_value)\n \
                                                $dnslist_text
                         logwrite             = ACL105076 H=$sender_fullhost $sender_address_domain blacklisted at $dnslist_domain - $domain set to $acl_m105076_act

         deny            condition            = ${if eq {$acl_m105076_act}{reject}}
                         message              = $acl_m105076_msg


The latest version of rblinfo (20171002 at time of writing) is much more efficient and lacks mind-boggling column sorting behaviour which affected the cumulative total. I’ll miss that bug. :slight_smile:

admin@vm1:~$ /srv/.all-sites/utils/rblinfo                                                                                   
   13 rbl services configured (non-spamhaus might 'tag')
   11 rbl services show log rejection messages

  service                    sites     rejections
---------------------------------------------------
  zen.spamhaus.org              26          1,259
  b.barracudacentral.org        15            104
  truncate.gbudb.net            15             40
  all.spamrats.com              13             22
  multi.uribl.com               15             22
  excommunicado.co.uk            4             19
  bl.spamcop.net                11             11
  dnsbl.sorbs.net                1              9
  dbl.spamhaus.org              15              4
  all.s5h.net                    9              1
  bl.mailspike.net              13              1
  dnsbl.dronebl.org              7              0
  rhsbl.sorbs.net                8              0
  TOTAL                          -          1,492
---------------------------------------------------
  spamassassin                  26             17
  clamav                        26              0
---------------------------------------------------
  v20171002 : ~0.17s

#24

Thanks for the latest version. Ironically your setup has flagged up that bigv.io is listed on the SpamhausDBL:

https://www.spamhaus.org/dbl/removal/record/bigv.io

Which is causing mail rejection issues on one of my machines.


#25

Hell, yes - I must accelerate the whitelisting feature to accommodate dodgy bytemark customers. :wink:

It seems to be clear now so I suspect there’s been (rapid) action behind the scenes.


#26

Hi All

A second service appears to have gone rogue, and should no longer be used. See Status of bl.spamcannibal.org: DEAD.

Accordingly, I’ve knocked them out of the current version of 76-x-dns-blacklists-extra.

Meanwhile…

admin@vm1:/$ /srv/.all-sites/utils/rblinfo

  service                         sites     rejections
--------------------------------------------------------
  zen.spamhaus.org                   28            772
  all.s5h.net                         9             77
  b.barracudacentral.org             17             76
  ubl.unsubscore.com                 13             35
  all.spamrats.com                   15             24
  dbl.spamhaus.org                   17             23
  truncate.gbudb.net                 17             22
  bl.spamcop.net                     13             11
  hostkarma.junkemailfilter.com      13             10
  bl.mailspike.net                   15              7
  multi.uribl.com                    17              5
  dnsbl.dronebl.org                   7              1
  all.sh5.net                         1              0
  dnsbl.sorbs.net                     1              0
  excommunicado.co.uk                 2              0
  rhsbl.sorbs.net                     8              0
  TOTAL                               -          1,063
--------------------------------------------------------
  spamassassin                       28             24
  clamav                             28              1
--------------------------------------------------------
  v20180212 : ~0.20s

#27

Hi @alphacabbage1

I’m using symbiosis Jessie and after a bout of spam recently decided to give your rblinfo script a try.

I set it up the first time and rblinfo reported the service (only zen.spamhaus.org at the time along with the spamassisn and clamav with the script date and completion time as shown above.

But since I’ve added in the actual blacklists, when I run the script it only says

admin@XXX:~$ bash rblinfo

17 rbl services configured (non-spamhaus might ‘tag’)
1 rbl services show log rejection messages

And does not show anything else. mail is coming in and out fine still.
I guess I have a typo somewhere - any idea why it would stop reporting the services ( and Sa and clamav) ?

I’m also a little bit puzzled as late last year you say that you were intending to split the blacklists like so

/srv/.all-sites/config/blacklists/dnsbl-services.conf
/srv/.all-sites/config/blacklists/rhsbl-services.conf

But The 76-xxx file just has the rbl’s all in one place ? Did you get to implement checking the rbl’s list in the conf files as above or not ?

Thanks


#28

Hi @joolsr1

I’m assuming there’s a valid # /etc/exim4/symbiosis.d/10-acl/50-acl-check-rcpt/76-x-dns-blacklists-extra section in /etc/exim4/exim4.conf. Your config files sound fine – I haven’t made the potential change to the configuration structure. [1]

So, maybe the traffic you’re seeing isn’t listed on the services you’re using.

zgrep -i " blacklisted at " /var/log/exim4/rejectlog* should show any ip or rhsbl rejections.

To check that it’s working you could telnet to your server on 25 and target a mailbox using a known bad address, e.g., one taken from the excommunicado list.

EHLO test
MAIL FROM: <test@listed-domain.example.com>
RCPT TO: <my-brilliant-user@my-brilliant-site.com>
QUIT

Alternatively, exim’s delivery test mode may be useful.


[1] I’m still in two minds about the optimum configuration pattern. It seems a bit naff to hardcode the services (as now) but if it’s down to config file(s) I’m seeking an efficient lookup with a symbiosis-friendly way to allow individual service configurations (acting on specific return codes, perhaps negated) and actions (tag or reject). In theory, I think there can be one dnslists entry covering ip and sender domain lookups but I’m not sure how to set up appropriate rejection messages … so I let it float.


#29

Hi and thanks for your help

The extra blacklists are trapping spam so i’m sure everything is set up as it should be - but the only issue is that rblinfo is a bit ‘cut off’.
Just wondering - our main domain name has a ‘dash’ in it - that wouldn’t be tripping up your script could it ?

I did your other tests and the blacklist section is in place in /etc/exim4/exim4.com and i am seeing newly blacklisted traffic on other domains apart from spamhaus

So I’m at a loss. I might just give it a reboot - but it shouldnt be necessary …


#30

Ah, sorry - I did a good job of misreading your earlier post. So, interesting, no idea :wink: but there are several hyphenated domains here and I’ve not noticed any issues. If you’re not seeing anything after the 1 rbl services show log rejection messages it sounds like a silent crash, or a misplaced debug version. The last line should show version number and timing. Are you using the current 20180729?

admin@vm1:~$ /srv/.all-sites/utils/rblinfo

   16 rbl services configured (non-spamhaus might 'tag')
   13 rbl services show log rejection messages

  service                         sites     rejections
--------------------------------------------------------
  zen.spamhaus.org                   29            305
  all.s5h.net                        10             50
  b.barracudacentral.org             18             24
  ubl.unsubscore.com                 14             13
  truncate.gbudb.net                 18             10
  hostkarma.junkemailfilter.com      14              9
  multi.uribl.com                    18              7
  all.spamrats.com                   12              6
  dbl.spamhaus.org                   18              2
  bl.mailspike.net                   15              1
  bl.spamcop.net                     14              1
  dnsbl.dronebl.org                   8              1
  dyna.spamrats.com                   5              1
  rhsbl.sorbs.net                     8              0
  noptr.spamrats.com                  5              0
  dnsbl.sorbs.net                     1              0
  TOTAL                               -            430
--------------------------------------------------------
  spamassassin                       29             14
  clamav                             29              0
--------------------------------------------------------
  v20180729 : ~0.19s

#31

Yes my read out is as follows

/srv/rblinfo

14 rbl services configured (non-spamhaus might ‘tag’)
7 rbl services show log rejection messages

Then it stops …

Yes I’m using the current rblinfo file. When I first tried it the listing worked perfectly, it was only after i configured as such

Create the file /etc/exim4/symbiosis.d/10-acl/50-acl-check-rcpt/76-x-dns-blacklists-extra, with content as quoted below. ‘sudo’ is required to write to this directory, so as a windows head, I’d use ‘sudo nano /path/file’ under putty and paste from Notepad++ [make sure Edit->EOL Conversion is set to “Windows (CR LF)” to avoid some very funky substitutions].

cd /etc/exim4 and run sudo make. Under symbiosis-jessie this will automatically restart the service (assuming the basic syntax checks work).


#32

The plot thickens. It’ll probably require a debug version of rblinfo. First though, does including/excluding the excommunicado.co.uk service make any difference?


#33

Very odd : I removed a number of the entries in the blacklist folder and eventually when i removed rm truncate.gbudb.net and then reran rblinfo all the stats reappeared !

No idea why … I will add them back in with truncate.gbudb.net and see where we get to


#34

Added all entries back in the returned list from rblinfo fills up correctly accordingly. But when i add truncate.gbudb.net again the listing gets curtailed as before.

I will keep this entry off my list. Is there anything that may cause a problem with this particular url in the blacklists?

./rblinfo

17 rbl services configured (non-spamhaus might ‘tag’)
8 rbl services show log rejection messages

service sites rejections

zen.spamhaus.org 1 658
b.barracudacentral.org 1 14
all.s5h.net 1 9
dnsbl.sorbs.net 1 4
all.spamrats.com 1 2
ubl.unsubscore.com 1 2
dbl.spamhaus.org 1 1
multi.uribl.com 1 1
bl.mailspike.net 1 0
bl.spamcop.net 1 0
dnsbl.dronebl.org 1 0
dyna.spamrats.com 1 0
excommunicado.co.uk 1 0
hostkarma.junkemailfilter.com 1 0
noptr.spamrats.com 1 0
rhsbl.sorbs.net 1 0
spam.spamrats.com 1 0
TOTAL - 691

spamassassin 1 127
clamav 1 0
locally blacklisted sender - 0

v20180729 : ~0.85s


#35

I can’t think of anything that would explain it. I’ll PM to see if we can sort it…


#36

I’m not too bothered about using this particular one but thought you may want to know as yes something odd is going on :wink:

Thanks for your help so far. We’ve had an extra 33 emails blocked in a day thanks to the extra blacklists


#37

Sorry to dredge this up a few months later but I’m also getting this issue.

I implemented the spam solution above quite some time ago but only today added the rblinfo script. I also get the summary or with services are configured/have log entries, but no more than this.

I’m on Stretch. Did you discover anything more about why this might not be reporting?


#38

Oops, I’ve discovered that the blacklisting seems to occur in the exim4 mainlog and not the rejectlog. Is this to do with whether mail is tagged or outright rejected?

I’ve switched the log file to mainlog and am now getting the full report.


#39

Yes, ‘tag’ wont reject and the rblinfo script only looks for rejections [ via logfiles=("/var/log/exim4/rejectlog*") ]. To-date I’ve only manually scanned the main log to keep track of ‘tag’ hits and potential false positives when using new rbl services.

I didn’t manage to reproduce or understand @joolsr1’s ‘truncate’ issue but I’m guessing it’s not related.


#40

So if I switch to rejection, should I switch back to rejectlog. Or I guess (for a little extra overhead) I could use /var/log/exim/*log* ? :slight_smile:

Edit: Oh well that doesn’t quite work.