User Tools

Site Tools


howtos:url_spam_block_postfix

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
howtos:url_spam_block_postfix [08/08/2006 09:45] domingohowtos:url_spam_block_postfix [02/12/2018 21:34] (current) – external edit 127.0.0.1
Line 1: Line 1:
 +====== Postfix body_checks script ======
  
 +This script downloads two lists of URL's that has been spotted in spam mails and reported to http://spamvertised.abusebutler.com and http://www.spamcop.net.
 +
 +The idea is that if this URL is found in the mail it must be spam and thus discarded.
 +
 +Run it as a cronjob every 5 minutes (or more often if you like).
 +
 +**Beware: Data is supplied by humans and not all URL's are "dangerous" - I have seen legitimate ones like w3.org and symantec.com **
 +
 +===== main.cf/master.cf configuration =====
 +
 +Insert the following into main.cf:
 +
 +<code>
 +body_checks = pcre:/etc/postfix/advanced_block_urls
 +</code>
 +
 +You may want to have a second smtpd instance running for outgoing mail to save resources on your gateway.
 +Put the following into master.cf:
 +
 +<code>
 +192.168.1.1:10050 inet n        -        n        -        -  smtpd
 +      -o receive_override_options=no_header_body_checks
 +</code>
 +
 +This will open a second smtp daemon listening on port 10050. The IP 192.168.1.1 should reflect your own IP address. When routing mail from the inside through the gateway just forward the mail to port 10050 instead of 25. That's it.
 +
 +===== The script =====
 +The following script extracts the domain from the two sources and discards subdomains and/or directories and not completed (when they are too long and shortend as www.longdomain/andsomemore...).
 +
 +Also I have tried to take care of some nations way of using commercial domains like UK (bbcnews.co.uk) and ZA (www.iol.co.za). If you know of others doing the same, just let me know and I correct the script to get more usefull data out of it.
 +
 +Please let  [[domingo@domingo.dk|me]] know if it doesn't work or you have good ideas.
 +
 +<file>
 +#!/bin/bash
 +
 +DOWNLOAD_DIRECTORY="/etc/postfix"
 +ABUSEBUTLER_URL="http://spamvertised.abusebutler.com/spamvertised.php?rep=last24"
 +SPAMCOP_URL="http://www.spamcop.net/w3m?action=inprogress&type=www"
 +OUT_FILE="bad_url_list"
 +OUT_FILE_IP="bad_url_list_ip"
 +PRE_POSTFIX_MAP_FILE="pre_block_urls"
 +OUT_FILE_FULL_PATH=$DOWNLOAD_DIRECTORY/$OUT_FILE
 +POSTFIX_MAP_FILE="advanced_block_urls"
 +POSTFIX_MAP_FILE_FULL_PATH=$DOWNLOAD_DIRECTORY/$POSTFIX_MAP_FILE
 +
 +rm $OUT_FILE_FULL_PATH
 +rm $POSTFIX_MAP_FILE_FULL_PATH
 +rm $DOWNLOAD_DIRECTORY/$PRE_POSTFIX_MAP_FILE
 +rm $DOWNLOAD_DIRECTORY/$OUT_FILE_IP
 +
 +#Get URL that consist of IP addresses and remove any ports
 +lynx -dump $ABUSEBUTLER_URL |egrep -v -i "uptime|abusebutler|spamvertised"|grep http |cut -c 14- | egrep "[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}" | cut -d: -f1 | sort -u > $DOWNLOAD_DIRECTORY/$OUT_FILE_IP
 +
 +#Get normal URL without IP addresses
 +lynx -dump $ABUSEBUTLER_URL |egrep -v -i "uptime|abusebutler|spamvertised"|grep http |cut -c 14- | fgrep -v ... | egrep -v "[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}" | sort -u > $OUT_FILE_FULL_PATH
 +
 +#Strip down to single domain
 +for x in $( cat $OUT_FILE_FULL_PATH ); do
 +        SUBDOMAIN=""
 +        TLD=`echo -n $x | awk -F "." '{elems=split($0, a); print $elems}'` >/dev/null 2>&1
 +        DOMAIN=`echo -n $x | awk -F "." '{elems=split($0, a);elems-- ; print $elems}' >/dev/null 2>&1
 +         if [ $DOMAIN.$TLD = co.uk -o $DOMAIN.$TLD = co.za ]; then
 +                SUBDOMAIN=`echo -n $z | awk -F "." '{elems=split($0, a);elems-- ;elems-- ; print $elems}'`
 +                DOMAIN=$DOMAIN.$SUBDOMAIN
 +         fi
 +        echo $DOMAIN.$TLD >> $DOWNLOAD_DIRECTORY/$PRE_POSTFIX_MAP_FILE
 +done
 +
 +#Make mapfile from Abusebutler
 +for y in $( cat $DOWNLOAD_DIRECTORY/$PRE_POSTFIX_MAP_FILE ); do
 +        echo "/$y        REJECT Abusebutler body_check failed" >> $POSTFIX_MAP_FILE_FULL_PATH
 +done
 +
 +#Get URL that consist of IP addresses and remove any ports
 +lynx -dump -nolist $SPAMCOP_URL | sed -n 's/.*http\:\/\/\(.*\).*/\1/p'| sed 's/\/$//' | cut -d/ -f1 | egrep "[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}" | cut -d: -f1 | sort -u >>  $DOWNLOAD_DIRECTORY/$OUT_FILE_IP
 +
 +#Get normal URL's without IP addresses
 +lynx -dump -nolist $SPAMCOP_URL | sed -n 's/.*http\:\/\/\(.*\).*/\1/p'| sed 's/\/$//' | cut -d/ -f1 | fgrep -v ... | egrep -v "[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}" | cut -d: -f1 | sort -u > $OUT_FILE_FULL_PATH
 +
 +#Strip down to single domain
 +for z in $( cat $OUT_FILE_FULL_PATH ); do
 +        SUBDOMAIN=""
 +        TLD=`echo -n $z | awk -F "." '{elems=split($0, a); print $elems}' ##>/dev/null 2>&1
 +        DOMAIN=`echo -n $z | awk -F "." '{elems=split($0, a);elems-- ; print $elems}'` ## >/dev/null 2>&1
 +         if [ $DOMAIN.$TLD = co.uk -o $DOMAIN.$TLD = co.za  ]; then
 +                SUBDOMAIN=`echo -n $z | awk -F "." '{elems=split($0, a);elems-- ;elems-- ; print $elems}'`
 +                DOMAIN=$DOMAIN.$SUBDOMAIN
 +         fi
 +
 +        echo $DOMAIN.$TLD >> $DOWNLOAD_DIRECTORY/$PRE_POSTFIX_MAP_FILE
 +
 +
 +done
 +
 +#Make mapfile from SpamCop
 +for b in $( cat $DOWNLOAD_DIRECTORY/$PRE_POSTFIX_MAP_FILE ); do
 +        b=`echo -n $b |cut -d/ -f1`  >/dev/null 2>&1
 +        echo "/$b/        REJECT SpamCop body_checks failed" >> $POSTFIX_MAP_FILE_FULL_PATH
 +done
 +
 +#Needs to reload Postfix to read the new file
 +/usr/sbin/postfix reload >/dev/null 2>&1
 +</file>
 +
 +FIXME Doesn't handle URL's when it is an ip address. Those are put in a seperat file for now and not used.
 +
 +FIXME You have to reload Postfix everytime the files are updated. This is not feasible. The URL's should be put into a MySQL instead.
 +
 +FIXME Doesn't handle mistyped URL's consisting of partial ip addresses.