fail2ban + ansible

I use fail2ban to slow down and stop brute force attempts against my servers. One drawback of the way it is currently implemented is that if they are really persistant, attackers keep coming back.

If they have managed to trip fail2ban multiple times, that means they have either attacked two different servers, or have come back to the same one multiple times. This means that I am quite happy to permanently ban them from access to my systems.

This requires some setup on the servers in question.

auto lo
iface lo inet loopback

pre-up iptables-restore < /etc/firewall-rules
post-up iptables-restore -n < /etc/iptables.blocklist


*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FIREWALL - [0:0]
:blocklist - [0:0]
-A INPUT -j blocklist
-A INPUT -j FIREWALL
-A FORWARD -j FIREWALL
-A FIREWALL -i lo -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 25 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 81 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 110 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 143 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 443 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 465 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 587 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 993 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 995 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 4650 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 4949 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 22 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p udp -m udp --sport 53 -j ACCEPT
-A FIREWALL -p tcp -m tcp --sport 53 -j ACCEPT
-A FIREWALL -p udp -m udp --dport 53 -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 53 -j ACCEPT
-A FIREWALL -p udp -m udp --dport 123 -j ACCEPT
-A FIREWALL -p udp -m udp --sport 123 -j ACCEPT
-A FIREWALL -p udp -m udp --sport 6277 -j ACCEPT
-A FIREWALL -p udp -m udp --sport 24441 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A FIREWALL -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j REJECT --reject-with icmp-port-unreachable
-A FIREWALL -p udp -m udp -j REJECT --reject-with icmp-port-unreachable
-A FIREWALL -p icmp -j DROP
COMMIT

The file /etc/iptables.blocklist starts out pretty baren.

*filter
:blocklist - [0:0]
COMMIT

It took me a little while to figure out what the bare minimum needed configuration was to be able to 'source' an iptables file.

I use two ways to update the /etc/iptables.blocklist file. One is to use a short shell script, which also has an option to kick off the ansible update as well.

The append.sh script typically takes two arguments, append.sh -i <IPADDRESS> will add the IP before the COMMIT line at the end. append.sh -p will 'push' the configuration using ansible. There is one other option append.sh -f <FILENAME> -i <IPADDRESS> to specify a different blocklist filename.

The other is to use a perl script that parses the subject lines for all of the email that I get from fail2ban looking for the ip address in the subject line.

#!/bin/dash

# Copyright (c) 2013, grephead.com, LLC
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#     * Redistributions of source code must retain the above copyright
#       notice, this list of conditions and the following disclaimer.
#     * Redistributions in binary form must reproduce the above copyright
#       notice, this list of conditions and the following disclaimer in the
#       documentation and/or other materials provided with the distribution.
#     * Neither the name of grephead.com, LLC nor the names of its
#       contributors may be used to endorse or promote products derived from
#       this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY grephead.com, LLC "AS IS" AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL grephead.com, LLC BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

# append an IPv4 address to the blocklist iptable
# pushing the changes using ansible

# limited PATH
PATH=/bin:/usr/bin

# default filename
filename='iptables.blocklist'

while getopts i:f:p f
do
   case ${f} in 
      i)  ipaddress=${OPTARG} ;;
      f)  filename=${OPTARG} ;;
      p)  push=1 ;;
   esac 
done

# check to see if ${ipaddress} is zero length, not specified.
if [ -z ${ipaddress}  ];then
   echo 'Missing IP address, not updating file'
else 

   # check to make sure that the IP is not already blocked
   # if not, insert before COMMIT string

   if [ -f ${filename} ]; then
      if grep -q -- "-A blocklist -s ${ipaddress} -j DROP" ${filename}; then
            echo "${ipaddress} already blocked"
      else

sed -i.bak -e "/COMMIT/i \
-A blocklist -s ${ipaddress} -j DROP" iptables.blocklist

      fi
   else
      echo "${filename} does not exist"
   fi
fi

# if -p is specified, push updated blocklist using ansible playbook
if ( [ "${push}" = "1" ] && [ -f blocklist_playbook.yml ] )
then
   ansible-playbook blocklist_playbook.yml
fi

The perl script uses File::Slurp and Mail::Box modules.

#!/usr/bin/perl

# The MIT License (MIT)
# 
# Copyright (c) 2013 Matt Okeson-Harlow
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to
# deal in the Software without restriction, including without limitation the
# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
# sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.

use strict;
use warnings;
use Mail::Box::Manager;
use File::Slurp qw( :std :edit );

my ( %ip_lookup, %ip_count, @append );
my $count = 0;

## Directory and file locations
my $fail2ban_folder = qq{$ENV{HOME}/Maildir/.grephead.fail2ban};
my $trash_folder    = qq{$ENV{HOME}/Maildir/.Trash};
my $iptables        = qq{$ENV{HOME}/btsync/ansible/iptables.blocklist};

## mailbox handlers
my $mgr    = Mail::Box::Manager->new;
my $folder = $mgr->open( folder => $fail2ban_folder, access => q{rw} );
my $trash  = $mgr->open( folder => $trash_folder, access => q{rw} );

## Slurp iptables rules into an array
my @blocklist = read_file $iptables;

## process each message in the fail2ban folder, looking for 'banned $ip'
for my $email ( $folder->messages ) {
   my $subject = $email->subject;

   # [Fail2Ban] ssh: banned 122.226.109.178
   if ( $subject =~ m{banned \s+ (\d+[.]\d+[.]\d+[.]\d+)}xms ) {
      my $ip = $1;
      $ip_count{$ip}++;
      push @{ $ip_lookup{$ip} }, $count;
   }
   $count++;
}

## look for $num instances of an IP address, add it to the append
## array and move the related email to the trash.
for my $ip ( keys %ip_count ) {
   if ( $ip_count{$ip} > 1 and not grep /$ip/, @blocklist ) {
      print qq{$ip_count{$ip}\t$ip\n};
      my $iptables_line = qq{-A blocklist -s $ip -j DROP};
      push @append, $iptables_line;
      for my $index ( @{ $ip_lookup{$ip} } ) {
            print $index . qq{\t}
               . $folder->message($index)->subject . qq{\n};
            $mgr->moveMessage( $trash_folder, $folder->message($index) );
      }
   }
}

## find 'COMMIT' in iptables file, insert new rules before it
my $append_string = join "\n", @append;
edit_file sub {s/(COMMIT)/$append_string\n$1/g}, $iptables;

After the fail2ban_filter.pl script is run, the append.sh -p script needs to be run to push the changes using ansible or run the ansible-playbook blocklist_playbook.yml command.

[grephead]
vader.grephead.com
yoda.grephead.com
leia.grephead.com
# push out updated blocklist iptables rules and add them to the running
# firewall

# 2013/04/16 02:17:35 

---
- hosts: grephead
sudo: yes
connection: ssh

tasks:
   - name: update iptables.blocklist
      action: copy src=iptables.blocklist dest=/etc/iptables.blocklist owner=root group=root mode=0644

   - name: merge into iptables
      action: shell /sbin/iptables-restore -n < /etc/iptables.blocklist

# vim: set ts=2:sw=2:sts=2:ft=yaml