Splunk Howto - Splunk for Fail2ban, get a the Fail2ban Multi-host frontend with Splunk!

Last Update: 01/13/2012

Current Version = 1.1

Splunk (if you don't yet know it) is an incredibly powerful solution that collects, indexes and exploits any kind of data from any system, offering you as many solution as you need and even the possibility to create custom applications with graphical front-ends. (dashboards, reports, saved searches...)

In a few words, i am really impressed by Splunk, i think i've been looking for this for many many years!

Don't hesitate to take a look at main Splunk Website, you will easily find a lot of information and great documentations: http://www.splunk.com/

Splunk can be used for free with some little restrictions. (not more than 500Mb of input data per day)

Details: http://docs.splunk.com/Documentation/Splunk/5.0.1/Admin/TypesofSplunklicenses

I developed my first Splunk application "Splunk For Fail2ban" to provide a cool frontend and log managing tool associated with the well known and powerful Fail2ban tool. (take a look at my older post: http://youresuchageek.blogspot.fr/2012/11/howto-fail2ban-secure-your-network.html)

To install this addon, follow this link on Splunkbase or install it through the standard Splunk application process search online: 

http://splunk-base.splunk.com/apps/72087/splunk-for-fail2ban

You can also get it from here, install through the Splunk Application Manager:

SplunkForFail2ban V1.1

Splunk pre-requirements:

Ensure to install requirements Splunk addons:

• Sideview utils addon
• Google Maps addon
• Maxmind geo location addon

Splunk For Fail2ban provides:

A complete Dashboard Overview of Fail2ban activity for all managed systems: 

Overview of last 5 connection Fail2ban denied events, Number of events in selected Time Range

Fail2ban Alert trend per Jail, Top country origin of denied IP with graphical and data overview

Top Denied IP, Top denied per Jail with graphical and data overview

Top Fail2ban reporting servers for multi-hosts management, graphical and data overview

Google Maps Dashboard, identify the source of connexion attempts

A Fail2ban Event search interface with selection per kind of data (IPs, ID, Jail...)

Pre-defined major searches to get all the most important information

Denied Hosts by IP:

Denied Hosts by Jail:

Denied Hosts by Reporting Fail2ban server:

Top denied by IP:

Top Denied by Jail:

Top denied by IP with geo location:

Top denied by IP with geo location and DNS resolution:

Top denied by country origin:

Installation and utilization

Introduction

Installing and configuring Splunk is out of the scope of this post, still installing Splunk is really easy and well done, in 10 minutes you'll be done ^^

Download Splunk !

As a brieve description, here is how Splunk for Fail2ban works:

- We modify Fail2ban to add a specific message for each ban action and containing fields Splunk will analyse

- Through Syslog, we can manage as many Fail2ban servers as required

- Splunk collects our data and produces the IT intelligency

Installation and configuration will be done in a few steps:

1. Modifying Fail2ban configuration files related to the ban action (the goal is send fields we will analyse with Splunk)

2. Setting up Fail2ban to log to Syslog system

3. Setting up Syslog to trap custom Fail2ban events into a specific log file (can be local or remote Syslog if numerous Fail2ban hosts)

4. Installation and configuring Splunk for Fail2ban

Part 1: Configure Fail2ban

1. Set Fail2ban output to Syslog

I recommend the use of "rsyslog" as your main Syslog management, it comes with much more improvement than the standard Syslog. (http://www.rsyslog.com/)

First, we need to set Fail2ban to log its messages into Syslog instead of a standard log file.

To do so, edit "/etc/fail2ban/fail2ban.conf" and set:

logtarget = SYSLOG

2. Modify Fail2ban configuration file for the ban action

Depending of your wish, you can set Fail2ban to use 1 of these 3 actions: (by editing /etc/fail2ban/jail.conf)

• action_ = Fail2ban will temporarely ban the IP source host
• action_mw = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request
• action_mwl = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request and log traces

In any case you need to modify the corresponding action configuration file to include a specific command line that will log any useful filed for Splunk.

Depending of your configuration, edit:

using the default action "action_" with the mta set to "mta = mail" :

• action_ --> Edit "/etc/fail2ban/action.d/mail.conf"
• action_mw --> Edit "/etc/fail2ban/action.d/mail-whois.conf"
• action_mwl --> Edit "/etc/fail2ban/action.d/mail-whois-lines.conf"

using the default action "action_" with the mta set to "mta = sendmail" :

• action_ --> Edit "/etc/fail2ban/action.d/sendmail.conf" 
• action_mw --> Edit "/etc/fail2ban/action.d/sendmail-whois.conf"
• action_mwl --> Edit "/etc/fail2ban/action.d/sendmail-whois-lines.conf"

Insert a new command line for the action section called "actionban" and predecing the existing action just before the "printf" command:

logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] " &

The action configuration file using mail "/etc/fail2ban/action.d/mail-whois-lines.conf" will look like this:

actionban = logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
            Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
            logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] " & \
     printf %%b "Hi,\n
     The IP <ip> has just been banned by Fail2Ban after <failures> attempts against <name>.\n\n
     Here are more information about <ip>:\n
     `whois <ip>`\n\n
     Lines containing IP:<ip> in <logpath>\n
     `grep '\<<ip>\>' <logpath>`\n\n
     Regards,\n
     Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip>" <dest>

The action configuration file using sendmail "/etc/fail2ban/action.d/sendmail-whois-lines.conf" will look like this:

actionban = logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
            Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
            logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] " & \
            printf %%b "Subject: [Fail2Ban] <name>: banned <ip>
            Date: `date -u +"%%a, %%d %%h %%Y %%T +0000"`
            From: Fail2Ban <<sender>>
            To: <dest>\n
            Hi,\n
            The IP <ip> has just been banned by Fail2Ban after
            <failures> attempts against <name>.\n\n
            Here are more information about <ip>:\n
            `/usr/bin/whois <ip>`\n\n
            Lines containing IP:<ip> in <logpath>\n
            `/bin/grep '\<<ip>\>' <logpath>`\n\n
            Regards,\n
            Fail2Ban" | /usr/sbin/sendmail -f <sender> <dest>

So just set your action configuration file corresponding to your MTA with the configuration above, and restart Fail2ban.

3. Test Fail2ban 

Now let's test your system, generate a ban event (try to log in through SSH with bad credentials) and check

your Syslog file to find the generated event. (look for the pattern "fail2ban.banevent")

You should find a ban event like this:

Jan 11 20:24:34 myhostname logger[30720]: [fail2ban.banevent]: fail2ban_host: [myfail2ban] Banhost: [xx.xx.xx.xx] jailname: [ssh] numberoffailures: [6] logmessage: [ Jan 11 20:24:32 myhostname sshd[30706]: Received disconnect from xx.xx.xx.xx: 11: Bye Bye [preauth] ] 

Now you're done with Fail2ban, let's configure Syslog ^^

Part 2: Configure Syslog - Standalone and Multi-Hosts

In 2 steps:

• if you want to manage different Fail2ban servers from Splunk, then read the Multiple Fail2ban client configuration note
• If you just one host to manage (Fail2ban and Splunk are installed in the same host), then just follow the common configuration section

MULTIPLE FAIL2BAN CLIENT CONFIGURATION NOTE: Remote and centralized Syslog configuration

Configuring Syslog to send events from a Syslog host to a remote Syslog server is out of the scope of this guide.

Therefore, if you want to collect fail2ban events from different hosts, you can choose between different solutions, as:

• Sending events using Syslog to a remote centralized Syslog
• Sending events from local log file using Splunk forwarder module
• Others (homemade scripts, file sharing...)

I would recommend using Rsyslog (default enhanced Syslog for many Linux systems) to achieve this, which is in deed easy enough, robust and efficient.

Here is in 2 steps a quick rsyslog centralized configuration: (remember to restart rsyslog after each modification)

1. In each client rsyslog host, modify "/etc/rsyslog.conf" and add a section to send any events to your Syslog server: (adapt the example IP)

"/etc/rsyslog.conf"

*.* @192.168.1.254:514 

2. In syslog server configuration, create a configuration file that will trapp any remote client Syslog events and put then into a dedicated per host log file:

Ensure your configuration name will be read after the fail2ban syslog config file you will create after. (see above, this is very )

Create "/etc/rsyslog.d/10-fail2ban.conf" with the following content: (Note: The fail2ban config we will create after will be called 08 to be read before this and intercept messages)

"/etc/rsyslog.d/10-fail2ban.conf"

template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n" 
:inputname, isequal, "imudp" ?PerHostLog;RemoteHostFileFormat
& ~

Restart rsyslog after any config modification.

COMMON CONFIGURATION for Single and Multiple (for the centralized rsyslog server) Fail2ban installation: 

1. Set Syslog to trap ban events to a dedicated logfile

This configuration part will depend on your system and needs, i recommend the use of "rsyslog"

The goal is to configure syslog to trap any event containing a key word "[fail2ban.banevent]" into a dedicated log file

In Debian/Ubuntu systeprintfms for example, create an rsyslog configuration file, example:

Create "/etc/rsyslog.d/08-fail2ban.conf" with the following content: 

"/etc/rsyslog.d/08-fail2ban.conf"

:msg, contains, "[fail2ban.banevent]" /var/log/fail2ban_banevent.log
& ~

Restart rsyslog to take effect:

sudo service rsyslog restart

2. Generate a ban event and check your logfile

Generate a new ban event and check your log file, you should see a new ban event message! 

If you are ok with that, then you're done with system configuration ^^ 

Part 3: Configuration of Splunk (the easy part!)

Here comes the easier part with no doubts :-)

1. Create a new index "fail2ban_index" dedicated to Fail2ban input

Go to "manager", "Indexes", then create a new index with default params and call it: fail2ban_index

2. Configure Input file

Go to "manager", "Data Input" and configure MANUALLY a new input file pointing to your Fail2ban log file, with following settings:

Host:

You can let the default settings, it does not mind as we don't use it to recognize the fail2ban reporting server.

Source type:

- Set the source Type: Manual
- Source type: fail2ban_banevent

Index:

- Set the destination Index: fail2ban_index

Good news, you're done!!!
Just wait a few minutes to let Splunk get the content of your fail2ban log file, then go to the splunk application Splunk for Fail2ban

Don't hesitate to share any comment with me, this is my very first Splunk application and it may still needs some improvement :-)

Howto : Fail2ban, Secure your Network services with Fail2ban and ban malicious hosts

Last Update: 11/13/2012


The Goal: 

Fail2Ban is a very flexible and powerful tool to help you improving your security by temporarily banishing hosts IPs trying to have malicious activity against your services. (brute force attack...)

Easy to install and configure, Fail2Ban will permanently read your files logs and look for suspicious connection attempts or activity, based on provided / custom rules.

Depending on your configuration, Fail2Ban will ban the remote IP using Netfilter rules (iptables) and then unban it depending on your settings.

This a very great tool you MUST have in your server if you opened any service like ssh, http/https with basic authentication, smtp...and much more.


Major sources: 

Official Fail2ban Wiki:
http://www.fail2ban.org/wiki/index.php/Main_Page

Various articles:
http://www.fanatips.org/Apache_%2F%2F_Nginx_tips/Secure_Nginx_with_fail2ban
http://snippets.aktagon.com/snippets/554-How-to-Secure-an-nginx-Server-with-Fail2Ban

Summary of steps: 

Step 1: Requirements

Step 2: Installation and base configuration

Step 3: Enabling services

Step 4: Test your configuration

Step 5: Adding new customized services, example with Nginx Web Server

FAQ & Issues

Step 1: Requirements

Fail2ban is quite easy to install and setup, still depending on your needs you have some requirements:

Iptables:

Fail2ban will require iptables to banish suspicious hosts, it is not a requirement to setup your iptables Firewall configuration (even if recommended).

Fail2ban will insert a drop line, whenever you are using iptables or not.

It is probably installed by defaut (use the command "iptables" to check), if not:

$ sudo apt-get install iptables

MTA configuration for mail alerting:

If you want Fail2ban to able to send you alerting mails in case of action done (banish, unbanish...), then you need to setup your MTA.

Your system probably only be a client of a mail central hub (eg. your ISP), if this is the case i recommend to install "ssmtp" wich is very light and simple to setup, but you may prefer to use sendmail, postfix or anything else.

If your system is a mail server itself, i guess you already know how to configure it :-)

Installation: 

$ sudo apt-get install ssmtp

Configuration (real simple!): 

Edit "/etc/ssmtp/ssmtp.conf" and set:

- mailhub: Define here the smtp server of your local ISP (eg. myisp.smtp.com)

- rewriteDomain: if you want the sender domain to be rewritten with the value of your choice

For most of cases, this will be the only modifications required, in case of need see ssmtp configuration pages. (ask Google ^^)

Quick send test:

$ echo "Testing" | mail -s Subject dest@domain.com

If you receive the mail then you're done. (check /var/log/mail* if required)

Step 2: Installation and base configuration

Installation:

Very simple, For Debian/Ubuntu based systems:

$ sudo apt-get install fail2ban

Initial Configuration:

Edit main Fail2Ban configuration file: "/etc/fail2ban/jail.conf" and define:

1. Local hosts to ignore:

If you want to ignore some local hosts, add them to your configuration, example with 192.168.0.0/24 local network:

# "ignoreip" can be an IP address, a CIDR mask or a DNS host
ignoreip = 127.0.0.1/8 192.168.0.0/24
bantime  = 600
maxretry = 3

2. Mail alert recipient:

If you want to send alter when an action has been done, set the recipient

# Destination email address used solely for the interpolations in
# jail.{conf,local} configuration files.
destemail = admin@mydomain.com

3. MTA:

Sendmail is defined as default command called for sending mails, both should work but i prefer to call standard "mail" command:

# email action. Since 0.8.1 upstream fail2ban uses sendmail
# MTA for the mailing. Change mta configuration parameter to mail
# if you want to revert to conventional 'mail'.
mta = mail

4. Default action when a host has banished

In default configuration, when a host has been banned, the information is just logged and no mail report will be sent.

If you want a report to be sent by mail (associated with a whois request report for the concerned IP), set:

action = %(action_mw)s

If you want a report to be sent by mail with a log extract (associated with a whois request report for the concerned IP), set:

action = %(action_mwl)s

Step 3: Enabling services

SSH example:

The next step will be to define which Network services you will want to supervise, in default configuration Fail2Ban will only monitor SSH for both login failed and DDOS attacks, extract:

[ssh]

enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 6

The import thing here will be the enabled value set to "true" (^^) but also the port, in case of host banishing the iptables rules will be created using this port.

Fail2ban will create multiport iptables rules, so you can add as many ports as you require.
Example if your SSH is running on several port (example 22 for internal and OTHER for external purposes) then you can set:

[ssh]

enabled  = true
port     = ssh,OTHER
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 6

Other services important vars values:

In each service section, you can overwrite default vars values or set specific ones for your needs.
Here are some you may need to set.

1. Protocol type:

The default protocol if not set will be tcp, you can set it to "udp" or "all" if you need to block bother tcp/udp ports.

protocol = all

2. Ban time:

You may want to overwrite the default bantime, then you just need to it in the service concerned section, time is set in seconds, example for 10 minutes banishment:

bantime = 600 # 10 minutes

3. Ports:

As seen before with the SSH example, you can set as many ports as you required in the port section.

You can call them by names if they are present in /etc/services or by the port number.

Each port must be separated by a coma.

4. Action:

Defaults actions in response of matched events are defined in jail.conf, but you can overwrite values for each specific network service.

In the following example, let's say our default configuration sends emails alert with log extract ("action = %(action_mwl)s") but for a specific Network service we just it to be banned without any altering.

Then you will overwrite the action value in your service section:

action_ = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s

Enable your services:

Depending on your configuration, you need to enable Network services as you require.
Follow each section to enable if required.

If you need to create your own service, follow the example of Nginx as above.

Step 4: Testing your configuration

Testing your configuration is very simple :-)

Just test accessing your SSH host outside of your local network (using your Smartphone or whatever you want!) and put false credentials.

Upon the max retry failures, your host should have been banished, you will see in fail2ban logs:

/var/log/fail2ban.log

2012-11-13 10:41:43,487 fail2ban.actions: WARNING [ss] Ban xxx.xxx.xxx.xxx

Locally, a new iptables rule has been created inside the jail dedicated iptables section (automatically created by fail2ban at boot time):

Chain fail2ban-ssh (1 references)
target     prot opt source               destination         
DROP       all  --  37.160.52.0          anywhere            
RETURN     all  --  anywhere             anywhere            

And you should (if you set it) have received a mail with all required information.

Depending on your configuration, XX minutes/days or whatever after having been ban, the host will be unban and the iptables drop rule will be removed.

Powerful, simple, beautiful :-)

Step 5: Adding new customized services, example with nginx Web Server

One of very great things with Fail2ban is that you can easily add any customized service you would require, you just add to define regex expression that will be used to match patterns and define Network section.

Here comes a full example for Nginx Web Server, we will monitor basic authentication failures, badbots, prevent script execution... and off course this can be easily customized to your needs.

Nginx Web Server Fail2Ban integration:

1. Create Nginx filters

Create each required file with its content:

/etc/fail2ban/filter.d/nginx-auth.conf

#
# Auth filter /etc/fail2ban/filter.d/nginx-auth.conf:
#
# Blocks IPs that fail to authenticate using basic authentication
#
[INCLUDES]                                                                                                                    
                                                                                                                              
# Read common prefixes. If any customizations available -- read them from                                                     
# common.local                                                                                                                
before = common.conf 

[Definition]
 
failregex = no user/password was provided for basic authentication.*client: <HOST>
            user .* was not found in.*client: <HOST>
            user .* password mismatch.*client: <HOST>
  
ignoreregex =

/etc/fail2ban/filter.d/nginx-login.conf

#
# Login filter /etc/fail2ban/filter.d/nginx-login.conf:
#
# Blocks IPs that fail to authenticate using web application's log in page
#
# Scan access log for HTTP 200 + POST /sessions => failed log in
[INCLUDES]                                                                                                                    
                                                                                                                              
# Read common prefixes. If any customizations available -- read them from                                                     
# common.local                                                                                                                
before = common.conf 

[Definition]
failregex = ^<HOST> -.*POST /sessions HTTP/1\.." 200
ignoreregex =

/etc/fail2ban/filter.d/nginx-noscript.conf

# Noscript filter /etc/fail2ban/filter.d/nginx-noscript.conf:
#
# Block IPs trying to execute scripts such as .php, .pl, .exe and other funny scripts.
#
# Matches e.g.
# 192.168.1.1 - - "GET /something.php
#
[INCLUDES]                                                                                                                    
                                                                                                                              
# Read common prefixes. If any customizations available -- read them from                                                     
# common.local                                                                                                                
before = common.conf 

[Definition]
failregex = ^<HOST> -.*GET.*(\.php|\.asp|\.exe|\.pl|\.cgi|\scgi)
ignoreregex =

/etc/fail2ban/filter.d/nginx-proxy.conf

# Proxy filter /etc/fail2ban/filter.d/nginx-proxy.conf:
#
# Block IPs trying to use server as proxy.
#
# Matches e.g.
# 192.168.1.1 - - "GET http://www.something.com/
#
[INCLUDES]                                                                                                                    
                                                                                                                              
# Read common prefixes. If any customizations available -- read them from                                                     
# common.local                                                                                                                
before = common.conf 

[Definition]
failregex = ^<HOST> -.*GET http.*
ignoreregex =

2. Create Nginx configuration

Edit Fail2Ban configuration  "/etc/fail2ban/jail.conf" and add:

Note: 

If Nginx is listening to non standard http/https port, think to adapt to your configuration

Also, nginx-noscript will prevent any script execution on your sites, if this is not what you need just comment out this section.

/etc/fail2ban/jail.conf

[nginx-auth]
enabled = true
filter = nginx-auth
port = http,https
logpath = /var/log/nginx*/*error*.log
bantime = 600 # 10 minutes
maxretry = 6
 
[nginx-login]
enabled = true
filter = nginx-login
port = http,https
logpath = /var/log/nginx*/*access*.log
bantime = 600 # 10 minutes
maxretry = 6
  
[nginx-badbots]
enabled  = true
filter = apache-badbots
port = http,https
logpath = /var/log/nginx*/*access*.log
bantime = 86400 # 1 day
maxretry = 1
  
[nginx-noscript]
enabled = true
port = http,https
filter = nginx-noscript
logpath = /var/log/nginx*/*access*.log
axretry = 6
bantime  = 86400 # 1 day
  
[nginx-proxy]
enabled = true
port = http,https
filter = nginx-proxy
logpath = /var/log/nginx*/*access*.log
maxretry = 0
bantime  = 86400 # 1 day

Finally, restart fail2ban and you're done! (sudo service fail2ban restart)

To test your configuration, just try to access to your basic authentification protected Web Site with bad credentials, as for the SSH example your host will be ban and you'll get the report :-)

FAQ & Issues

• How to prevent Fail2ban from sending alerts mails when stopped/started

When email alerting is configured, fail2ban will send you one mail per jail configured (one per Network service) for each action (stop/start).

This can be as usefull as boring :-)

If you want to disable this behavior:

- Go in "/etc/fail2ban/actions.d/"

- You will find 6 configurations files related to sendmail & mail, depending on what you set (sendmail vs mail) you just have to comment out actionstart and actionstop:

"action_": comment "actionstart" & "actionstop" in action.d/sendmail.conf
"action_mw": comment "actionstart" & "actionstop" in action.d/sendmail-whois.conf
"action_mwl": comment "actionstart" & "actionstop" in action.d/sendmail-whois-lines.conf

"action_": comment "actionstart" & "actionstop" in action.d/mail.conf
"action_mw": comment "actionstart" & "actionstop" in action.d/mail-whois.conf
"action_mwl": comment "actionstart" & "actionstop" in action.d/mail-whois-lines.conf

- Restart fail2ban, you're done!