Last Update: 01/13/2012
Current Version = 1.1
Splunk (if you don't yet know it) is an incredibly powerful solution that collects, indexes and exploits any kind of data from any system, offering you as many solution as you need and even the possibility to create custom applications with graphical front-ends. (dashboards, reports, saved searches...)
In a few words, i am really impressed by Splunk, i think i've been looking for this for many many years!
Don't hesitate to take a look at main Splunk Website, you will easily find a lot of information and great documentations: http://www.splunk.com/
Splunk can be used for free with some little restrictions. (not more than 500Mb of input data per day)
I developed my first Splunk application "Splunk For Fail2ban" to provide a cool frontend and log managing tool associated with the well known and powerful Fail2ban tool. (take a look at my older post: http://youresuchageek.blogspot.fr/2012/11/howto-fail2ban-secure-your-network.html)
To install this addon, follow this link on Splunkbase or install it through the standard Splunk application process search online:
You can also get it from here, install through the Splunk Application Manager:
Splunk pre-requirements:
Ensure to install requirements Splunk addons:
Splunk For Fail2ban provides:
A complete Dashboard Overview of Fail2ban activity for all managed systems:
Overview of last 5 connection Fail2ban denied events, Number of events in selected Time Range
Fail2ban Alert trend per Jail, Top country origin of denied IP with graphical and data overview
Top Denied IP, Top denied per Jail with graphical and data overview
Top Fail2ban reporting servers for multi-hosts management, graphical and data overview
Google Maps Dashboard, identify the source of connexion attempts
A Fail2ban Event search interface with selection per kind of data (IPs, ID, Jail...)
Pre-defined major searches to get all the most important information
Denied Hosts by IP:
Denied Hosts by Jail:
Denied Hosts by Reporting Fail2ban server:
Top denied by IP:
Top Denied by Jail:
Top denied by IP with geo location:
Top denied by IP with geo location and DNS resolution:
Top denied by country origin:
Installation and utilization
Introduction
Installing and configuring Splunk is out of the scope of this post, still installing Splunk is really easy and well done, in 10 minutes you'll be done ^^
As a brieve description, here is how Splunk for Fail2ban works:
- We modify Fail2ban to add a specific message for each ban action and containing fields Splunk will analyse
- Through Syslog, we can manage as many Fail2ban servers as required
- Splunk collects our data and produces the IT intelligency
Installation and configuration will be done in a few steps:
1. Modifying Fail2ban configuration files related to the ban action (the goal is send fields we will analyse with Splunk)
2. Setting up Fail2ban to log to Syslog system
3. Setting up Syslog to trap custom Fail2ban events into a specific log file (can be local or remote Syslog if numerous Fail2ban hosts)
4. Installation and configuring Splunk for Fail2ban
Part 1: Configure Fail2ban
1. Set Fail2ban output to Syslog
I recommend the use of "rsyslog" as your main Syslog management, it comes with much more improvement than the standard Syslog. (http://www.rsyslog.com/)
First, we need to set Fail2ban to log its messages into Syslog instead of a standard log file.
To do so, edit "/etc/fail2ban/fail2ban.conf" and set:
logtarget = SYSLOG2. Modify Fail2ban configuration file for the ban action
Depending of your wish, you can set Fail2ban to use 1 of these 3 actions: (by editing /etc/fail2ban/jail.conf)
- action_ = Fail2ban will temporarely ban the IP source host
- action_mw = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request
- action_mwl = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request and log traces
In any case you need to modify the corresponding action configuration file to include a specific command line that will log any useful filed for Splunk.
Depending of your configuration, edit:
using the default action "action_" with the mta set to "mta = mail" :
- action_ --> Edit "/etc/fail2ban/action.d/mail.conf"
- action_mw --> Edit "/etc/fail2ban/action.d/mail-whois.conf"
- action_mwl --> Edit "/etc/fail2ban/action.d/mail-whois-lines.conf"
using the default action "action_" with the mta set to "mta = sendmail" :
- action_ --> Edit "/etc/fail2ban/action.d/sendmail.conf"
- action_mw --> Edit "/etc/fail2ban/action.d/sendmail-whois.conf"
- action_mwl --> Edit "/etc/fail2ban/action.d/sendmail-whois-lines.conf"
Insert a new command line for the action section called "actionban" and predecing the existing action just before the "printf" command:
logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] " &The action configuration file using mail "/etc/fail2ban/action.d/mail-whois-lines.conf" will look like this:
actionban = logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] " & \
printf %%b "Hi,\n
The IP <ip> has just been banned by Fail2Ban after <failures> attempts against <name>.\n\n
Here are more information about <ip>:\n
`whois <ip>`\n\n
Lines containing IP:<ip> in <logpath>\n
`grep '\<<ip>\>' <logpath>`\n\n
Regards,\n
Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip>" <dest>The action configuration file using sendmail "/etc/fail2ban/action.d/sendmail-whois-lines.conf" will look like this:
actionban = logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] " & \
printf %%b "Subject: [Fail2Ban] <name>: banned <ip>
Date: `date -u +"%%a, %%d %%h %%Y %%T +0000"`
From: Fail2Ban <<sender>>
To: <dest>\n
Hi,\n
The IP <ip> has just been banned by Fail2Ban after
<failures> attempts against <name>.\n\n
Here are more information about <ip>:\n
`/usr/bin/whois <ip>`\n\n
Lines containing IP:<ip> in <logpath>\n
`/bin/grep '\<<ip>\>' <logpath>`\n\n
Regards,\n
Fail2Ban" | /usr/sbin/sendmail -f <sender> <dest>So just set your action configuration file corresponding to your MTA with the configuration above, and restart Fail2ban.
3. Test Fail2ban
Now let's test your system, generate a ban event (try to log in through SSH with bad credentials) and check
your Syslog file to find the generated event. (look for the pattern "fail2ban.banevent")
You should find a ban event like this:
Jan 11 20:24:34 myhostname logger[30720]: [fail2ban.banevent]: fail2ban_host: [myfail2ban] Banhost: [xx.xx.xx.xx] jailname: [ssh] numberoffailures: [6] logmessage: [ Jan 11 20:24:32 myhostname sshd[30706]: Received disconnect from xx.xx.xx.xx: 11: Bye Bye [preauth] ] Now you're done with Fail2ban, let's configure Syslog ^^
Part 2: Configure Syslog - Standalone and Multi-Hosts
In 2 steps:
- if you want to manage different Fail2ban servers from Splunk, then read the Multiple Fail2ban client configuration note
- If you just one host to manage (Fail2ban and Splunk are installed in the same host), then just follow the common configuration section
MULTIPLE FAIL2BAN CLIENT CONFIGURATION NOTE: Remote and centralized Syslog configuration
Configuring Syslog to send events from a Syslog host to a remote Syslog server is out of the scope of this guide.
Therefore, if you want to collect fail2ban events from different hosts, you can choose between different solutions, as:
- Sending events using Syslog to a remote centralized Syslog
- Sending events from local log file using Splunk forwarder module
- Others (homemade scripts, file sharing...)
I would recommend using Rsyslog (default enhanced Syslog for many Linux systems) to achieve this, which is in deed easy enough, robust and efficient.
Here is in 2 steps a quick rsyslog centralized configuration: (remember to restart rsyslog after each modification)
1. In each client rsyslog host, modify "/etc/rsyslog.conf" and add a section to send any events to your Syslog server: (adapt the example IP)
"/etc/rsyslog.conf"
*.* @192.168.1.254:514 2. In syslog server configuration, create a configuration file that will trapp any remote client Syslog events and put then into a dedicated per host log file:
Ensure your configuration name will be read after the fail2ban syslog config file you will create after. (see above, this is very )
Create "/etc/rsyslog.d/10-fail2ban.conf" with the following content: (Note: The fail2ban config we will create after will be called 08 to be read before this and intercept messages)
"/etc/rsyslog.d/10-fail2ban.conf"
template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n"
:inputname, isequal, "imudp" ?PerHostLog;RemoteHostFileFormat
& ~Restart rsyslog after any config modification.
COMMON CONFIGURATION for Single and Multiple (for the centralized rsyslog server) Fail2ban installation:
1. Set Syslog to trap ban events to a dedicated logfile
This configuration part will depend on your system and needs, i recommend the use of "rsyslog"
The goal is to configure syslog to trap any event containing a key word "[fail2ban.banevent]" into a dedicated log file
In Debian/Ubuntu systeprintfms for example, create an rsyslog configuration file, example:
Create "/etc/rsyslog.d/08-fail2ban.conf" with the following content:
"/etc/rsyslog.d/08-fail2ban.conf"
:msg, contains, "[fail2ban.banevent]" /var/log/fail2ban_banevent.log
& ~Restart rsyslog to take effect:
sudo service rsyslog restart2. Generate a ban event and check your logfile
Generate a new ban event and check your log file, you should see a new ban event message!
If you are ok with that, then you're done with system configuration ^^
Part 3: Configuration of Splunk (the easy part!)
Here comes the easier part with no doubts :-)1. Create a new index "fail2ban_index" dedicated to Fail2ban inputGo to "manager", "Indexes", then create a new index with default params and call it: fail2ban_index
2. Configure Input fileGo to "manager", "Data Input" and configure MANUALLY a new input file pointing to your Fail2ban log file, with following settings:
Host:
You can let the default settings, it does not mind as we don't use it to recognize the fail2ban reporting server.
Source type:
- Set the source Type: Manual
- Source type: fail2ban_banevent
Index:
- Set the destination Index: fail2ban_index
Good news, you're done!!!
Just wait a few minutes to let Splunk get the content of your fail2ban log file, then go to the splunk application Splunk for Fail2ban
Don't hesitate to share any comment with me, this is my very first Splunk application and it may still needs some improvement :-)
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.