Howto: Raspberry Pi Root NFS share - boot your System over NFS share and definitively deal with Flash data corruption

The Goal: 

If you have a Raspberry Pi and a Linux Server (or a NAS), you shall really be interested by this post! :-)

I had a lot of issue with my main Rpi when overclocked generating File System data corruption...

And finally, the real best solution, solid and efficient has been to convert my root installation into booting rootfs over NFS.

I recommend the most easy solution to first have a running installation of your system into your Flash card and simply migrate it to root fs over NFS.

Since i've done this, never had any system freeze, corruption or event kernel panic with my Rpi overclocked to turbo mode :-)

Also, you should ensure before beginning that your system is up to date (sudo apt-get update && sudo apt-get dist-upgrade -f) and your have the last firmware version (sudo rpi-update)

Major source: 

• http://raspberrypi.org/phpBB3/viewtopic.php?f=63&t=5974
• http://raspberrypi.stackexchange.com/questions/628/how-do-i-configure-the-raspberry-pi-to-boot-with-an-nfs-root

Summary of steps: 

Step 1: Set your NFS share
Step 2: Copy your root fs into your NFS share
Step 3: Modify your Raspberry Pi boot configuration
Step 4: Adapt your Rpi fstab
Step 5: Boot your Rpi!

Step 6: Correct your swap configuration by migrating to a loop device

Memorandum

Step 1: Set your NFS share

If you have a Linux Home Server or NAS, then you probably already share data using NFS.

To set a NFS share dedicated for your Rpi Root fs, add your share into "/etc/exports":

Adapt <raspberrypi_ip> with your Rpi LAN IP or LAN subnet if you prefer

# Raspberry Root FS                                     
/data/rpi_rootfs <raspberrypi_ip>(rw,sync,no_root_squash,no_subtree_check)

Under Debian / Ubuntu, reload your NFS server config:

sudo /etc/init.d/nfs-kernel-server reload

Step 2: Copy your Rpi root fs into your NFS share

Then simply copy all of your Rpi root fs into your new nfs share, you can do it directly under the Rpi or by plugging your Flash card into a client computer:

Example with a client computer having the flash card and NFS share mounted:

cp -rav /media/mmcblk0p2/* /data/rpi_rootfs/ 

Step 3: Modify your Raspberry Pi boot configuration

The only partition you will need to keep in your Rpi Flash card will be the boot partition (first partition), containing main boot configuration files and the Rpi firmware.

To boot over NFS, we need to modify the file "/boot/cmdline.txt" (contained into the first fat partition of your Flash card) to add/correct some sections:

• root= --> Will be pointing to "/dev/nfs"
• nfsroot=<nfs_server_ip>:/data/rpi_rootfs,udp,vers=3 ip=dhcp (replace<nfs_server_ip> with your NFS server IP)
• rootfstype=nfs
• smsc95xx.turbo_mode=N --> is a workaround to prevent kernel panic under high network load (i recommend this)

Note:

In this example, we use DHCP to set the Rpi Lan IP at boot time, this is in my opinion the easiest way to do as you preset a fix address in your DHCP server for your Rpi.

Still you can also manually a fix IP at boot time.

Also note we will be using NFS V3 running under UDP for better performances. (see Memorandum for performances fine tuning)

"cmdline.txt" example with DHCP (the file must contain only one line):

Replace:

• <nfs_server_ip> with the NFS server IP

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1
root=/dev/nfs nfsroot=<nfs_server_ip>:/data/rpi_rootfs,udp,vers=3
ip=dhcp rootfstype=nfs smsc95xx.turbo_mode=N 

"cmdline.txt" example with Fix IP(the file must contain only one line):

Replace:

• <raspberrypi_ip> with the Lan IP of your Rpi
• <nfs_server_ip> with the NFS server IP
• <default_gateway> with the IP of your local gateway

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1
root=/dev/nfs nfsroot=<nfs_server_ip>:/data/rpi_rootfs,udp,vers=3
ip=<raspberrypi_ip>:<nfs_server_ip>:<default_gateway>:<mask>:rpi:eth0:off rootfstype=nfs smsc95xx.turbo_mode=N

Step 4: Adapt your Rpi fstab

Now edit the Rpi "/etc/fstab" file before trying to boot, do under your NFS server or client computer.

/data/rpi_rootfs/etc/fstab:

• Delete the original line corresponding to your root fs and pointing to the second partition of your flash card  (/dev/mmcblk0p2), we don't need anymore as it will automatically be mounted by the firmware at boot time

Step 5: Boot your Rpi!

Ok, let's go, time to boot :-)

If you follow all steps carefully, you system should boot with no major issue.

Therefore, you will not have anymore swap available, by default Raspbian uses dphys-swapfile os use a local file as swap.

We will correct this now.

Step 6: Correct your swap configuration by migrating to a loop device

By default, Raspbian uses dphys-swapfile to generate a local file being used as swap, this won't work anymore when booting under NFS.

I don't recommend to use your Flash card as a swap partition, this may generates system freeze or kernel panics if you have data corruption:

The better way is to set a local file as a loop device that will be used a swap device, here is how.

Clean current non working swap file and uninstall dphys-swapfile:

sudo apt-get remove --purge dphys-swapfile
sudo rm /var/swap
sudo rm /etc/init.d/dphys-swapfile
sudo update-rc.d dphys-swapfile remove

Create a new swap file, create the loop swap device and activate swap (exemple with 1GB swap):

sudo dd if=/dev/zero of=/var/swapfile bs=1M count=1024
sudo losetup /dev/loop0 /var/swapfile
sudo mkswap /dev/loop0
sudo swapon /dev/loop0

Check your current swap availability:

$ free

Output example:

             total       used       free     shared    buffers     cached
Mem:        237656     213092      24564          0         24      93192
-/+ buffers/cache:     119876     117780
Swap:      1048572       1556    1047016

Make it permanent, edit "/etc/rc.local" and add this section before "exit 0":

echo "Setting up loopy/var/swapfile.."
sleep 2
losetup /dev/loop0 /var/swapfile
mkswap /dev/loop0
swapon /dev/loop0

Step 7: Other tunings

There is also some other little thing to tune:

Edit "/etc/default/rcS" and:

• add:ASYNCMOUNTNFS=no
• set: RAMRUN=yes
• set: RAMLOCK=yes

Edit "/etc/sysctl.conf" and:

• add or set: vm.min_free_kbytes = 12288 

This will ensure the system will always have 12Mb or RAM free to prevent kernel panic risk, your may try lower value if your prefer.

Memorandum

• NFS version and fine tuning

You may want to try different settings to get the better performance possible.

First, if you test your write speed, using dd will be very easy:

Create a 10 Mb file test:

$ dd if=/dev/zero of=/tmp/test.file bs=1M count=10

Output sample:

10+0 enregistrements lus                                                                                                      
10+0 enregistrements écrits                                                                                                   
10485760 octets (10 MB) copiés, 1,52525 s, 6,9 MB/s

You may want to try with different NFS version, change in cmdline.txt:

• Change the section nfsroot "vers=2/3"

You may want to try TCP versus UDP

• Change the section nfsroot  "udp" or "tcp"

You may want to try different values of "rsize" and "wsize", example with NFS V3 and TCP:

Example:

• root=/dev/nfs nfsroot=<nfs_server_ip>:/data/rpi_rootfsrsize=32768,wsize=32768,tcp,vers=3

In my case, it did not really change anything, so i kept kernel default values for wsize and rsize, udp with NFS V3.

• Netfilter Iptables

When modifying your Iptables configuration, keep in mind that NFS traffic with your NFS server will result in system halt.

Applying default outbound policy to DROP (usually "iptables -P OUTPUT DROP") will in system crash.

You should apply instead "iptables -P OUTPUT ACCEPT" which will permit any outbound traffic from your Rpi (not a big deal, usually you trust your own machine)

Also you can ensure to accept NFS traffic with your NFS server before applying any other rules.

Splunk Howto - Splunk for Fail2ban, get a the Fail2ban Multi-host frontend with Splunk!

Last Update: 01/13/2012

Current Version = 1.1

Splunk (if you don't yet know it) is an incredibly powerful solution that collects, indexes and exploits any kind of data from any system, offering you as many solution as you need and even the possibility to create custom applications with graphical front-ends. (dashboards, reports, saved searches...)

In a few words, i am really impressed by Splunk, i think i've been looking for this for many many years!

Don't hesitate to take a look at main Splunk Website, you will easily find a lot of information and great documentations: http://www.splunk.com/

Splunk can be used for free with some little restrictions. (not more than 500Mb of input data per day)

Details: http://docs.splunk.com/Documentation/Splunk/5.0.1/Admin/TypesofSplunklicenses

I developed my first Splunk application "Splunk For Fail2ban" to provide a cool frontend and log managing tool associated with the well known and powerful Fail2ban tool. (take a look at my older post: http://youresuchageek.blogspot.fr/2012/11/howto-fail2ban-secure-your-network.html)

To install this addon, follow this link on Splunkbase or install it through the standard Splunk application process search online: 

http://splunk-base.splunk.com/apps/72087/splunk-for-fail2ban

You can also get it from here, install through the Splunk Application Manager:

SplunkForFail2ban V1.1

Splunk pre-requirements:

Ensure to install requirements Splunk addons:

• Sideview utils addon
• Google Maps addon
• Maxmind geo location addon

Splunk For Fail2ban provides:

A complete Dashboard Overview of Fail2ban activity for all managed systems: 

Overview of last 5 connection Fail2ban denied events, Number of events in selected Time Range

Fail2ban Alert trend per Jail, Top country origin of denied IP with graphical and data overview

Top Denied IP, Top denied per Jail with graphical and data overview

Top Fail2ban reporting servers for multi-hosts management, graphical and data overview

Google Maps Dashboard, identify the source of connexion attempts

A Fail2ban Event search interface with selection per kind of data (IPs, ID, Jail...)

Pre-defined major searches to get all the most important information

Denied Hosts by IP:

Denied Hosts by Jail:

Denied Hosts by Reporting Fail2ban server:

Top denied by IP:

Top Denied by Jail:

Top denied by IP with geo location:

Top denied by IP with geo location and DNS resolution:

Top denied by country origin:

Installation and utilization

Introduction

Installing and configuring Splunk is out of the scope of this post, still installing Splunk is really easy and well done, in 10 minutes you'll be done ^^

Download Splunk !

As a brieve description, here is how Splunk for Fail2ban works:

- We modify Fail2ban to add a specific message for each ban action and containing fields Splunk will analyse

- Through Syslog, we can manage as many Fail2ban servers as required

- Splunk collects our data and produces the IT intelligency

Installation and configuration will be done in a few steps:

1. Modifying Fail2ban configuration files related to the ban action (the goal is send fields we will analyse with Splunk)

2. Setting up Fail2ban to log to Syslog system

3. Setting up Syslog to trap custom Fail2ban events into a specific log file (can be local or remote Syslog if numerous Fail2ban hosts)

4. Installation and configuring Splunk for Fail2ban

Part 1: Configure Fail2ban

1. Set Fail2ban output to Syslog

I recommend the use of "rsyslog" as your main Syslog management, it comes with much more improvement than the standard Syslog. (http://www.rsyslog.com/)

First, we need to set Fail2ban to log its messages into Syslog instead of a standard log file.

To do so, edit "/etc/fail2ban/fail2ban.conf" and set:

logtarget = SYSLOG

2. Modify Fail2ban configuration file for the ban action

Depending of your wish, you can set Fail2ban to use 1 of these 3 actions: (by editing /etc/fail2ban/jail.conf)

• action_ = Fail2ban will temporarely ban the IP source host
• action_mw = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request
• action_mwl = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request and log traces

In any case you need to modify the corresponding action configuration file to include a specific command line that will log any useful filed for Splunk.

Depending of your configuration, edit:

using the default action "action_" with the mta set to "mta = mail" :

• action_ --> Edit "/etc/fail2ban/action.d/mail.conf"
• action_mw --> Edit "/etc/fail2ban/action.d/mail-whois.conf"
• action_mwl --> Edit "/etc/fail2ban/action.d/mail-whois-lines.conf"

using the default action "action_" with the mta set to "mta = sendmail" :

• action_ --> Edit "/etc/fail2ban/action.d/sendmail.conf" 
• action_mw --> Edit "/etc/fail2ban/action.d/sendmail-whois.conf"
• action_mwl --> Edit "/etc/fail2ban/action.d/sendmail-whois-lines.conf"

Insert a new command line for the action section called "actionban" and predecing the existing action just before the "printf" command:

logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] " &

The action configuration file using mail "/etc/fail2ban/action.d/mail-whois-lines.conf" will look like this:

actionban = logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
            Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
            logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] " & \
     printf %%b "Hi,\n
     The IP <ip> has just been banned by Fail2Ban after <failures> attempts against <name>.\n\n
     Here are more information about <ip>:\n
     `whois <ip>`\n\n
     Lines containing IP:<ip> in <logpath>\n
     `grep '\<<ip>\>' <logpath>`\n\n
     Regards,\n
     Fail2Ban"|mail -s "[Fail2Ban] <name>: banned <ip>" <dest>

The action configuration file using sendmail "/etc/fail2ban/action.d/sendmail-whois-lines.conf" will look like this:

actionban = logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
            Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
            logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] " & \
            printf %%b "Subject: [Fail2Ban] <name>: banned <ip>
            Date: `date -u +"%%a, %%d %%h %%Y %%T +0000"`
            From: Fail2Ban <<sender>>
            To: <dest>\n
            Hi,\n
            The IP <ip> has just been banned by Fail2Ban after
            <failures> attempts against <name>.\n\n
            Here are more information about <ip>:\n
            `/usr/bin/whois <ip>`\n\n
            Lines containing IP:<ip> in <logpath>\n
            `/bin/grep '\<<ip>\>' <logpath>`\n\n
            Regards,\n
            Fail2Ban" | /usr/sbin/sendmail -f <sender> <dest>

So just set your action configuration file corresponding to your MTA with the configuration above, and restart Fail2ban.

3. Test Fail2ban 

Now let's test your system, generate a ban event (try to log in through SSH with bad credentials) and check

your Syslog file to find the generated event. (look for the pattern "fail2ban.banevent")

You should find a ban event like this:

Jan 11 20:24:34 myhostname logger[30720]: [fail2ban.banevent]: fail2ban_host: [myfail2ban] Banhost: [xx.xx.xx.xx] jailname: [ssh] numberoffailures: [6] logmessage: [ Jan 11 20:24:32 myhostname sshd[30706]: Received disconnect from xx.xx.xx.xx: 11: Bye Bye [preauth] ] 

Now you're done with Fail2ban, let's configure Syslog ^^

Part 2: Configure Syslog - Standalone and Multi-Hosts

In 2 steps:

• if you want to manage different Fail2ban servers from Splunk, then read the Multiple Fail2ban client configuration note
• If you just one host to manage (Fail2ban and Splunk are installed in the same host), then just follow the common configuration section

MULTIPLE FAIL2BAN CLIENT CONFIGURATION NOTE: Remote and centralized Syslog configuration

Configuring Syslog to send events from a Syslog host to a remote Syslog server is out of the scope of this guide.

Therefore, if you want to collect fail2ban events from different hosts, you can choose between different solutions, as:

• Sending events using Syslog to a remote centralized Syslog
• Sending events from local log file using Splunk forwarder module
• Others (homemade scripts, file sharing...)

I would recommend using Rsyslog (default enhanced Syslog for many Linux systems) to achieve this, which is in deed easy enough, robust and efficient.

Here is in 2 steps a quick rsyslog centralized configuration: (remember to restart rsyslog after each modification)

1. In each client rsyslog host, modify "/etc/rsyslog.conf" and add a section to send any events to your Syslog server: (adapt the example IP)

"/etc/rsyslog.conf"

*.* @192.168.1.254:514 

2. In syslog server configuration, create a configuration file that will trapp any remote client Syslog events and put then into a dedicated per host log file:

Ensure your configuration name will be read after the fail2ban syslog config file you will create after. (see above, this is very )

Create "/etc/rsyslog.d/10-fail2ban.conf" with the following content: (Note: The fail2ban config we will create after will be called 08 to be read before this and intercept messages)

"/etc/rsyslog.d/10-fail2ban.conf"

template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n" 
:inputname, isequal, "imudp" ?PerHostLog;RemoteHostFileFormat
& ~

Restart rsyslog after any config modification.

COMMON CONFIGURATION for Single and Multiple (for the centralized rsyslog server) Fail2ban installation: 

1. Set Syslog to trap ban events to a dedicated logfile

This configuration part will depend on your system and needs, i recommend the use of "rsyslog"

The goal is to configure syslog to trap any event containing a key word "[fail2ban.banevent]" into a dedicated log file

In Debian/Ubuntu systeprintfms for example, create an rsyslog configuration file, example:

Create "/etc/rsyslog.d/08-fail2ban.conf" with the following content: 

"/etc/rsyslog.d/08-fail2ban.conf"

:msg, contains, "[fail2ban.banevent]" /var/log/fail2ban_banevent.log
& ~

Restart rsyslog to take effect:

sudo service rsyslog restart

2. Generate a ban event and check your logfile

Generate a new ban event and check your log file, you should see a new ban event message! 

If you are ok with that, then you're done with system configuration ^^ 

Part 3: Configuration of Splunk (the easy part!)

Here comes the easier part with no doubts :-)

1. Create a new index "fail2ban_index" dedicated to Fail2ban input

Go to "manager", "Indexes", then create a new index with default params and call it: fail2ban_index

2. Configure Input file

Go to "manager", "Data Input" and configure MANUALLY a new input file pointing to your Fail2ban log file, with following settings:

Host:

You can let the default settings, it does not mind as we don't use it to recognize the fail2ban reporting server.

Source type:

- Set the source Type: Manual
- Source type: fail2ban_banevent

Index:

- Set the destination Index: fail2ban_index

Good news, you're done!!!
Just wait a few minutes to let Splunk get the content of your fail2ban log file, then go to the splunk application Splunk for Fail2ban

Don't hesitate to share any comment with me, this is my very first Splunk application and it may still needs some improvement :-)