full of sound and fury

fixing the raid

17 July 2014

I noticed a couple weeks ago that I kept getting errors in the logs for one of the disks in the multimedia array. This was upsetting, but since I had paid a little more for these disks to get the 5 year warranty, I wasn't too concerned. Then I did a warranty check on the disk. It was the disk that WD sent me to replace the last disk that failed.

Oh goody.

After futzing with the WD support website, they had changed it since the last time I had to RMA a disk, I finally got an advance RMA setup. The disk arrived last week, but I was unable, until today, to get around to replacing it.

So I powered down the server, pulled the bad disk ( installed 2011-04-20 according to the label I put on it ) and put the new one in it's place. Then I rebooted. The system complained about a degraded array, was I sure I wanted to continue, even though I hit 'y', it dropped to a minimal recovery shell. sigh

Reboot, let's try this again. Same thing.

dirty words

Reboot again, hit 'e' to edit the boot paramemeters in GRUB, add 'bootdegraded=true' and voila, it boots.

Now to it was as simple as:
Copy the partition map from the existing drive...

# sfdisk -d /dev/sde | sfdisk/dev/sda

Add the new disk into the array:

# mdadm --manage /dev/md127 --add /dev/sda1

And check on the progress:

 # cat /proc/mdstat 
 Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
 [raid4] [raid10] 
 md127 : active raid1 sda1[2] sde1[0]
       1953511936 blocks [2/1] [U_]
       [>....................]  recovery =  3.8% (75987264/1953511936) finish=218.8min speed=142988K/sec

unused devices: <none>

Not too shabby!

# df -kh /multimedia/
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/raid--lvm-movies  1.8T  841G  882G  49% /multimedia

Now I just need to get the old drive back into the shipping box and get it back to WD so they don't charge me $220 for the replacement disk.


PECL::Perl and Net::Telnet

30 January 2014

Well, I wasn't going to, but I decided to install the PECL::Perl extension for PHP afterall. So far it seems to work with the two modules that were proving to be difficult to find native PHP replacements for, Text::Diff and Net::Telnet.

Here is an example of using it with Net::Telnet:

<?php

$perl = new Perl();
$perl->eval('use Net::Telnet');

$t = $perl->eval(<<<'PERL_END'
new Net::Telnet(
    Prompt => '/\S+[#>][ ]?$/'
);
PERL_END
);

$t->open( '172.31.38.65' );
$t->login( $username, $password );

$lines = $t->array->cmd( 'show ver' );

print_r( $lines );
?>

I am sure that there are people that are going to hate this solution :)


PHP and networks

28 January 2014

Well, I did not install the perl PHP extension. I think my friend Benjamin summed it up the best when he responded to my post about it on facebook:

Surround your pullquote like this {" text to be quoted "}

So now I am back to looking for a good way to do diffs in PHP of files that are sometimes 75,000+ lines. Either my google-fu has improved and I have not seen some of these solutions before, or it has not and I blocked out the memory of having tested some of these solutions. We'll see which it is when I get into the office tomorrow.

Also, the other special kind of fun that I was trying to find a solution for was a replacement for the perl module Net::Netmask. Thanfully I found a github gist that had just what I was looking for!

Reproduced here are the ones that I adapted for what I needed, mainly so I can find them again later.

<?php

# REFERENCE - http://stackoverflow.com/questions/2942299/converting-cidr-address-to-subnet-mask-and-network-address

$ipNetmask = "192.168.1.12/16";
list($ip, $netmask) = split( "/", $ipNetmask );
$ip_elements_decimal = split( "[.]", $ip );
$netmask_result="";
for($i=1; $i <= $netmask; $i++) {
  $netmask_result .= "1";
}
for($i=$netmask+1; $i <= 32; $i++) {
    $netmask_result .= "0";
echo $netmask_result;
}
$netmask_ip_binary_array = str_split( $netmask_result, 8 );
$netmask_ip_decimal_array = array();
foreach( $netmask_ip_binary_array as $k => $v ){
    $netmask_ip_decimal_array[$k] = bindec( $v ); // "100" => 4
    $network_address_array[$k] = ( $netmask_ip_decimal_array[$k] & $ip_elements_decimal[$k] );
}
$network_address = join( ".", $network_address_array );

print_r($network_address);
?>
<?php
 
# REFERENCE http://stackoverflow.com/questions/594112/matching-an-ip-to-a-cidr-mask-in-php5
 
function cidr_match($ip, $range)
{
    list ($subnet, $bits) = split('/', $range);
    $ip = ip2long($ip);
    $subnet = ip2long($subnet);
    $mask = -1 << (32 - $bits);
    $subnet &= $mask; # nb: in case the supplied subnet wasn't correctly aligned
    return ($ip & $mask) == $subnet;
}
 
if (cidr_match("10.128.0.0", "10.128.1.0/16")) {
    echo 'match';
} else {
    echo 'no match';
}
 
echo cidr_match("10.128.0.0", "10.128.1.0/16");

?>
<?php
 
function mask2cidr($mask) {
 
$mask = split( "[.]", $mask );
 
$bits = 0;
 
foreach ($mask as $octect) {
    $bin = decbin($octect);
    $bin = str_replace ( "0" , "" , $bin);
    $bits = $bits + strlen($bin);
}
    return $bits;
}
 
echo mask2cidr("255.255.252.0");
 
?>


Here we go again

26 January 2014

I just can't seem to leave well enough alone.

Well, really, I kind of blame the current projects that I am working on for my employer. It's drawing me back into the world of webdesign. And while I wasn't unhappy with the octopress site that I was using, it felt a little too restrictive. Yeah, the guy that just wanted to be about to write on his blog decided to retool it. Again.

This time I am using vanilla jekyll but with bootstrap and jquery and fontawesome.

I'm not sure if my coworker is going to be happy that I found this stuff, and of course, now want to use them in the tools we are building...

Man, if only these tools would have been available ~16 years ago...


perl in PHP

24 January 2014

One of the projects that I am working on uses perl CGI scripts to do various things with a database and also getting information directly from routers and switches. One reason I did this in perl was because it was easier to port some of my existing commandline scripts to CGI.

The other reason was because I really did not want to use PHP to telnet to a router. I have seen the code that people use to do that, and it scares me.

So while poking around and researching what we would need to do to call perl scripts from php, in order to consolidate some of our development and get a little more speed... I ran across this link.

This has the potential to allow me to use perl for some of the things that I need, and php for the rest. I am pretty sure this is going to cause my coworker that is working on these projects with me, to go into a fit. :)

Next week I will test using Net::Telnet, Net::Netmask and Text::Diff from inside php... what's the worst that could happen?!

<?php
$perl = new Perl();
$perl->eval( 'use Net::Ping;' );
$ping = $perl->eval('Net::Ping->new');
foreach ( array( '172.27.1.3', '172.27.1.1' ) as $ip ) {
    if ( $ping->ping($ip) ) {
        echo "$ip is alive!\n";
    }
}
?>


Experimenting with nginx

18 January 2014

Seeing as how nginx is gaining popularity, partially due to it's performance on small virtual machines, aka virtual private servers, I decided to move my some of my personal sites to an nginx host from my current apache2 solution.

While I was in the process of doing this, I thought it would be handy to be able to create a new subdomain on the fly. There is already a wildcard configured in DNS, so now I just needed a wildcard setup in nginx.

This configuration checks for the existence of the directory named for the subdomain, if it doesn't exist, it redirects to www.domain.tld.

In theory, if all domains were in a standard base directory, I could have one configuration file that would route any website to the correct location. Maybe I'll experiment with that later.

And yes, I know that this kind of thing can be done with apache2 as well.

server {
    index index.html index.htm index.php;
    set $basepath "/home/username/";
    server_name ~^(?<sub>.+?)\.domain\.tld$;

    set $root $basepath/$sub.domain.tld;

    if ( !-d $root ) {
        rewrite ^ $scheme://www.domain.tld$uri redirect;
    }

    root $root;

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ /index.html;
    }
    # pass the PHP scripts to php-fpm
    #
    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
    }

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    location ~ /\.ht {
    deny all;
    }
}


fail2ban + ansible

07 October 2013

I use fail2ban to slow down and stop brute force attempts against my servers. One drawback of the way it is currently implemented is that if they are really persistant, attackers keep coming back.

If they have managed to trip fail2ban multiple times, that means they have either attacked two different servers, or have come back to the same one multiple times. This means that I am quite happy to permanently ban them from access to my systems.

This requires some setup on the servers in question.

auto lo
iface lo inet loopback

pre-up iptables-restore < /etc/firewall-rules
post-up iptables-restore -n < /etc/iptables.blocklist


*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FIREWALL - [0:0]
:blocklist - [0:0]
-A INPUT -j blocklist
-A INPUT -j FIREWALL
-A FORWARD -j FIREWALL
-A FIREWALL -i lo -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 25 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 81 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 110 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 143 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 443 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 465 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 587 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 993 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 995 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 4650 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 4949 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 22 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A FIREWALL -p udp -m udp --sport 53 -j ACCEPT
-A FIREWALL -p tcp -m tcp --sport 53 -j ACCEPT
-A FIREWALL -p udp -m udp --dport 53 -j ACCEPT
-A FIREWALL -p tcp -m tcp --dport 53 -j ACCEPT
-A FIREWALL -p udp -m udp --dport 123 -j ACCEPT
-A FIREWALL -p udp -m udp --sport 123 -j ACCEPT
-A FIREWALL -p udp -m udp --sport 6277 -j ACCEPT
-A FIREWALL -p udp -m udp --sport 24441 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A FIREWALL -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A FIREWALL -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j REJECT --reject-with icmp-port-unreachable
-A FIREWALL -p udp -m udp -j REJECT --reject-with icmp-port-unreachable
-A FIREWALL -p icmp -j DROP
COMMIT

The file /etc/iptables.blocklist starts out pretty baren.

*filter
:blocklist - [0:0]
COMMIT

It took me a little while to figure out what the bare minimum needed configuration was to be able to 'source' an iptables file.

I use two ways to update the /etc/iptables.blocklist file. One is to use a short shell script, which also has an option to kick off the ansible update as well.

The append.sh script typically takes two arguments, append.sh -i <IPADDRESS> will add the IP before the COMMIT line at the end. append.sh -p will 'push' the configuration using ansible. There is one other option append.sh -f <FILENAME> -i <IPADDRESS> to specify a different blocklist filename.

The other is to use a perl script that parses the subject lines for all of the email that I get from fail2ban looking for the ip address in the subject line.

#!/bin/dash

# Copyright (c) 2013, grephead.com, LLC
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#     * Redistributions of source code must retain the above copyright
#       notice, this list of conditions and the following disclaimer.
#     * Redistributions in binary form must reproduce the above copyright
#       notice, this list of conditions and the following disclaimer in the
#       documentation and/or other materials provided with the distribution.
#     * Neither the name of grephead.com, LLC nor the names of its
#       contributors may be used to endorse or promote products derived from
#       this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY grephead.com, LLC "AS IS" AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL grephead.com, LLC BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
# EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

# append an IPv4 address to the blocklist iptable
# pushing the changes using ansible

# limited PATH
PATH=/bin:/usr/bin

# default filename
filename='iptables.blocklist'

while getopts i:f:p f
do
    case ${f} in 
        i)  ipaddress=${OPTARG} ;;
        f)  filename=${OPTARG} ;;
        p)  push=1 ;;
    esac 
done

# check to see if ${ipaddress} is zero length, not specified.
if [ -z ${ipaddress}  ];then
    echo 'Missing IP address, not updating file'
else 

    # check to make sure that the IP is not already blocked
    # if not, insert before COMMIT string

    if [ -f ${filename} ]; then
        if grep -q -- "-A blocklist -s ${ipaddress} -j DROP" ${filename}; then
            echo "${ipaddress} already blocked"
        else

sed -i.bak -e "/COMMIT/i \
-A blocklist -s ${ipaddress} -j DROP" iptables.blocklist

        fi
    else
        echo "${filename} does not exist"
    fi
fi

# if -p is specified, push updated blocklist using ansible playbook
if ( [ "${push}" = "1" ] && [ -f blocklist_playbook.yml ] )
then
    ansible-playbook blocklist_playbook.yml
fi

The perl script uses File::Slurp and Mail::Box modules.

#!/usr/bin/perl

# The MIT License (MIT)
# 
# Copyright (c) 2013 Matt Okeson-Harlow
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to
# deal in the Software without restriction, including without limitation the
# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
# sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.

use strict;
use warnings;
use Mail::Box::Manager;
use File::Slurp qw( :std :edit );

my ( %ip_lookup, %ip_count, @append );
my $count = 0;

## Directory and file locations
my $fail2ban_folder = qq{$ENV{HOME}/Maildir/.grephead.fail2ban};
my $trash_folder    = qq{$ENV{HOME}/Maildir/.Trash};
my $iptables        = qq{$ENV{HOME}/btsync/ansible/iptables.blocklist};

## mailbox handlers
my $mgr    = Mail::Box::Manager->new;
my $folder = $mgr->open( folder => $fail2ban_folder, access => q{rw} );
my $trash  = $mgr->open( folder => $trash_folder, access => q{rw} );

## Slurp iptables rules into an array
my @blocklist = read_file $iptables;

## process each message in the fail2ban folder, looking for 'banned $ip'
for my $email ( $folder->messages ) {
    my $subject = $email->subject;

    # [Fail2Ban] ssh: banned 122.226.109.178
    if ( $subject =~ m{banned \s+ (\d+[.]\d+[.]\d+[.]\d+)}xms ) {
        my $ip = $1;
        $ip_count{$ip}++;
        push @{ $ip_lookup{$ip} }, $count;
    }
    $count++;
}

## look for $num instances of an IP address, add it to the append
## array and move the related email to the trash.
for my $ip ( keys %ip_count ) {
    if ( $ip_count{$ip} > 1 and not grep /$ip/, @blocklist ) {
        print qq{$ip_count{$ip}\t$ip\n};
        my $iptables_line = qq{-A blocklist -s $ip -j DROP};
        push @append, $iptables_line;
        for my $index ( @{ $ip_lookup{$ip} } ) {
            print $index . qq{\t}
                . $folder->message($index)->subject . qq{\n};
            $mgr->moveMessage( $trash_folder, $folder->message($index) );
        }
    }
}

## find 'COMMIT' in iptables file, insert new rules before it
my $append_string = join "\n", @append;
edit_file sub {s/(COMMIT)/$append_string\n$1/g}, $iptables;

After the fail2ban_filter.pl script is run, the append.sh -p script needs to be run to push the changes using ansible or run the ansible-playbook blocklist_playbook.yml command.

[grephead]
vader.grephead.com
yoda.grephead.com
leia.grephead.com
# push out updated blocklist iptables rules and add them to the running
# firewall

# 2013/04/16 02:17:35 

---
- hosts: grephead
  sudo: yes
  connection: ssh

  tasks:
    - name: update iptables.blocklist
      action: copy src=iptables.blocklist dest=/etc/iptables.blocklist owner=root group=root mode=0644

    - name: merge into iptables
      action: shell /sbin/iptables-restore -n < /etc/iptables.blocklist

# vim: set ts=2:sw=2:sts=2:ft=yaml


perl sort with pack

01 October 2013

A sort solution that I came up with for a project at work, it was the first time I used a pack command in a sort :)

sub bundle_sort {
    my $pkg = caller;
    no strict 'refs';

    pack( 'C4' => ${"${pkg}::a"} =~ /Bu1[.](\d+)/ )
    cmp
    pack( 'C4' => ${"${pkg}::b"} =~ /Bu1[.](\d+)/ )
    ||
    pack( 'C4' => ${"${pkg}::a"} =~ /source (\d+)[.](\d+)[.](\d+)[.](\d+)/ )
    cmp
    pack( 'C4' => ${"${pkg}::b"} =~ /source (\d+)[.](\d+)[.](\d+)[.](\d+)/ )
    ||
    pack( 'C4' => ${"${pkg}::a"} =~ /(\d+)[.](\d+)[.](\d+)[.](\d+) source/ )
    cmp
    pack( 'C4' => ${"${pkg}::b"} =~ /(\d+)[.](\d+)[.](\d+)[.](\d+) source/ )
}


raspberry pi + 17 inch monitor = awesome picture frame

24 July 2013

I recently bought a Raspberry Pi because I had seen some really cool things done with one and have some plans to do things with one involving amateur radio. But first I wanted to just get a feel for how it worked so I decided to do what a lot of people have done, and built a digital picture frame.

My wife and I had been looking at just buying an off the shelf picture frame, but they all seemed to have some kind of show stopper issue, errors with the software, problems with the view angle, limited storage, etc.

I ordered my raspberry pi and related hardware from amazon.com. This includes:

  • raspberry pi model B, rev 2.0, $43.17
  • a case, $11.49
  • 16GB class 10 SD card, $16.77
  • USB power adapter, $9.95
  • USB cable, $5.79
  • USB wifi adapter, $9.99
  • HDMI cable, $4.99
  • HDMI to DVI adapter, $4.00

I picked up a 17" Dell monitor from the local university surplus for $35.

I used the Debian Wheezy raspbian image.

The software to run the slideshow for the picture frame is 'feh'.

Subsonic is installed to act as a jukebox.

To sync data to the 'piframe' I used btsync.

The directory structure looks like this:

pi@piframe ~ $ ls -l piframe
total 36
drwxr-xr-x  2 pi pi  4096 Jul 24 14:52 archive/
drwx------  2 pi pi  4096 Jul 24 14:51 files/
drwxr-xr-x  2 pi pi 12288 Jul 24 15:10 images/
drwxr-xr-x  2 pi pi  4096 Jul 24 16:31 restart/
-rw-r--r--  1 pi pi    44 Jul 24 16:20 restart.bat
drwxr-xr-x  2 pi pi  4096 Jul 24 16:23 scripts/
drwxr-xr-x 16 pi pi  4096 Jul 24 14:51 tunes/

This is the script that does most of the work. It kills off any existing feh processes, sets the delay to be 15 seconds, and passes the options to feh. If the restart/restart_slideshow file exists, the file is deleted.

#!/bin/bash

# PiFrame feh start script

export DISPLAY=:0

/usr/bin/killall feh

DELAY=15

/usr/bin/feh --quiet \
  --hide-pointer \
  --recursive \
  --randomize \
  --full-screen \
  --slideshow-delay ${DELAY} \
  ${HOME}/piframe/images/ &

if [ -f ${HOME}/piframe/restart/restart_slideshow ];then
    rm ${HOME}/piframe/restart/restart_slideshow
fi

This file is added to ~/.config/autostart to call the slideshow.sh script on boot up.

[Desktop Entry]
Type=Application
Exec=/home/pi/piframe/scripts/slideshow.sh
mkdir .config/autostart
ln -s ~/piframe/scripts/slideshow.desktop ~/.config/autostart/

After installing incrond:

sudo apt-get install incrond

Then edit /etc/incrond.allow and add 'pi' or the username the slideshow will run under.

incrond is configured to monitor the piframe/restart directory. If any files are created in that directory, the piframe/scripts/slideshow.sh script is run.

/home/pi/piframe/restart IN_CREATE /home/pi/piframe/scripts/slideshow.sh

Since this is being setup to manage the slideshow from windows boxes, there is a bat file that creates a file in the restart directory. Once images are added or deleted to the piframe/images folder, run the restart.bat file and the slideshow will get restarted once btsync copies the files around.

echo restart > restart\restart_slideshow

I will have another post going into more detail on setting up btsync, as I am in the process of replacing Dropbox and SpiderOak sync with it.


testing

08 June 2013

Just a test after doing some ruby RVM updates.