Load balancing OpenVPN connections via IPVS (Linux Virtual Server)

First Steps
 
First off, I've performed the setup and configuration on my own local PC using VirtualBox VMs.  Network for all the VMs are set to bridge mode.  Here's how the servers are configured:

director
--------
Fedora 12
VIP=eth0:0 192.168.1.150
RIP=eth0 192.168.1.200

ovpnserver1
-----------
Ubuntu
RIP=eth0 192.168.1.201
VIP=lo:0 192.168.1.150 (no arp)

ovpnserver2
-----------
Ubuntu
RIP=eth0 192.168.1.202
VIP=lo:0 192.168.1.150 (no arp)

Here's how the servers are laid out:

director ------- ovpnserver1
       |
       |-------- ovpnserver2

IPVS will be configured via the direct routing method; as opposed to NAT or Tunneling.


Setting Up The First OpenVPN Server (ovpnserver1)

Begin by configuring the network interface.  Here's how the /etc/network/interfaces file looks like:

# The loopback network interface
auto lo
iface lo inet loopback

# VIP routing
auto lo:0
iface lo:0 inet static
address 192.168.1.150
netmask 255.255.255.255

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.201
network 192.168.1.0
netmask 255.255.255.0
gateway 192.168.1.1

Restart networking service if necessary:

# service networking restart

Next, install OpenVPN package in the server:

# apt-get install openvpn

Now configure OpenVPN:

# cd /etc/openvpn
# cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/ .
# mv easy-rsa rsa
# cd rsa

Edit the vars script and edit the appropriate variables at the end of the file (e.g. KEY_COUNTRY, KEY_CITY).  We then generate the necessary keys for the server:

# . ./vars
# ./clean-all
# ./build-dh
# ./pkitool --initca
# ./pkitool --server server

The CA and server keys/certs are now in /etc/openvpn/rsa/keys.  Once the keys and certs have been generated, we'll need to create a server configuration file.  For the purpose of this test, we'll authenticate the user's credentials using the supplied auth-pam.pl script.  This script authenticates OpenVPN against the system users.  Create the configuration file in /etc/openvpn/server.conf with the following contents:

port 1194
proto udp
dev tun

ca /etc/openvpn/rsa/keys/ca.crt
cert /etc/openvpn/rsa/keys/server.crt
key /etc/openvpn/rsa/keys/server.key
dh /etc/openvpn/rsa/keys/dh1024.pem

server 10.128.127.0 255.255.255.0
ifconfig-pool-persist ipp.txt

push "redirect-gateway def1"
push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"

keepalive 10 120

comp-lzo

max-clients 50

persist-key
persist-tun

status openvpn-status.log
log-append /var/log/openvpn.log
verb 3
mute 20

client-cert-not-required
username-as-common-name

auth-user-pass-verify auth-pam.pl via-file

Ensure that you have Perl::PAM module installed for the script to work:

# apt-get install libauthen-pam-perl

Enable IP forwarding:

# vim /etc/sysctl.conf

Ensure that the following line is in the file:

net.ipv4.ip_forward=1

Refresh sysctl params from /etc/sysctl.conf:

# sysctl -p

Enable NAT/MASQ rules in iptables:

# iptables -t nat -A POSTROUTING -s 10.128.127.0/24 -o eth0 -j MASQUERADE
# iptables-save

Now start OpenVPN daemon:

# service openvpn restart

Setting Up The Second OpenVPN Server (ovpnserver2)

Like the first server, we begin by configuring the network interface.  Here's how the /etc/network/interfaces file looks like:

# The loopback network interface
auto lo
iface lo inet loopback

# VIP routing
auto lo:0
iface lo:0 inet static
address 192.168.1.150
netmask 255.255.255.255

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.202
network 192.168.1.0
netmask 255.255.255.0
gateway 192.168.1.1

Restart networking service if necessary:

# service networking restart

Next, install OpenVPN package in the server:

# apt-get install openvpn

Instead of configuring this instance, we'll copy the entire /etc/openvpn folder from ovpnserver1 to ovpnserver2.  This is required as the CA certs/keys must match.


Setting Up The Load Balancer (director)

Configure the network interfaces like so:

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
HWADDR=08:00:27:81:7A:C2
IPADDR=192.168.1.200
BOOTPROTO=none
NETMASK=255.255.255.0
DNS2=208.67.222.222
TYPE=Ethernet
GATEWAY=192.168.1.1
DNS1=208.67.220.220
IPV6INIT=no
ONBOOT=yes
USERCTL=no
PREFIX=24
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

/etc/sysconfig/network-scripts/ifcfg-eth0:0

DEVICE=eth0:0
HWADDR=08:00:27:81:7A:C2
IPADDR=192.168.1.150
BOOTPROTO=none
NETMASK=255.255.255.0
DNS2=208.67.222.222
TYPE=Ethernet
GATEWAY=192.168.1.1
DNS1=208.67.220.220
IPV6INIT=no
ONBOOT=yes
USERCTL=no
PREFIX=24
NAME="System eth0:0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

Turn on ipvsadm daemon:

# chkconfig ipvsadm on
# chkconfig --list ipvsadm
ipvsadm         0:off   1:off   2:on    3:on    4:on    5:on    6:off

The ipvsadm daemon reads its config file from /etc/sysconfig/ipvsadm.  Create the file with the following contents:

-A -u 192.168.1.150:1194 -s rr
-a -u 192.168.1.150:1194 -r 192.168.1.201:1194
-a -u 192.168.1.150:1194 -r 192.168.1.202:1194

ipvsadm uses the direct routing method by default.  The first line in the file instructs ipvsadm to add a virtual service.  In this case, it's a UDP service denoted by the -u parameter.  The -s rr parameter instructs it to use the round-robin scheduling method.  The subsequent two lines adds servers to the virtual service.  Since we have 2 virtual machines, we specify both the IPs.  Restart the ipvsadm service once you're done:

# service ipvsadm restart

Type the following command to verify that you have a working setup:

# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
UDP  192.168.1.150:openvpn rr
  -> 192.168.1.201:openvpn        Route   1      0          0
  -> 192.168.1.202:openvpn        Route   1      0          0

OpenVPN Client Configuration File

The client configuration file is quite straight forward.  Instead of having multiple "remote" options, you'll just have one which will be pointing to the director's IP.  The other thing to note is to add the "float" option.  Below's the configuration file for this purpose:

client
dev tun
float
proto udp
remote 192.168.1.150 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
auth-user-pass
cipher BF-CBC
comp-lzo
verb 4
mute 20

Here's a snippet from openvpn's man page on the "float" option:

--float
    Allow  remote  peer to change its IP address and/or port number, such as due to DHCP (this is the default if --remote is not used).  --float when
    specified with --remote allows an OpenVPN session to initially connect to a peer at a known address, however if packets arrive from a new address
    and  pass  all  authentication  tests,  the new address will take control of the session.  This is useful when you are connecting to a peer which
    holds a dynamic address such as a dial-in user or DHCP client.

    Essentially, --float tells OpenVPN to accept authenticated packets from any address, not only the address which was  specified  in  the  --remote
    option.

Configuring OpenVPN + FreeRADIUS + MySQL

Here's a quick guide on how to configure OpenVPN to use FreeRADIUS for authentication.  I've only tried it on Ubuntu Karmic.  YMMV on other distros.

Download the radiusplugin here http://www.nongnu.org/radiusplugin/.  You'll have to compile it from source as there's currently no deb package for it.  Compile steps below:

# apt-get install libgcrypt11 libgcrypt11-dev build-essential
# make

The output will be a single radiusplugin.so file.  Now move the .so file and the .cnf file to the proper openvpn directory like so:

# cp radiusplugin.so /etc/openvpn/
# cp radiusplugin.cnf /etc/openvpn/

First off, edit the radiusplugin.cnf file.  Focus on the "server" section and ensure that the details are correct:

server
{
        # The UDP port for radius accounting.
        acctport=1813
        # The UDP port for radius authentication.
        authport=1812
        # The name or ip address of the radius server.
        name=127.0.0.1
        # How many times should the plugin send the if there is no response?
        retry=1
        # How long should the plugin wait for a response?
        wait=1
        # The shared secret.
        sharedsecret=testing123
}

Next up, edit the openvpn server config file and add the following line:

plugin /etc/openvpn/radiusplugin.so /etc/openvpn/radiusplugin.cnf

Restart openvpn service and start freeradius service and login using a username/password pair which is defined in FreeRADIUS. 

You should now have working setup.  More info below:

• http://www.roessner-net.com/?p=4
  

OpenVPN with pam_mysql username/password authentication

Here's how I setup OpenVPN with pam_mysql using only username/password authentication (insecure, but really easy on the admin side :P).  Target platform is Ubuntu Jaunty.  For starters, you'll need the following installed:

• openvpn
• mysql - server installed and running

Get the latest version of pam_mysql from http://pam-mysql.sourceforge.net/.  Untar it into a working directory.  The latest version as of writing this article is 0.7RC1.  The pam_mysql package in Jaunty is broken.  You'll need to patch the Makefile.in with the following patch (thanks to http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=418500#50):

--- Makefile.in.chold   2008-07-14 10:25:53.000000000 +0200
+++ Makefile.in 2008-07-14 10:26:06.000000000 +0200
@@ -110,7 +110,7 @@
 CPPFLAGS = @CPPFLAGS@
 LDFLAGS = @LDFLAGS@
 LIBS = @LIBS@
-pam_mysql_la_LIBADD =
+pam_mysql_la_LIBADD = -lpam
 pam_mysql_la_OBJECTS =  pam_mysql.lo
 CFLAGS = @CFLAGS@
 COMPILE = $(CC) $(DEFS) $(INCLUDES) $(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)

Save the above as patch.in in the same folder where you untar pam_mysql.  Then run the following command:

# patch -p0 <patch.in

That command will perform the patch process by using the patch.in file.  Next, we'll need to grab some deb packages to compile the module:

# apt-get install libmysql++-dev libpam0g-dev libgsasl7 libgsasl7-dev

Once that's done, proceed with the usual make commands (please note the extra configure options to enable SHA1 and MD5 options in the module):

# ./configure --with-cyrus-sasl2 --with-openssl
# make
# make install

The module should now be installed in /lib/security/pam_mysql.so.  Now to configure OpenVPN.  Here's my server configuration file:

port 1194
proto udp
dev tun

ca /etc/openvpn/easy-rsa/ca.crt
cert /etc/openvpn/easy-rsa/server.crt
key /etc/openvpn/easy-rsa/server.key
dh /etc/openvpn/easy-rsa/dh1024.pem

server 10.128.127.0 255.255.255.0
ifconfig-pool-persist ipp.txt

push "redirect-gateway def1"
push "dhcp-option DNS 1.1.1.1"
push "dhcp-option DNS 2.2.2.2"

keepalive 10 120

comp-lzo

max-clients 50

persist-key
persist-tun

status openvpn-status.log
log-append /var/log/openvpn.log
verb 3
mute 20

client-cert-not-required
username-as-common-name

plugin /usr/lib/openvpn/openvpn-auth-pam.so openvpn

The most important lines are the last three lines which have been bold.  Follow the commands below to setup OpenVPN server:

cd /etc/openvpn
cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/ easy-rsa
cd easy-rsa
vim vars # Edit KEY_* vars appropriately
source ./vars
./clean-all
./build-dh
./pkitool --initca
./pkitool --server server

To allow traffic to be routed from clients to the server, perform the following tasks:

echo 1 > /proc/sys/net/ipv4/ip_forward
edit /etc/sysctl.conf
Uncomment/add the line: net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 10.128.127.0/24 -o eth0 -j MASQUERADE
iptables-save

Now to configure PAM.  Create a file called openvpn in /etc/pam.d.  Below are its contents:

auth optional /lib/security/pam_mysql.so user=root passwd=pass host=localhost db=vpn_db table=tbl_user usercolumn=userid passwdcolumn=password where=active=1 sqllog=no crypt=4 verbose=0

account required /lib/security/pam_mysql.so user=root passwd=pass host=localhost db=vpn_db table=tbl_user usercolumn=userid passwdcolumn=password where=active=1 sqllog=no crypt=4 verbose=0

crypt=4 instructs pam_mysql to use SHA1.  There are other options which you can view in the pam_mysql README file.  OpenVPN client configuration file is shown below:

client
dev tun
proto udp
remote my.vpn.server.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
auth-user-pass
cipher BF-CBC
comp-lzo
verb 4
mute 20

Very lean and clean client config file.