Load balancing OpenVPN connections via IPVS (Linux Virtual Server)

First Steps
 
First off, I've performed the setup and configuration on my own local PC using VirtualBox VMs.  Network for all the VMs are set to bridge mode.  Here's how the servers are configured:

director
--------
Fedora 12
VIP=eth0:0 192.168.1.150
RIP=eth0 192.168.1.200

ovpnserver1
-----------
Ubuntu
RIP=eth0 192.168.1.201
VIP=lo:0 192.168.1.150 (no arp)

ovpnserver2
-----------
Ubuntu
RIP=eth0 192.168.1.202
VIP=lo:0 192.168.1.150 (no arp)

Here's how the servers are laid out:

director ------- ovpnserver1
       |
       |-------- ovpnserver2

IPVS will be configured via the direct routing method; as opposed to NAT or Tunneling.


Setting Up The First OpenVPN Server (ovpnserver1)

Begin by configuring the network interface.  Here's how the /etc/network/interfaces file looks like:

# The loopback network interface
auto lo
iface lo inet loopback

# VIP routing
auto lo:0
iface lo:0 inet static
address 192.168.1.150
netmask 255.255.255.255

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.201
network 192.168.1.0
netmask 255.255.255.0
gateway 192.168.1.1

Restart networking service if necessary:

# service networking restart

Next, install OpenVPN package in the server:

# apt-get install openvpn

Now configure OpenVPN:

# cd /etc/openvpn
# cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/ .
# mv easy-rsa rsa
# cd rsa

Edit the vars script and edit the appropriate variables at the end of the file (e.g. KEY_COUNTRY, KEY_CITY).  We then generate the necessary keys for the server:

# . ./vars
# ./clean-all
# ./build-dh
# ./pkitool --initca
# ./pkitool --server server

The CA and server keys/certs are now in /etc/openvpn/rsa/keys.  Once the keys and certs have been generated, we'll need to create a server configuration file.  For the purpose of this test, we'll authenticate the user's credentials using the supplied auth-pam.pl script.  This script authenticates OpenVPN against the system users.  Create the configuration file in /etc/openvpn/server.conf with the following contents:

port 1194
proto udp
dev tun

ca /etc/openvpn/rsa/keys/ca.crt
cert /etc/openvpn/rsa/keys/server.crt
key /etc/openvpn/rsa/keys/server.key
dh /etc/openvpn/rsa/keys/dh1024.pem

server 10.128.127.0 255.255.255.0
ifconfig-pool-persist ipp.txt

push "redirect-gateway def1"
push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"

keepalive 10 120

comp-lzo

max-clients 50

persist-key
persist-tun

status openvpn-status.log
log-append /var/log/openvpn.log
verb 3
mute 20

client-cert-not-required
username-as-common-name

auth-user-pass-verify auth-pam.pl via-file

Ensure that you have Perl::PAM module installed for the script to work:

# apt-get install libauthen-pam-perl

Enable IP forwarding:

# vim /etc/sysctl.conf

Ensure that the following line is in the file:

net.ipv4.ip_forward=1

Refresh sysctl params from /etc/sysctl.conf:

# sysctl -p

Enable NAT/MASQ rules in iptables:

# iptables -t nat -A POSTROUTING -s 10.128.127.0/24 -o eth0 -j MASQUERADE
# iptables-save

Now start OpenVPN daemon:

# service openvpn restart

Setting Up The Second OpenVPN Server (ovpnserver2)

Like the first server, we begin by configuring the network interface.  Here's how the /etc/network/interfaces file looks like:

# The loopback network interface
auto lo
iface lo inet loopback

# VIP routing
auto lo:0
iface lo:0 inet static
address 192.168.1.150
netmask 255.255.255.255

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.202
network 192.168.1.0
netmask 255.255.255.0
gateway 192.168.1.1

Restart networking service if necessary:

# service networking restart

Next, install OpenVPN package in the server:

# apt-get install openvpn

Instead of configuring this instance, we'll copy the entire /etc/openvpn folder from ovpnserver1 to ovpnserver2.  This is required as the CA certs/keys must match.


Setting Up The Load Balancer (director)

Configure the network interfaces like so:

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
HWADDR=08:00:27:81:7A:C2
IPADDR=192.168.1.200
BOOTPROTO=none
NETMASK=255.255.255.0
DNS2=208.67.222.222
TYPE=Ethernet
GATEWAY=192.168.1.1
DNS1=208.67.220.220
IPV6INIT=no
ONBOOT=yes
USERCTL=no
PREFIX=24
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

/etc/sysconfig/network-scripts/ifcfg-eth0:0

DEVICE=eth0:0
HWADDR=08:00:27:81:7A:C2
IPADDR=192.168.1.150
BOOTPROTO=none
NETMASK=255.255.255.0
DNS2=208.67.222.222
TYPE=Ethernet
GATEWAY=192.168.1.1
DNS1=208.67.220.220
IPV6INIT=no
ONBOOT=yes
USERCTL=no
PREFIX=24
NAME="System eth0:0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

Turn on ipvsadm daemon:

# chkconfig ipvsadm on
# chkconfig --list ipvsadm
ipvsadm         0:off   1:off   2:on    3:on    4:on    5:on    6:off

The ipvsadm daemon reads its config file from /etc/sysconfig/ipvsadm.  Create the file with the following contents:

-A -u 192.168.1.150:1194 -s rr
-a -u 192.168.1.150:1194 -r 192.168.1.201:1194
-a -u 192.168.1.150:1194 -r 192.168.1.202:1194

ipvsadm uses the direct routing method by default.  The first line in the file instructs ipvsadm to add a virtual service.  In this case, it's a UDP service denoted by the -u parameter.  The -s rr parameter instructs it to use the round-robin scheduling method.  The subsequent two lines adds servers to the virtual service.  Since we have 2 virtual machines, we specify both the IPs.  Restart the ipvsadm service once you're done:

# service ipvsadm restart

Type the following command to verify that you have a working setup:

# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
UDP  192.168.1.150:openvpn rr
  -> 192.168.1.201:openvpn        Route   1      0          0
  -> 192.168.1.202:openvpn        Route   1      0          0

OpenVPN Client Configuration File

The client configuration file is quite straight forward.  Instead of having multiple "remote" options, you'll just have one which will be pointing to the director's IP.  The other thing to note is to add the "float" option.  Below's the configuration file for this purpose:

client
dev tun
float
proto udp
remote 192.168.1.150 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
auth-user-pass
cipher BF-CBC
comp-lzo
verb 4
mute 20

Here's a snippet from openvpn's man page on the "float" option:

--float
    Allow  remote  peer to change its IP address and/or port number, such as due to DHCP (this is the default if --remote is not used).  --float when
    specified with --remote allows an OpenVPN session to initially connect to a peer at a known address, however if packets arrive from a new address
    and  pass  all  authentication  tests,  the new address will take control of the session.  This is useful when you are connecting to a peer which
    holds a dynamic address such as a dial-in user or DHCP client.

    Essentially, --float tells OpenVPN to accept authenticated packets from any address, not only the address which was  specified  in  the  --remote
    option.