Unable to send email to own domain via GMail SMTP using Postfix

I've got Postfix setup on Ubuntu to send emails via GMail SMTP.  Somehow, it doesn't seem to recognize my domain's own users.  Postfix log as follows:


Nov 20 21:40:08 ubuntu postfix/qmgr[12418]: 1EA5A2040813: from=app@mydom.com, size=543, nrcpt=1 (queue active)
Nov 20 21:40:08 ubuntu postfix/local[8895]: 1EA5A2040813: to=app@mydom.com, relay=local, delay=0.14, delays=0.09/0/0/0.06, dsn=5.1.1, status=bounced (unknown user: "app")

The fix is really simple.  Remove or comment the following line in main.cf:

mydestination = $myhostname, $mydomain

Restart Postfix and you're ready to rock :)

Inconsistency detected by ld.so: dl-open.c

Cleaned my Ubuntu laptop with Ubuntu Tweak and was left with a weird error when using Chrome and Firefox.  Chrome stopped loading pages while Firefox crashed completely.  Found out the problem only after starting Firefox from the console.  Error string reported by FFX:

Inconsistency detected by ld.so: dl-open.c: 667: _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT' failed!

Used strace on Firefox and obtained the following output:


open("/lib/x86_64-linux-gnu/tls/x86_64/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu/tls/x86_64", 0x7fffdd0b6c00) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/tls/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu/tls", 0x7fffdd0b6c00) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/x86_64/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu/x86_64", 0x7fffdd0b6c00) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
open("/usr/lib/x86_64-linux-gnu/tls/x86_64/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/tls/x86_64", 0x7fffdd0b6c00) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/tls/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/tls", 0x7fffdd0b6c00) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/x86_64/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/x86_64", 0x7fffdd0b6c00) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu", {st_mode=S_IFDIR|0755, st_size=49152, ...}) = 0
open("/lib/tls/x86_64/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/tls/x86_64", 0x7fffdd0b6c00) = -1 ENOENT (No such file or directory)
open("/lib/tls/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/tls", 0x7fffdd0b6c00)        = -1 ENOENT (No such file or directory)
open("/lib/x86_64/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64", 0x7fffdd0b6c00)     = -1 ENOENT (No such file or directory)
open("/lib/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/lib/tls/x86_64/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls/x86_64", 0x7fffdd0b6c00) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls", 0x7fffdd0b6c00)    = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64", 0x7fffdd0b6c00) = -1 ENOENT (No such file or directory)
open("/usr/lib/libGL.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib", {st_mode=S_IFDIR|0755, st_size=73728, ...}) = 0
munmap(0x7fde2d69b000, 121572)          = 0
writev(2, [{"Inconsistency detected by ld.so:"..., 33}, {"dl-open.c", 9}, {": ", 2}, {"667", 3}, {": ", 2}, {"_dl_open", 8}, {": ", 2}, {"Assertion `", 11}, {"_dl_debug_initialize (0, args.ns"..., 61}, {"' failed!\n", 10}], 10Inconsistency detected by ld.so: dl-open.c: 667: _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT' failed!) = 141

libGL.so.1 is there but it just doesn't get recognized by ld.  After digging the web for a few hours, finally found the solution.  Re-install some packages:

sudo apt-get install --reinstall libgl1-mesa-glx:amd64 libgl1-mesa-glx:i386 libgl1-mesa-dri:amd64 libgl1-mesa-dri:i386

Haven't had such crazy stuff in Ubuntu for a while :)

Controlling Mouse Speed in Ubuntu when System Settings Doesn't Cut It (aka xinput)

My laptop comes with a Targus cordless mouse which moves too fast even when both Accelleration and Sensitivity in Pointer Speed settings is set to the lowest in Ubuntu. 

There's another method to control input speed via the xinput command.

To begin, execute "xinput list" to obtain the list of inputs available:

mike@myhost:~$ xinput list
⎡ Virtual core pointer                        id=2    [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                  id=4    [slave  pointer  (2)]
⎜   ↳ SONiX Targus Soft-Touch Cordless Mouse      id=8    [slave  pointer  (2)]
⎣ Virtual core keyboard                       id=3    [master keyboard (2)]
    ↳ Virtual core XTEST keyboard                 id=5    [slave  keyboard (3)]
    ↳ Power Button                                id=6    [slave  keyboard (3)]
    ↳ Power Button                                id=7    [slave  keyboard (3)]
    ↳ Dell Dell USB Entry Keyboard                id=9    [slave  keyboard (3)]
    ↳ Dell WMI hotkeys                            id=10    [slave  keyboard (3)]

From the output above, it's quite obvious which input device we'll be working with (id=8 if you can't spot it).  Next, we'll move on to list down the device properties:

mike@myhost:~$ xinput list-props 8
Device 'SONiX Targus Soft-Touch Cordless Mouse':
        Device Enabled (143):   1
        Coordinate Transformation Matrix (145): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
        Device Accel Profile (260):     0
        Device Accel Constant Deceleration (261):       1.000000
        Device Accel Adaptive Deceleration (262):       1.000000
        Device Accel Velocity Scaling (263):    10.000000
        Evdev Axis Inversion (264):     0, 0
        Evdev Axes Swap (266):  0
        Axis Labels (267):      "Rel X" (153), "Rel Y" (154)
        Button Labels (268):    "Button Left" (146), "Button Middle" (147), "Button Right" (148), "Button Wheel Up" (149), "Button Wheel Down" (150), "Button Horiz Wheel Left" (151), "Button Horiz Wheel Right" (152), "Button Side" (524), "Button Extra" (525), "Button Unknown" (259), "Button Unknown" (259), "Button Unknown" (259), "Button Unknown" (259)
        Evdev Middle Button Emulation (269):    0
        Evdev Middle Button Timeout (270):      50
        Evdev Wheel Emulation (271):    0
        Evdev Wheel Emulation Axes (272):       0, 0, 4, 5
        Evdev Wheel Emulation Inertia (273):    10
        Evdev Wheel Emulation Timeout (274):    200
        Evdev Wheel Emulation Button (275):     4
        Evdev Drag Lock Buttons (276):  0

To properly "decelerate" the mouse pointer, we'll need to play around with 2 properties:

• Device Accel Constant Deceleration (261)
• Device Accel Adaptive Deceleration (262)

To update a property value, use the "--set-prop" option like below:

mike@myhost:~$ xinput --set-prop 8 261 4

Note the values used:

• 8: device id
• 261: property id
• 4: value of the property to be set

Setting both property value to 4 works for me; YMMV.

Have fun tweaking your input :)

rsyslog: Logging to remote server

Now that we've setup the rsyslogd server to accept incoming connections (NOTE: UDP somehow didn't work.  I had to configure rsyslogd to listen to TCP instead); we can now configure the "client" rsyslogd instance to log to a remote server.

In the "client" server, we'll need to edit the same configuration file /etc/rsyslog.conf.  I've added the highlighted lines below:

[...]
###############
#### RULES ####
###############

# Log all messages to this rsyslogd host
*.* @@myhost:514
#
# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
[...]

The rest of the config file remains unchanged.  Reload/restart the service once we're done:

myclient:~# /etc/init.d/rsyslog reload
Reloading enhanced syslogd: rsyslogd.

We can now use the logger command to send log messages to the syslog.  Here's what I used:

# logger -t CLIENT_TEST "This is a test to test the test"

The following entry should be logged in myhost /var/log/syslog file:

Dec 12 15:17:52 myhost CLIENT_TEST: This is a test to test the test

rsyslog: Enabling remote logging service in Ubuntu

Newer versions of Ubuntu (since 9.10 according to rsyslog wiki: http://wiki.rsyslog.com/index.php/Ubuntu) comes with rsyslog instead of sysklogd.  Was trying to enable it via the sysklogd way by adding the "-r" option in the startup script.  That obviously didn't work :)

What you'll need to do is just uncomment 2 lines in the /etc/rsyslog.conf file:

# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

That's if you want to provide UDP syslog service.  Uncomment the following 2 lines if you want to provide TCP syslog service:

# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Once you've made the changes, either reload or restart the rsyslogd service:

myhost:/etc# /etc/init.d/rsyslog reload
Reloading enhanced syslogd: rsyslogd.

I've chose to enable UDP for my server.  We'll use netstat to check if rsyslogd is listening to the specified port:

myhost:/etc# netstat -tlnup | grep 514
udp        0      0 0.0.0.0:514             0.0.0.0:*                           13282/rsyslogd
udp6       0      0 :::514                  :::*                                13282/rsyslogd

rsyslogd is indeed listening to the proper port and protocol. 

Next stop, actual logging and probably more configuration :)

Setting up Hadoop in clustered mode in Ubuntu

Overview

This entry details the steps I took to setup Hadoop in a clustered setup in Ubuntu 11.10.  Hadoop version 0.20.205.0 was used to setup the environment.  The Hadoop cluster consists of 3 servers/nodes:

• node616 ==> namenode, tasktracker, datanode, jobtracker, secondarynamenode
• node617 ==> datanode, jobtracker
• node618 ==> datanode, jobtracker

In an actual production setup, thet namenode shouldn't act as datanode, jobtracker and secondarynamenode.  But for the purpose of this setup, things will be simplified :)


Server setup

Ensure that the /etc/hosts file in all servers are updated properly.  All my servers have the following entry:

192.168.1.1    node616
192.168.1.2    node617
192.168.1.3    node618

This is to ensure that the configuration files stay the same in all servers.

The following directories must be created beforehand to store Hadoop related data:

• /opt/hdfs/cache    ==> HDFS cache storage
• /opt/hdfs/data    ==> HDFS data node storage
• /opt/hdfs/name    ==> HDFS name node storage

SSH setup

Before we proceed to actual setup, the user running Hadoop must be able to ssh to the servers without a passphrase.  Test this out by issuing the following command:

$ ssh node616

If it prompts for a password, execute the following commands:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

The public key needs to be copied to all data nodes/slaves once they're setup in a later stage.


Namenode setup

Obtain Hadoop binary distribution from the main site (http://hadoop.apache.org).  Place it to a location in the server.  I've used /opt/hadoop for all installations. 

The extracted directory contents should look like the output below:

one616:/opt/hadoop# ls -l
total 7144
drwxr-xr-x  2 root root    4096 2011-11-25 16:58 bin
-rw-rw-r--  1 root root  112062 2011-10-07 14:19 build.xml
drwxr-xr-x  4 root root    4096 2011-10-07 14:24 c++
-rw-rw-r--  1 root root  433928 2011-10-07 14:19 CHANGES.txt
drwxr-xr-x  2 root root    4096 2011-11-30 12:23 conf
drwxr-xr-x 11 root root    4096 2011-10-07 14:19 contrib
drwxr-xr-x  3 root root    4096 2011-10-07 14:20 etc
-rw-rw-r--  1 root root    6839 2011-10-07 14:19 hadoop-ant-0.20.205.0.jar
-rw-rw-r--  1 root root 3700955 2011-10-07 14:24 hadoop-core-0.20.205.0.jar
-rw-rw-r--  1 root root  142465 2011-10-07 14:19 hadoop-examples-0.20.205.0.jar
-rw-rw-r--  1 root root 2487116 2011-10-07 14:24 hadoop-test-0.20.205.0.jar
-rw-rw-r--  1 root root  287776 2011-10-07 14:19 hadoop-tools-0.20.205.0.jar
drwxr-xr-x  3 root root    4096 2011-10-07 14:20 include
drwxr-xr-x  2 root root    4096 2011-11-22 14:28 ivy
-rw-rw-r--  1 root root   10389 2011-10-07 14:19 ivy.xml
drwxr-xr-x  6 root root    4096 2011-11-22 14:28 lib
drwxr-xr-x  2 root root    4096 2011-11-22 14:28 libexec
-rw-rw-r--  1 root root   13366 2011-10-07 14:19 LICENSE.txt
drwxr-xr-x  4 root root    4096 2011-12-07 12:10 logs
-rw-rw-r--  1 root root     101 2011-10-07 14:19 NOTICE.txt
drwxr-xr-x  4 root root    4096 2011-11-29 10:36 out
-rw-rw-r--  1 root root    1366 2011-10-07 14:19 README.txt
drwxr-xr-x  2 root root    4096 2011-11-22 14:28 sbin
drwxr-xr-x  4 root root    4096 2011-10-07 14:20 share
drwxr-xr-x  9 root root    4096 2011-10-07 14:19 webapps

Navigate to the conf directory and edit the core-site.xml file.  The default file should look like the following:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
</configuration>

Now we'll have to add 2 properties to make this a clustered setup: 

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
     <property>
         <name>fs.default.name</name>
         <value>hdfs://node616:9000</value>
     </property>

     <property>
         <name>hadoop.tmp.dir</name>
         <value>/opt/hdfs/cache</value>
     </property>
</configuration>

• fs.default.name ==> Sets the default file system name.  Since we're setting up a clustered environment, we'll set this to point to the namenode hostname and port; which in this case is the current machine.

• hadoop.tmp.dir ==> A base for other temporary directories.  Points to /tmp by default.  But I had a problem with that as the Linux /tmp mount point is usually very small and caused problems.  The following exception was thrown if I did not explicitly set this property:

java.io.IOException: File /user/root/testfile could only be replicated to 0 nodes, instead of 1

For more properties, please consult the following URL: http://hadoop.apache.org/common/docs/current/core-default.html
   
Next comes the hdfs-site.xml file which we'll customize it like the following:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>
     <property>
         <name>dfs.replication</name>
         <value>1</value>
     </property>
        <property>
                <name>dfs.name.dir</name>
                <value>/opt/hdfs/name</value>
        </property>
        <property>
                <name>dfs.data.dir</name>
                <value>/opt/hdfs/data</value>
        </property>
</configuration>

• dfs.replication ==> Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.  Since we only have one node, we'll set it to 1 for the time being.

• dfs.name.dir ==>     Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.

• dfs.data.dir ==> Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored.

More configuration parameters here: http://hadoop.apache.org/common/docs/current/hdfs-default.html

Lastly, we come to the MapReduce site configuration file; mapred-site.xml.  Output below shows the updated version:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>
     <property>
         <name>mapred.job.tracker</name>
         <value>node616:9001</value>
     </property>
</configuration>

• mapred.job.tracker ==> Node specific port which job tracker process is running on.

Edit the masters file and change the localhost value to node616.  Ditto slaves file.  By default, the host is set to localhost in both files.  However, since we're using proper host names, it's better to update the entry so that all master and slaves nodes can use.

One last thing before starting up the service is to initialize the HDFS namenode directory.  Execute the following command:

node616:/opt/hadoop/bin$ ./hadoop namenode -format

Everything should be configured correctly :)  We can run Hadoop by going into the bin directory:

node616:/opt/hadoop/bin$ ./start-all.sh
starting namenode, logging to /opt/hadoop/libexec/../logs/hadoop-root-namenode-node616.outWarning: $HADOOP_HOME is deprecated.
node616: starting datanode, logging to /opt/hadoop/libexec/../logs/hadoop-root-datanode-node616.outnode616: Warning: $HADOOP_HOME is deprecated.node616:node616: starting secondarynamenode, logging to /opt/hadoop/libexec/../logs/hadoop-root-secondarynamenode-node616.outnode616: Warning: $HADOOP_HOME is deprecated.node616:starting jobtracker, logging to /opt/hadoop/libexec/../logs/hadoop-root-jobtracker-node616.outWarning: $HADOOP_HOME is deprecated.
node616: starting tasktracker, logging to /opt/hadoop/libexec/../logs/hadoop-root-tasktracker-node616.outnode616: Warning: $HADOOP_HOME is deprecated.node616:

A quick check via ps:

hadoop     29004 28217  0 09:31 pts/0    00:00:07 /usr/bin/java -Dproc_jar -Xmx256m -Dhadoop.log.dir=/opt/hadoop/libexec/../logs -Dhadoop.log.file=hadoop.log -Dhadoop.hom
hadoop     30630     1  1 16:07 pts/0    00:00:02 /usr/bin/java -Dproc_namenode -Xmx1000m -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Dc
hadoop     30743     1  3 16:07 ?        00:00:04 /usr/bin/java -Dproc_datanode -Xmx1000m -server -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=f
hadoop     30858     1  1 16:07 ?        00:00:01 /usr/bin/java -Dproc_secondarynamenode -Xmx1000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote -Dhadoop.
hadoop     30940     1  2 16:07 pts/0    00:00:02 /usr/bin/java -Dproc_jobtracker -Xmx1000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote -Dhadoop.log.dir
hadoop     31048     1  2 16:07 ?        00:00:03 /usr/bin/java -Dproc_tasktracker -Xmx1000m -Dhadoop.log.dir=/opt/hadoop/libexec/../logs -Dhadoop.log.file=hadoop-hadoop-ta

Now that we can see all processes are running, go ahead and visit the following URLs:

• http://node616:50030 ==> Map/Reduce admin
• http://node616:50070 ==> NameNode admin   

Let's try copying some files over to the HDFS:

node616:/opt/hadoop/bin$ ./hadoop fs -copyFromLocal 20m.log .

And let's see if it's there:

node616:~$ hadoop fs -ls
Found 1 items
-rw-r--r--   3 hadoop supergroup 5840878894 2011-11-29 09:21 /user/hadoop/20m.log

So far so good :)

Once you're done, shutdown the Hadoop processes by executing stop-all.sh:

node616:/opt/hadoop/bin# ./stop-all.sh
stopping jobtracker
node616: stopping tasktracker
stopping namenode
node616: stopping datanode
node616: stopping secondarynamenode

Data nodes/slaves

Now that the namenode is up, we can proceed to setup our slaves. 

Since we know that we'll have an additional two servers, we can add those entries in to the conf/slaves file:

node616
node617
node618

If there's a need to add more in the future, the slaves nodes can be added dynamically.

Edit the hdfs-site.xml file and change the dfs.replication value from 1 to 3.  This ensures that the data blocks are replicated to 3 nodes (which is actually the default value).

Next, tar the entire Hadoop directory in the Namenode by executing the following command:

node616:/opt$ tar czvf hadoop.tar.gz hadoop

Transfer the tarball to the other servers (i.e. node617 and node618) and untar it.  Make sure the /opt/hdfs directories have been created.  Once the package has been extracted, go back to the namenode (node616) and execute the start-all.sh script.  It should output the following:

node616:/opt/hadoop/bin# ./start-all.sh
starting namenode, logging to /opt/hadoop/libexec/../logs/hadoop-root-namenode-node616.out
Warning: $HADOOP_HOME is deprecated.
node617: starting datanode, logging to /opt/hadoop/libexec/../logs/hadoop-root-datanode-node617.out
node616: starting datanode, logging to /opt/hadoop/libexec/../logs/hadoop-root-datanode-node616.out
node618: starting datanode, logging to /opt/hadoop/libexec/../logs/hadoop-root-datanode-node618.out
node617: Warning: $HADOOP_HOME is deprecated.
node617:
node616: Warning: $HADOOP_HOME is deprecated.
node616:
node618: Warning: $HADOOP_HOME is deprecated.
node618:
node616: starting secondarynamenode, logging to /opt/hadoop/libexec/../logs/hadoop-root-secondarynamenode-node616.out
node616: Warning: $HADOOP_HOME is deprecated.
node616:
starting jobtracker, logging to /opt/hadoop/libexec/../logs/hadoop-root-jobtracker-node616.out
Warning: $HADOOP_HOME is deprecated.
node618: starting tasktracker, logging to /opt/hadoop/libexec/../logs/hadoop-root-tasktracker-node618.out
node617: starting tasktracker, logging to /opt/hadoop/libexec/../logs/hadoop-root-tasktracker-node617.out
node616: starting tasktracker, logging to /opt/hadoop/libexec/../logs/hadoop-root-tasktracker-node616.out
node618: Warning: $HADOOP_HOME is deprecated.
node618:
node617: Warning: $HADOOP_HOME is deprecated.
node617:
node616: Warning: $HADOOP_HOME is deprecated.
node616:

Notice that the script will remotely start the data and task tracker services in the slave nodes.  Visit the NameNode admin at http://node616:50070 to confirm the number of live nodes in the cluster.


Stopping/Starting Services in a node

To stop or start specific a specific service in just one node, use the bin/hadoop-daemon.sh script.  As an example, to stop the datanode and the tasktracker processes in node618, I'll do:

/opt/hadoop/bin/hadoop-daemon.sh stop datanode
/opt/hadoop/bin/hadoop-daemon.sh stop tasktracker

To start them up, simply substitute "stop" with "start" in the commands above.


References

• http://wiki.apache.org/hadoop/FAQ

Connecting to remote MySQL via SSH in Ubuntu

I had to access my MySQL server via SSH tunnel on my Ubuntu desktop machine.  First up, setup ssh to tunnel the server's MySQL port (default 3306) to my Ubuntu's desktop machine port 13306:

ssh mylogin@myserver.com -p 4265 -L 13306:127.0.0.1:3306

However, when I tried accessing port 13306 on my Ubuntu desktop, it failed:

$ mysql -u root -p -P 13306
Enter password:
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

It seems the default localhost server used by mysql client does a socket connection instead of TCP/IP.  In order to overcome this, I had to use the --host option:

$ mysql -u root -p -P 13306 --host 127.0.0.1
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 115946
Server version: 5.1.49-3 (Debian)
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> 

That took me a while to figure out :)

Load balancing OpenVPN connections via IPVS (Linux Virtual Server)

First Steps
 
First off, I've performed the setup and configuration on my own local PC using VirtualBox VMs.  Network for all the VMs are set to bridge mode.  Here's how the servers are configured:

director
--------
Fedora 12
VIP=eth0:0 192.168.1.150
RIP=eth0 192.168.1.200

ovpnserver1
-----------
Ubuntu
RIP=eth0 192.168.1.201
VIP=lo:0 192.168.1.150 (no arp)

ovpnserver2
-----------
Ubuntu
RIP=eth0 192.168.1.202
VIP=lo:0 192.168.1.150 (no arp)

Here's how the servers are laid out:

director ------- ovpnserver1
       |
       |-------- ovpnserver2

IPVS will be configured via the direct routing method; as opposed to NAT or Tunneling.


Setting Up The First OpenVPN Server (ovpnserver1)

Begin by configuring the network interface.  Here's how the /etc/network/interfaces file looks like:

# The loopback network interface
auto lo
iface lo inet loopback

# VIP routing
auto lo:0
iface lo:0 inet static
address 192.168.1.150
netmask 255.255.255.255

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.201
network 192.168.1.0
netmask 255.255.255.0
gateway 192.168.1.1

Restart networking service if necessary:

# service networking restart

Next, install OpenVPN package in the server:

# apt-get install openvpn

Now configure OpenVPN:

# cd /etc/openvpn
# cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/ .
# mv easy-rsa rsa
# cd rsa

Edit the vars script and edit the appropriate variables at the end of the file (e.g. KEY_COUNTRY, KEY_CITY).  We then generate the necessary keys for the server:

# . ./vars
# ./clean-all
# ./build-dh
# ./pkitool --initca
# ./pkitool --server server

The CA and server keys/certs are now in /etc/openvpn/rsa/keys.  Once the keys and certs have been generated, we'll need to create a server configuration file.  For the purpose of this test, we'll authenticate the user's credentials using the supplied auth-pam.pl script.  This script authenticates OpenVPN against the system users.  Create the configuration file in /etc/openvpn/server.conf with the following contents:

port 1194
proto udp
dev tun

ca /etc/openvpn/rsa/keys/ca.crt
cert /etc/openvpn/rsa/keys/server.crt
key /etc/openvpn/rsa/keys/server.key
dh /etc/openvpn/rsa/keys/dh1024.pem

server 10.128.127.0 255.255.255.0
ifconfig-pool-persist ipp.txt

push "redirect-gateway def1"
push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"

keepalive 10 120

comp-lzo

max-clients 50

persist-key
persist-tun

status openvpn-status.log
log-append /var/log/openvpn.log
verb 3
mute 20

client-cert-not-required
username-as-common-name

auth-user-pass-verify auth-pam.pl via-file

Ensure that you have Perl::PAM module installed for the script to work:

# apt-get install libauthen-pam-perl

Enable IP forwarding:

# vim /etc/sysctl.conf

Ensure that the following line is in the file:

net.ipv4.ip_forward=1

Refresh sysctl params from /etc/sysctl.conf:

# sysctl -p

Enable NAT/MASQ rules in iptables:

# iptables -t nat -A POSTROUTING -s 10.128.127.0/24 -o eth0 -j MASQUERADE
# iptables-save

Now start OpenVPN daemon:

# service openvpn restart

Setting Up The Second OpenVPN Server (ovpnserver2)

Like the first server, we begin by configuring the network interface.  Here's how the /etc/network/interfaces file looks like:

# The loopback network interface
auto lo
iface lo inet loopback

# VIP routing
auto lo:0
iface lo:0 inet static
address 192.168.1.150
netmask 255.255.255.255

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.202
network 192.168.1.0
netmask 255.255.255.0
gateway 192.168.1.1

Restart networking service if necessary:

# service networking restart

Next, install OpenVPN package in the server:

# apt-get install openvpn

Instead of configuring this instance, we'll copy the entire /etc/openvpn folder from ovpnserver1 to ovpnserver2.  This is required as the CA certs/keys must match.


Setting Up The Load Balancer (director)

Configure the network interfaces like so:

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
HWADDR=08:00:27:81:7A:C2
IPADDR=192.168.1.200
BOOTPROTO=none
NETMASK=255.255.255.0
DNS2=208.67.222.222
TYPE=Ethernet
GATEWAY=192.168.1.1
DNS1=208.67.220.220
IPV6INIT=no
ONBOOT=yes
USERCTL=no
PREFIX=24
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

/etc/sysconfig/network-scripts/ifcfg-eth0:0

DEVICE=eth0:0
HWADDR=08:00:27:81:7A:C2
IPADDR=192.168.1.150
BOOTPROTO=none
NETMASK=255.255.255.0
DNS2=208.67.222.222
TYPE=Ethernet
GATEWAY=192.168.1.1
DNS1=208.67.220.220
IPV6INIT=no
ONBOOT=yes
USERCTL=no
PREFIX=24
NAME="System eth0:0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

Turn on ipvsadm daemon:

# chkconfig ipvsadm on
# chkconfig --list ipvsadm
ipvsadm         0:off   1:off   2:on    3:on    4:on    5:on    6:off

The ipvsadm daemon reads its config file from /etc/sysconfig/ipvsadm.  Create the file with the following contents:

-A -u 192.168.1.150:1194 -s rr
-a -u 192.168.1.150:1194 -r 192.168.1.201:1194
-a -u 192.168.1.150:1194 -r 192.168.1.202:1194

ipvsadm uses the direct routing method by default.  The first line in the file instructs ipvsadm to add a virtual service.  In this case, it's a UDP service denoted by the -u parameter.  The -s rr parameter instructs it to use the round-robin scheduling method.  The subsequent two lines adds servers to the virtual service.  Since we have 2 virtual machines, we specify both the IPs.  Restart the ipvsadm service once you're done:

# service ipvsadm restart

Type the following command to verify that you have a working setup:

# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
UDP  192.168.1.150:openvpn rr
  -> 192.168.1.201:openvpn        Route   1      0          0
  -> 192.168.1.202:openvpn        Route   1      0          0

OpenVPN Client Configuration File

The client configuration file is quite straight forward.  Instead of having multiple "remote" options, you'll just have one which will be pointing to the director's IP.  The other thing to note is to add the "float" option.  Below's the configuration file for this purpose:

client
dev tun
float
proto udp
remote 192.168.1.150 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
auth-user-pass
cipher BF-CBC
comp-lzo
verb 4
mute 20

Here's a snippet from openvpn's man page on the "float" option:

--float
    Allow  remote  peer to change its IP address and/or port number, such as due to DHCP (this is the default if --remote is not used).  --float when
    specified with --remote allows an OpenVPN session to initially connect to a peer at a known address, however if packets arrive from a new address
    and  pass  all  authentication  tests,  the new address will take control of the session.  This is useful when you are connecting to a peer which
    holds a dynamic address such as a dial-in user or DHCP client.

    Essentially, --float tells OpenVPN to accept authenticated packets from any address, not only the address which was  specified  in  the  --remote
    option.

Installing Shorewall in Ubuntu

I started off with the simple command:

# apt-get install shorewall shorewall-doc

Thought everything was installed properly.  But I got the following error instead:

# shorewall start

Compiling...

   No shorewall compiler installed

Seems I'm missing one more package:

# apt-get install shorewall-perl

Once the shorewall-perl package is you should get the following output:

# shorewall start

Compiling...

   ERROR: No firewall zone defined

Now read up on configuring Shorewall! :)

Configuring OpenVPN + FreeRADIUS + MySQL

Here's a quick guide on how to configure OpenVPN to use FreeRADIUS for authentication.  I've only tried it on Ubuntu Karmic.  YMMV on other distros.

Download the radiusplugin here http://www.nongnu.org/radiusplugin/.  You'll have to compile it from source as there's currently no deb package for it.  Compile steps below:

# apt-get install libgcrypt11 libgcrypt11-dev build-essential
# make

The output will be a single radiusplugin.so file.  Now move the .so file and the .cnf file to the proper openvpn directory like so:

# cp radiusplugin.so /etc/openvpn/
# cp radiusplugin.cnf /etc/openvpn/

First off, edit the radiusplugin.cnf file.  Focus on the "server" section and ensure that the details are correct:

server
{
        # The UDP port for radius accounting.
        acctport=1813
        # The UDP port for radius authentication.
        authport=1812
        # The name or ip address of the radius server.
        name=127.0.0.1
        # How many times should the plugin send the if there is no response?
        retry=1
        # How long should the plugin wait for a response?
        wait=1
        # The shared secret.
        sharedsecret=testing123
}

Next up, edit the openvpn server config file and add the following line:

plugin /etc/openvpn/radiusplugin.so /etc/openvpn/radiusplugin.cnf

Restart openvpn service and start freeradius service and login using a username/password pair which is defined in FreeRADIUS. 

You should now have working setup.  More info below:

• http://www.roessner-net.com/?p=4
  

Installing and configuring daloRADIUS

With FreeRADIUS+MySQL installed, it's now turn to install daloRADIUS; a web-based management system for FreeRADIUS.  Before installation, ensure that Apache, PHP and MySQL have been installed and running.  You'll need PHP PEAR as well.  Use the following command to install the required libs:

# apt-get install php-pear
# pear install DB

Get the latest version of daloRADIUS from http://sourceforge.net/projects/daloradius/files/.  Extract the tarball to /var/www/daloradius.  You'll need to import daloRADIUS DB schema from /var/www/daloradius/contrib/db/mysql-daloradius.sql:

# mysql -u root -p radius < mysql-daloradius.sql

Even after importing the schema, there seems to be a missing table i.e. usergroup.  Get the missing table schema from fr1-mysql-freeradius.sql in the same directory.  Once the table has been imported, edit the /var/www/daloradius/library/daloradius.conf.php for DB access credentials.

Access the daloRADIUS installation via a web browser at http://<hostname>/daloradius.  Login as administrator/radius.

You should now have a working daloRADIUS installation.  

Installing and configuring FreeRADIUS with MySQL authentication

Here's a *near* step-by-step guide to setup FreeRADIUS with MySQL backed authentication done on Ubuntu 9.10 (Karmic).  Start by installing the necessary deb files:

# apt-get install freeradius freeradius-mysql

Add the following lines to the /etc/freeradius/users file: 

abc     Cleartext-Password := "123"

Stop freeradius service just in case:

/etc/init.d/freeradius stop

Now test using the radtest client:

# radtest abc 123 localhost 1812 testing123
Sending Access-Request of id 149 to 127.0.0.1 port 1812
        User-Name = "abc"
        User-Password = "123"
        NAS-IP-Address = 208.67.219.132
        NAS-Port = 1812
rad_recv: Access-Accept packet from host 127.0.0.1 port 1812, id=149, length=20

Ensure that the response is "Access-Accept".  The following should be displayed at the server console:

rad_recv: Access-Request packet from host 127.0.0.1 port 33425, id=149, length=55
        User-Name = "abc"
        User-Password = "123"
        NAS-IP-Address = 208.67.219.132
        NAS-Port = 1812
+- entering group authorize {...}
++[preprocess] returns ok
++[chap] returns noop
++[mschap] returns noop
[suffix] No '@' in User-Name = "abc", looking up realm NULL
[suffix] No such realm "NULL"
++[suffix] returns noop
[eap] No EAP-Message, not doing EAP
++[eap] returns noop
++[unix] returns notfound
[files] users: Matched entry abc at line 61
++[files] returns ok
++[expiration] returns noop
++[logintime] returns noop
++[pap] returns updated
Found Auth-Type = PAP
+- entering group PAP {...}
[pap] login attempt with password "123"
[pap] Using clear text password "123"
[pap] User authenticated successfully
++[pap] returns ok
+- entering group post-auth {...}
++[exec] returns noop
Sending Access-Accept of id 149 to 127.0.0.1 port 33425
Finished request 2.
Going to the next request
Waking up in 4.9 seconds.
Cleaning up request 2 ID 149 with timestamp +3641
Ready to process requests.

Now on to MySQL setup.  First, create a database where FreeRADIUS will store its data.  We'll call it radius:

create database radius

Import the MySQL schema from /etc/freeradius/sql/mysql/schema.sql:

mysql -u root -p radius < /etc/freeradius/sql/mysql/schema.sql

You should have 7 tables as shown below:

radacct
radcheck
radgroupcheck
radgroupreply
radpostauth
radreply
radusergroup

Edit the file /etc/freeradius/sql.conf and change the following parameters to suite your environment:

server = "localhost"
login = "root"
password = "password"
radius_db = "radius"

Enable the SQL configuration in /etc/freeradius/radiusd.conf by uncommenting the following line:

$INCLUDE sql.conf

Enable SQL configuration in the default enabled site /etc/freeradius/sites-available/default:

authorize {
    ...
    sql
    ...
}

accounting {
    ...
    sql
    ...
}

session {
    ...
    sql
    ...
}

post-auth {
    ...
    sql
    ...
}

Insert the following record into radcheck table:

INSERT INTO `radius`.`radcheck` (`id` ,`username` ,`attribute` ,`op` ,`value`)
VALUES (NULL , 'test', 'MD5-Password', ':=', MD5( '1234' )

Note: More info on the 'op' value can be obtained by reading the 'unlang' man page.  Snippet shown below:

Operators
      The operator used to assign the value of the attribute may be one of the following, with the given meaning.

      =      Add the attribute to the list, if and only if an attribute of the same name is already present in that list.

      :=     Add  the attribute to the list.  If any attribute of the same name is already present in that list, its value is replaced with the value of the current
             attribute.

      +=     Add the attribute to the tail of the list, even if attributes of the same name are already present in the list.

Enforcement and Filtering Operators
      The following operators may also be used in addition to the ones listed above.  Their function is to perform enforcement or filtering on attributes in a list.

      -=     Remove all matching attributes from the list.  Both the attribute name and value have to match in order for the attribute to be removed from the list.

      ==     Remove all non-matching attributes from the list.  Both the attribute name and value have to match in order for the attribute to remain in the list.

             Note that this operator is very different than the '=' operator listed above!

      <=     Enforce that the integer value of the attribute is less than or equal to the value given here.  If there is no attribute of the same name in the  list,
             the  attribute  is  added  with  the  given value, is with "+=".  If an attribute in the list exists, and has value less than given here, it's value is
             unchanged.  If an attribute in the list exists, and has a value greater than given here, then that value is replaced with the one given here.

             This operator is valid only for attributes of integer type.

      >=     Enforce that the integer value of the attribute is greater than or equal to the value given here.  If there is no attribute of the  same  name  in  the
             list, the attribute is added with the given value, is with "+=".  If an attribute in the list exists, and has value greater than given here, it's value
             is unchanged.  If an attribute in the list exists, and has value less than given here, then that value is replaced with the one given here.

             This operator is valid only for attributes of integer type.

Start freeradius in debug mode by using the command below:

# freeradius -X

In another terminal console, use radtest again to test the connection:

# radtest test 1234 localhost 1812 testing123
root@mike-laptop:/etc/freeradius/sites-available# radtest test 1234 localhost 1812 testing123
Sending Access-Request of id 147 to 127.0.0.1 port 1812
        User-Name = "test"
        User-Password = "1234"
        NAS-IP-Address = 208.67.219.132
        NAS-Port = 1812
rad_recv: Access-Accept packet from host 127.0.0.1 port 1812, id=147, length=20

Again, make sure the response contains "Access-Accept".  Following output should be shown at the server terminal:

++[preprocess] returns ok
++[chap] returns noop
++[mschap] returns noop
[suffix] No '@' in User-Name = "test", looking up realm NULL
[suffix] No such realm "NULL"
++[suffix] returns noop
[eap] No EAP-Message, not doing EAP
++[eap] returns noop
++[unix] returns notfound
        expand: %{User-Name} -> test
[sql] sql_set_user escaped user --> 'test'
rlm_sql (sql): Reserving sql socket id: 3
        expand: SELECT id, username, attribute, value, op           FROM radcheck           WHERE username = '%{SQL-User-Name}'           ORDER BY id -> SELECT id, username, attribute, value, op           FROM radcheck           WHERE username = 'test'           ORDER BY id
[sql] User found in radcheck table
        expand: SELECT id, username, attribute, value, op           FROM radreply           WHERE username = '%{SQL-User-Name}'           ORDER BY id -> SELECT id, username, attribute, value, op           FROM radreply           WHERE username = 'test'           ORDER BY id
        expand: SELECT groupname           FROM radusergroup           WHERE username = '%{SQL-User-Name}'           ORDER BY priority -> SELECT groupname           FROM radusergroup           WHERE username = 'test'           ORDER BY priority
rlm_sql (sql): Released sql socket id: 3
++[sql] returns ok
++[expiration] returns noop
++[logintime] returns noop
[pap] Normalizing MD5-Password from hex encoding
++[pap] returns updated
Found Auth-Type = PAP
+- entering group PAP {...}
[pap] login attempt with password "1234"
[pap] Using MD5 encryption.
[pap] User authenticated successfully
++[pap] returns ok
+- entering group post-auth {...}
        expand: %{User-Name} -> test
[sql] sql_set_user escaped user --> 'test'
        expand: %{User-Password} -> 1234
        expand: INSERT INTO radpostauth                           (username, pass, reply, authdate)                           VALUES (                           '%{User-Name}',                           '%{%{User-Password}:-%{Chap-Password}}',                           '%{reply:Packet-Type}', '%S') -> INSERT INTO radpostauth                           (username, pass, reply, authdate)                           VALUES (                           'test',                           '1234',                           'Access-Accept', '2010-01-10 12:55:21')
rlm_sql (sql) in sql_postauth: query is INSERT INTO radpostauth                           (username, pass, reply, authdate)                           VALUES (                           'test',                           '1234',                           'Access-Accept', '2010-01-10 12:55:21')

There should also be some records in the radpostauth table:

mysql> select * from radpostauth;
+----+----------+------+---------------+---------------------+
| id | username | pass | reply         | authdate            |
+----+----------+------+---------------+---------------------+
| 14 | test     | 1234 | Access-Accept | 2010-01-10 12:00:12 |
| 15 | test     | 1234 | Access-Accept | 2010-01-10 12:19:39 |
| 16 | test     | 1234 | Access-Accept | 2010-01-10 12:55:21 |
+----+----------+------+---------------+---------------------+
3 rows in set (0.00 sec)

As you can see, it's rather dangerous to have the clear text password shown in the table.  If you want to turn off that feature, comment out the 'sql' directive in the /etc/freeradius/sites-available/default file under 'post-auth' section.

Have fun!