Limiting grep output

Here's a quick way to limit output of the grep command.

Most of the time, we issue the following to find needle in the haystack:

grep needle file.txt

This prints out the matching pattern. If the output is way too long and we only need a section of it, we can use the extended grep (a.k.a. egrep) option.

Resizing images in batch using mogrify

Here's a quick way to resize images via shell using mogrify. It's part of ImageMagick package.

mogrify -resize 60% *.jpg

This will resize all files ending with .jpg extension to 60% of their original size.

We're only scratching the surface of what mogrify can do. Have fun exploring :)

Configuring Apache and DNS to access site without www prefixawesomehost.com

When hosting a site, it's convenient to just type the domain name e.g. mydomain.com instead of pre-fixing it with www.  This can be easily configured using Apache and by adding one additional entry to the DNS.  For this article's purpose, we'll be using awesomehost.com as the domain.

To start off with, add a new A DNS entry to point to the same IP as the A entry which maps to www.awesomehost.com.  You'll have something like below:

Name	Type	Value
awesomehost.com	A	192.168.0.1
www.awesomehost.com	A	192.168.0.1

Once that's done, we configure Apache.  Inside the VirtualHost directive, add the Alias directive as well as the rewrite condition:

<VirtualHost *:80>
ServerName awesomehost.com
ServerAlias www.awesomehost.com

DocumentRoot /home/blah/public_html
        <Directory /home/blah/public_html/>
AllowOverride All
Order allow,deny
allow from all
        </Directory>

RewriteEngine on
RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^ http://%1%{REQUEST_URI} [L,R=301]

</VirtualHost>

The rewrite condition redirects all requests beginning with www to the domain without www prefix.

Controlling Mouse Speed in Ubuntu when System Settings Doesn't Cut It (aka xinput)

My laptop comes with a Targus cordless mouse which moves too fast even when both Accelleration and Sensitivity in Pointer Speed settings is set to the lowest in Ubuntu. 

There's another method to control input speed via the xinput command.

To begin, execute "xinput list" to obtain the list of inputs available:

mike@myhost:~$ xinput list
⎡ Virtual core pointer                        id=2    [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                  id=4    [slave  pointer  (2)]
⎜   ↳ SONiX Targus Soft-Touch Cordless Mouse      id=8    [slave  pointer  (2)]
⎣ Virtual core keyboard                       id=3    [master keyboard (2)]
    ↳ Virtual core XTEST keyboard                 id=5    [slave  keyboard (3)]
    ↳ Power Button                                id=6    [slave  keyboard (3)]
    ↳ Power Button                                id=7    [slave  keyboard (3)]
    ↳ Dell Dell USB Entry Keyboard                id=9    [slave  keyboard (3)]
    ↳ Dell WMI hotkeys                            id=10    [slave  keyboard (3)]

From the output above, it's quite obvious which input device we'll be working with (id=8 if you can't spot it).  Next, we'll move on to list down the device properties:

mike@myhost:~$ xinput list-props 8
Device 'SONiX Targus Soft-Touch Cordless Mouse':
        Device Enabled (143):   1
        Coordinate Transformation Matrix (145): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
        Device Accel Profile (260):     0
        Device Accel Constant Deceleration (261):       1.000000
        Device Accel Adaptive Deceleration (262):       1.000000
        Device Accel Velocity Scaling (263):    10.000000
        Evdev Axis Inversion (264):     0, 0
        Evdev Axes Swap (266):  0
        Axis Labels (267):      "Rel X" (153), "Rel Y" (154)
        Button Labels (268):    "Button Left" (146), "Button Middle" (147), "Button Right" (148), "Button Wheel Up" (149), "Button Wheel Down" (150), "Button Horiz Wheel Left" (151), "Button Horiz Wheel Right" (152), "Button Side" (524), "Button Extra" (525), "Button Unknown" (259), "Button Unknown" (259), "Button Unknown" (259), "Button Unknown" (259)
        Evdev Middle Button Emulation (269):    0
        Evdev Middle Button Timeout (270):      50
        Evdev Wheel Emulation (271):    0
        Evdev Wheel Emulation Axes (272):       0, 0, 4, 5
        Evdev Wheel Emulation Inertia (273):    10
        Evdev Wheel Emulation Timeout (274):    200
        Evdev Wheel Emulation Button (275):     4
        Evdev Drag Lock Buttons (276):  0

To properly "decelerate" the mouse pointer, we'll need to play around with 2 properties:

• Device Accel Constant Deceleration (261)
• Device Accel Adaptive Deceleration (262)

To update a property value, use the "--set-prop" option like below:

mike@myhost:~$ xinput --set-prop 8 261 4

Note the values used:

• 8: device id
• 261: property id
• 4: value of the property to be set

Setting both property value to 4 works for me; YMMV.

Have fun tweaking your input :)

Scrollback in GNU Screen

This has always evaded me: How do you do scroll up the screenbuffer when using screen?  Found the answer right here:

http://www.linuxscrew.com/2008/11/14/faq-how-to-scrollback-in-gnu-screen/

I've tried the Ctrl + A + [ method; but not Ctrl + Esc as I'm using Windows as my main machine and Ctrl + Esc can also be used as the Windows key.

Just hope I remember this :)

Measuring network throughput with iperf

The iperf tool has been around for ages and yeah, I've been late to the game :)

To install iperf in Ubuntu, it's a simple apt-get away:

$ sudo apt-get install iperf

To run the test, you'll need 2 machines.  One to act as the server and the other as the client.  Start by starting the server:

$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------

If there are no errors, we can then start the test by executing iperf in the client machine:

$ iperf -c server -t 10
------------------------------------------------------------
Client connecting to server, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.100 port 40913 connected with 192.168.0.101 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec    164 MBytes    138 Mbits/sec

The argument -t 10 states that we want the test to run for only 10 seconds. 

There are other options which you can play with like:

    -d ==> Do a bidirectional test simultaneously
    -u ==> use UDP rather than TCP
   
View the list of options by executing:

$ iperf --help

There's also a GUI wrapper written in Java for iperf available at http://code.google.com/p/xjperf/.  It offers a nice GUI which allows you to choose different options to run iperf with and also displays output from iperf in a nice graph:

Copying a running VM to external host

Friend had a small incident the other day where KVM/QEMU seg-faulted and lost communication with a running VM.  The VM was still running but it isn't writing to persistent storage.  We decided to do a live backup of the running VM.  Although this is definitely NOT recommended, we were left with no other choice as shutting down the VM would mean losing all changes made. 

The VM is running Linux with non LVM-ed partitions.

To start, run the following command as root in the VM to make a copy of the drive and send it over SSH to an external host:

dd if=/dev/sda bs=1k conv=sync,noerror | gzip -c | ssh -c blowfish myuser@extern.host "dd of=/images/damaged-vm.gz bs=1k"

Once that's done, access the host where the file is transferred to and decompress the file:

# gunzip damaged-vm.gz

You can try to either boot up the image via virsh or mount it via loopback to copy any files.  We chose the latter. 

Run the following commands as root.

Find the first unused loop device:

# losetup -f
/dev/loop0

Now we setup the loop device against the image that was transferred:

# losetup /dev/loop0 /images/damaged-vm

Use kpartx to map the partitions:

# kpartx -av /dev/loop0
add map loop0p1 : 0 29333504 linear /dev/loop0 2048
add map loop0p5 : 0 1380352 linear /dev/loop0 29337600

Proceed to mount the partitions:

# mount /dev/mapper/loop0p5 /mnt/p5

References:

• http://www.inference.phy.cam.ac.uk/saw27/notes/backup-hard-disk-partitions.html

Creating network aliases for single network interface

First some assumptions:

• network interface is aliased to eth0
• eth0 assigned to 192.168.1.1
  

Here's how you assign more than one IP to a single network interface in Ubuntu/Debian:

# ifconfig eth0:0 192.168.1.2 up

That's it.  Wait for a while for the interface to become active and you'll be able to configure services to serve from that IP.  To make the change permanent, edit the /etc/network/interfaces file and add the following lines:

auto eth0:0
iface eth0:0 inet static
        address 192.168.1.2
        netmask 255.255.255.0
        gateway 192.168.1.0

Removing invalid/encoded characters from filenames

If you see the following output in your file system:

drwxrwxrwx 2 www-data www-data 4096 Jan  5 03:30 ?????? - ?Q?U?H-?X?Y??Like?t???

drwxrwxrwx 2 www-data www-data 4096 Jan  5 03:41 ???O?? ?@?????j?t???

It'll be difficult to work with the files.  By using "detox", you can remove all invalid characters from the files/folders.  In Ubuntu, simply perform the following command to install it:

# apt-get install detox

Then, do this:

# detox *Like*

If you like, perform a trial/dry run first by adding the --dry-run option.  Here's the output of the cleaned up folder name:

drwxrwxrwx 2 www-data www-data 4096 Jan  6 09:09 e_N-Q_U_H-yen_XAY_N_Like_t_U_

(L)AMP installation on Ubuntu

Just got a new server up and running.  Here are the commands to install Apache2, PHP5 and MySQL:

# apt-get install apache2 libapache2-mod-php5 mysql-server-5.1 php5-mysql php5-mcrypt

Make sure you do a restart.  Sometimes the apt-get process doesn't restart Apache properly to ensure the PHP5 module is loaded.

# /etc/init.d/apache2 restart

Compiling and installing libtorrent/rtorrent

Here are the steps to compile and install libtorrent/rtorrent.  The latest stable version of libtorrent/rtorrent is 0.12.6 and 0.8.6 respectively as of writing this article.  The tarballs can be obtained here.  Following commands must be run as root.

# apt-get install build-essential libssl-dev libssl0.9.8 libsigc++-2.0-dev libncurses5-dev libncursesw5-dev libcurl4-openssl-dev libssl-dev libssl0.9.8 libsigc++-2.0-dev libncurses5-dev libncursesw5-dev libcurl4-openssl-dev

# tar zxvf libtorrent-0.12.6.tar.gz

# cd libtorrent-0.12.6

# ./configure

# make

# make install

# ldconfig

# tar zxvf rtorrent-0.8.6.tar.gz

# cd rtorrent-0.8.6

# ./configure

# make

# make install

That's it :)  rtorrent is now installed in /usr/local/bin.

OpenVPN with pam_mysql username/password authentication

Here's how I setup OpenVPN with pam_mysql using only username/password authentication (insecure, but really easy on the admin side :P).  Target platform is Ubuntu Jaunty.  For starters, you'll need the following installed:

• openvpn
• mysql - server installed and running

Get the latest version of pam_mysql from http://pam-mysql.sourceforge.net/.  Untar it into a working directory.  The latest version as of writing this article is 0.7RC1.  The pam_mysql package in Jaunty is broken.  You'll need to patch the Makefile.in with the following patch (thanks to http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=418500#50):

--- Makefile.in.chold   2008-07-14 10:25:53.000000000 +0200
+++ Makefile.in 2008-07-14 10:26:06.000000000 +0200
@@ -110,7 +110,7 @@
 CPPFLAGS = @CPPFLAGS@
 LDFLAGS = @LDFLAGS@
 LIBS = @LIBS@
-pam_mysql_la_LIBADD =
+pam_mysql_la_LIBADD = -lpam
 pam_mysql_la_OBJECTS =  pam_mysql.lo
 CFLAGS = @CFLAGS@
 COMPILE = $(CC) $(DEFS) $(INCLUDES) $(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)

Save the above as patch.in in the same folder where you untar pam_mysql.  Then run the following command:

# patch -p0 <patch.in

That command will perform the patch process by using the patch.in file.  Next, we'll need to grab some deb packages to compile the module:

# apt-get install libmysql++-dev libpam0g-dev libgsasl7 libgsasl7-dev

Once that's done, proceed with the usual make commands (please note the extra configure options to enable SHA1 and MD5 options in the module):

# ./configure --with-cyrus-sasl2 --with-openssl
# make
# make install

The module should now be installed in /lib/security/pam_mysql.so.  Now to configure OpenVPN.  Here's my server configuration file:

port 1194
proto udp
dev tun

ca /etc/openvpn/easy-rsa/ca.crt
cert /etc/openvpn/easy-rsa/server.crt
key /etc/openvpn/easy-rsa/server.key
dh /etc/openvpn/easy-rsa/dh1024.pem

server 10.128.127.0 255.255.255.0
ifconfig-pool-persist ipp.txt

push "redirect-gateway def1"
push "dhcp-option DNS 1.1.1.1"
push "dhcp-option DNS 2.2.2.2"

keepalive 10 120

comp-lzo

max-clients 50

persist-key
persist-tun

status openvpn-status.log
log-append /var/log/openvpn.log
verb 3
mute 20

client-cert-not-required
username-as-common-name

plugin /usr/lib/openvpn/openvpn-auth-pam.so openvpn

The most important lines are the last three lines which have been bold.  Follow the commands below to setup OpenVPN server:

cd /etc/openvpn
cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/ easy-rsa
cd easy-rsa
vim vars # Edit KEY_* vars appropriately
source ./vars
./clean-all
./build-dh
./pkitool --initca
./pkitool --server server

To allow traffic to be routed from clients to the server, perform the following tasks:

echo 1 > /proc/sys/net/ipv4/ip_forward
edit /etc/sysctl.conf
Uncomment/add the line: net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 10.128.127.0/24 -o eth0 -j MASQUERADE
iptables-save

Now to configure PAM.  Create a file called openvpn in /etc/pam.d.  Below are its contents:

auth optional /lib/security/pam_mysql.so user=root passwd=pass host=localhost db=vpn_db table=tbl_user usercolumn=userid passwdcolumn=password where=active=1 sqllog=no crypt=4 verbose=0

account required /lib/security/pam_mysql.so user=root passwd=pass host=localhost db=vpn_db table=tbl_user usercolumn=userid passwdcolumn=password where=active=1 sqllog=no crypt=4 verbose=0

crypt=4 instructs pam_mysql to use SHA1.  There are other options which you can view in the pam_mysql README file.  OpenVPN client configuration file is shown below:

client
dev tun
proto udp
remote my.vpn.server.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
auth-user-pass
cipher BF-CBC
comp-lzo
verb 4
mute 20

Very lean and clean client config file.