Category Archives: Linux

OpenVPN for iOS

Today, OpenVPN Technologies released OpenVPN Connect for iOS. Finally, we can use OpenVPN on all major platforms. I know many of my blog readers have been waiting for this: my article on the iOS VPN API is one of the most popular articles on my blog.

OpenVPN Connect is not based on the classic GPL OpenVPN software (supposedly GPL and App Store are not compatible), but supposed to be fully compatible with any OpenVPN server running version 2.1 or higher (including IPv6 support with servers running the recently-released version 2.3). Supposedly it can even be managed using the “Custom SSL” option in iPhone Configuration Utility.

Two points I’d like to mention which might temporarily disappoint some people:

  • It currently requires client certificates (but the help promises that that’ll change soon).
  • Layer 2 tap interfaces are not supported. As I noted in my VPN API blog post, iOS provides a utun interface, which only does layer 3.

Go check it out on the App Store or have a look at Gert Döring’s Google+ post.

Update December 2013: Version 1.0.2 (just released) finally works for me. 1.0.0 didn’t work without client certificates and 1.0.1 had some weird SSL library issue where it would reject my server certificate. In 1.0.2 I was  able to just drop my .ovpn file into iTunes and was up and running immediately, including IPv6 support.

Converting Xen Linux VMs to VMWare

A year ago I wrote about how to convert from Xen to VMWare (which is a similar process to a Xen virtual-to-physical or V2P conversion). Now I found a much simpler solution, thanks to http://www.zomo.co.uk/2012/04/moving-disks-from-xen-to-kvm/ .

In this example, I’m using LVM disks, but the process is no different from using Xen disk images.

  1. Install Debian Wheezy into a VMWare virtual machine. Attach a secondary virtual disk (it will be called /dev/sdc from now on) that’s sized about 500 MB larger than your Xen DomU (just to be safe). Fire up the VM. All subsequent commands will be run from inside that VM.
  2. Check whether your DomU disk has a partition table: ssh root@xen fdisk -l /dev/xenvg/4f89402b-8587-4139-8447-1da6d0571733.disk0. If it does, proceed to step 3. If it does not, proceed to step 4.
  3. Clone the Xen DomU onto the secondary virtual disk via SSH: ssh root@xen dd bs=1048576 if=/dev/xenvg/4f89402b-8587-4139-8447-1da6d0571733.disk0 | dd bs=1048576 of=/dev/sdc. Proceed to step 7.
  4. Zero out the beginning of the target disk: dd if=/dev/zero of=/dev/sdc bs=1048576 count=16
  5. Partition it and add a primary partition 8 MB into the disk: fdisk /dev/sdc, o Enter w Enter, fdisk /dev/sdc, n Enter p Enter 1 Enter 16384 Enter Enter, w Enter
  6. Clone the Xen DomU onto the secondary virtual disk’s first partition via SSH: ssh root@lara dd bs=1048576 if=/dev/xenvg/4f89402b-8587-4139-8447-1da6d0571733.disk0 | dd bs=1048576 of=/dev/sdc1
  7. reboot
  8. Mount the disk: mount -t ext3 /dev/sdc1 /mnt; cd /mnt
  9. Fix fstab: nano etc/fstab: change root disk from to /dev/sda1
  10. Fix the virtual console: nano etc/inittab: replace hvc0 with tty1
  11. Chroot into the disk: mount -t proc none /mnt/proc; mount -t sysfs none /mnt/sys; mount -o bind /dev /mnt/dev; chroot /mnt /bin/bash
  12. Fix mtab so the Grub installer works: grep -v rootfs /proc/mounts > /etc/mtab
  13. Install Grub: apt-get install grub2. When the installer asks to which disks to install, deselect all disks.
  14. Install Grub to MBR: grub-install –force /dev/sdc
  15. Update Grub configuration: update-grub
  16. Leave the chroot: exit; umount /mnt/* /mnt
  17. shutdown

Now you can detach the secondary virtual disk and create a new VM with it. If everything worked correctly, it will boot up.

Hashing and verifying LDAP passwords in PHP

I recently migrated a PHP web application that used LDAP for authentication and MySQL for data to something entirely MySQL based. I needed the users to be able to continue using their old LDAP passwords, so I dumped the LDAP database and grabbed the userPassword field for each user, base64_decode()d it and wrote that to a MySQL table. These password hashes start with something like {crypt}, {MD5}, {SHA1} or {SSHA1}, or in very rare cases, are plain-text.

Here’s a PHP function I wrote that, given a plain-text $password, verifies it against such a $hash. This is what you’ll be calling from your authentication script to verify a given password against the hash.

function check_password($password, $hash)
 {
 if ($hash == '') // no password
 {
 //echo "No password";
 return FALSE;
 }
 
 if ($hash{0} != '{') // plaintext password
 {
 if ($password == $hash)
 return TRUE;
 return FALSE;
 }
 
 if (substr($hash,0,7) == '{crypt}')
 {
 if (crypt($password, substr($hash,7)) == substr($hash,7))
 return TRUE;
 return FALSE;
 }
 elseif (substr($hash,0,5) == '{MD5}')
 {
 $encrypted_password = '{MD5}' . base64_encode(md5( $password,TRUE));
 }
 elseif (substr($hash,0,6) == '{SHA1}')
 {
 $encrypted_password = '{SHA}' . base64_encode(sha1( $password, TRUE ));
 }
 elseif (substr($hash,0,6) == '{SSHA}')
 {
 $salt = substr(base64_decode(substr($hash,6)),20);
 $encrypted_password = '{SSHA}' . base64_encode(sha1( $password.$salt, TRUE ). $salt);
 }
 else
 {
 echo "Unsupported password hash format";
 return FALSE;
 }
 
 if ($hash == $encrypted_password)
 return TRUE;
 
 return FALSE;
 }

And here’s one that make a {SSHA} hash from a password (I did not implement all the other algorithms as by today’s standards, they are no longer secure). This is what you’ll be calling from your change password script to hash the password for storing in the database.

function hash_password($password) // SSHA with random 4-character salt
 {
 $salt = substr(str_shuffle(str_repeat('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789',4)),0,4);
 return '{SSHA}' . base64_encode(sha1( $password.$salt, TRUE ). $salt);
 }

Integrating BIND with AD-integrated Microsoft DNS

I recently set up BIND9 to run secondary zones for an ActiveDirectory-integrated DNS server (the reason being that I hated effectively losing internet access when I rebooted my W2k8R2 server). While that was really easy (add the Linux server to the nameservers tab in DNS Admin, allow zone transfers and notifications, add slave zones in the named.conf), I thought that it shouldn’t be too difficult to also automatically replicate AD-integrated Conditional Forwarders.

While they are easily found in the DC=DomainDnsZones and DC=ForestDnsZones branch inside the AD, it turns out that the server information is stored in dnsproperty attributes containing binary data. However, Microsoft actually provides a specification for their DNS data structures, which is certainly very commendable. But as it turns out, it appears to have been written by someone who had no clue about Endianness or how many bits are in a byte (*).
The essence is: everything is Big Endian, except for IP addresses (the spec claims they are Network Byte Order, but in reality they are Little Endian), and every occurence of “1 byte” in section 2.3.1.1 dnsProperty should be replaced with “4 byte”.

So after taking about two hours for something that I expected would only take a couple minutes to hack together, I ended up with 400 lines of code that generate a file you can include in your named.conf that will look something like this:
zone "google.com" {
type forward;
forward first;
forwarders { 74.82.42.42; 2001:470:20:0:0:0:0:2; };
};

zone “youtube.com” {
type forward;
forward first;
forwarders { 74.82.42.42; 2001:470:20:0:0:0:0:2; };
};
(For those curious, this sample configuration would point google.com and youtube.com at Hurricane Electric’s DNS server so that you get AAAA records, a.k.a. Google over IPv6)

After this worked, I decided to also pull my slave zone definitions through the same mechanism. It only took me a minute to do that.
zone "example.com" {
type slave;
file "slave_example.com";
masters { 10.0.0.1; };
allow-notify { 10.0.0.1; };
};

So here we are: BIND9 as a fully-blown sync partner for AD-integrated DNS zones. To add a zone or conditional forwarder to BIND, add it to AD, set it to replicate to all DNS/domain controllers in this domain or forest, add the BIND server to the nameservers tab and allow zone transfers and notifications, and wait for the cron job to kick in.

I ended up having to write this script in PHP because Python’s LDAP module appears to have a broken SASL implementation, and you need SASL to use Kerberos for an LDAP connection.

The PHP script takes two parameters (1. the AD server’s address or the AD DNS domain name; 2. the AD base DN) and requires a valid Kerberos ticket.
The shell script (which you will most likely want to run from a cron job), which shares much of its code with my script from ISC DHCPd: Dynamic DNS updates against secure Microsoft DNS, needs to be configured with your realm, domain, base DN, user name (principal) and path to a keytab for that user (instruction on how to generate the keytab using ktutil are in the script’s comments).

(*) After doing all this, I figured that people from projects like Samba that write open source software to re-implement or interface with Microsoft products are doing an absolutely amazing job. They most likely aren’t getting any better specs than the one I found on MS DNS (if they get specs at all), and yet still somehow create almost perfect software that is a lot more complex than the simple stuff I did here.

UPDATE 2011-10-30: Apparently, AD refuses all requests from Linux clients that come in via IPv6. To force IPv4, line 7 of the PHP script needs to be changed to $conn = ldap_connect(gethostbyname($adserver), 389);, which is also fixed in the downloadable script.

How-To: Converting Xen Linux VMs to VMWare ESXi

I have a couple Linux VMs I created on Xen using xen-create-image (as such, they are using pygrub and have one virtual disk file per partition). Now I want to migrate those over to a VMWare ESXi box. To convert your raw Xen disk images to VMWare vmdk files, do this:

1. In VMWare Fusion or Workstation, do a basic install of Debian Squeeze onto a flat-file (not split into 2GB segments and preallocated) VMDK that is slightly larger than your virtual Xen disk with a separate VMDK for swap.
2. Downgrade it to Grub 1 using apt-get install grub-legacy, grub-install /dev/sda, update-grub (as Grub 2 is not compatible with /boot/grub/menu.lst files as generated by xen-create-image).
3. Shut down and make a copy of the VMDK.
4. Boot the VM back up and re-install Grub2 using apt-get install grub.
5. Edit /boot/grub/grub.cfg and replace root=UUID=xxxxxxxxxx in the linux lines with root=/dev/sda1
6. Shut down the VM and attach the VMDK you copied in step 3 as an additional disk (this will be the target disk for our conversion).
7. Boot it up and make sure that you’re getting a Grub2 screen (i.e. it is not booting from the copied VMDK).
8. Using mount, check that your root disk is sda1 (which usually should be the first disk, not the copied disk). Using ls /dev/sd*, make sure it sees the target disk as sdc.
9. dd if=/path/to/xen/vm/disk.img of=/dev/sdc1 bs=1048576
10. mount /dev/sdc1 /mnt; cd /mnt
11. nano etc/fstab: replace swap disk /dev/xvda1 with /dev/sdb1 and root disk /dev/xvda2 with /dev/sda1
12. nano etc/inittab: replace hvc0 with tty1
13. nano boot/grub/menu.lst: replace /dev/xvda2 with /dev/sda1
14. umount /mnt
15. Attach the new virtual disk to a VM and boot a rescue system. There, drop to a shell on /dev/sda1 and apt-get update, apt-get install grub
16. Reboot
17. Done!

Xen 4.0 and Citrix WHQL PV drivers for Windows

Xen 4.0 is supposed to be able to use Citrix’s WHQL certified Windows paravirtualization drivers. Their advantage over the GPLPV drivers is that they are code-signed, meaning they run on 64-bit Windows without disabling some of Windows’ security features.

UPDATE 2011-10-17: Signed GPLPV drivers are now available. I have not yet tested them, but I assume the fix below is no longer necessary.

While the Citrix drivers included in XenServer 5.5 work (after making a single registry tweak), the more recent ones included in e.g. Xen Cloud Platform 1.0 do not work right away:

If you install the XCP drivers, make that registry tweak and reboot the DomU, you’ll notice messages like XENUTIL: WARNING: CloseFrontend: timed out in XenbusWaitForBackendStateChange: /local/domain/0/backend/console/[id]/0 in state INITIALISING; retry. in your /var/log/xen/qemu-dm-*.log and Windows just gets stuck during boot and keeps spinning forever. To get it back to work, you’ll need to
xenstore-rm /local/domain/0/backend/console/[id]
xenstore-rm /local/domain/0/backend/vfb/[id]

after starting the VM (thanks to Keith Coleman‘s mailing list post!).

To automatically run these commands upon DomU start, create a script named /usr/lib/xen/bin/qemu-dm-citrixpv with the following contents
#!/bin/sh

xenstore-rm /local/domain/0/backend/console/$2
xenstore-rm /local/domain/0/backend/vfb/$2

sh -c "sleep 10; xenstore-rm /local/domain/0/backend/console/$2; xenstore-rm /local/domain/0/backend/vfb/$2" &

exec /usr/lib/xen/bin/qemu-dm $*
and chmod +x it.

Then, edit your DomU config file and modify the device_model line and point it to your new script:
device_model = '/usr/lib/xen/bin/qemu-dm-citrixpv'

Now your Windows Server 2008 R2 x64 HVM-DomU is all set!

Asterisk: Remotely retrieving voicemail by pressing *

Many howtos around the internet on how to remotely access your voicemail box involve a dedicated extension reachable from the outside or an IVR menu entry. But wouldn’t it be much nicer if you could just press the * DTMF key during the announcement? Turns out, this is quite simple:

[incoming-external]
exten => s,1,Dial(SIP/1234,20)
exten => s,n,Voicemail(1234,us)
exten => a,1,VoiceMailMain(1234)
exten => a,n,Hangup()

And it even works when you’re using macros (like I am):

[incoming-external]
exten => 5551234,1,Macro(incoming-plus-voicemail,SIP/1234,20,1234)
exten => 5551337,1,Macro(incoming-plus-voicemail,SIP/1337,20,1337)

[macro-incoming-plus-voicemail] ; SIP/xxx, wait time, voicemail
exten => s,1,Dial(${ARG1},${ARG2}
exten => s,n,Voicemail(${ARG3},us)
;
push * during the announcement to access your mailbox
exten => a,1,VoiceMailMain(${ARG3})
exten => a,n,Hangup()

NFS Performance: rsize and wsize

NFS is supposed to be a very simple and fast network file protocol. However, when I tried to use it on my Xen box between a Debian Squeeze DomU and an NFS server running on the Debian Squeeze Dom0, I noticed that write performance was abysmal: any write more than a couple KB in size would not only slow down to a crawl, but also bog down the DomU, making it rather difficult to even cancel the write.

After some researching and testing, I tracked it down to the rsize and wsize mount options: they specify the size of the chunks sent at a single time. Apparently, they are set to 1M if you don’t specify anything else. In my case, wsize=131072 and rsize=262144 showed the highest write and read speeds respectively. However, wsize=131072 is not too far away from the cliff after which writing drops to a crawl, so I decided to back it down to 65536.

Patching DSDT in recent Linux kernels without recompiling

Up until a year or two ago, the Linux kernel let you replace the ACPI DSDT by adding a customized version to the initrd. However, more recent versions disable that by default. If you’re using Grub2 as your bootloader though, the alternative is simple: just add acpi /boot/dsdt.aml to your Grub config. If you prefer a more elegant solution, just add the attached 01_acpi.txt to /etc/grub.d, renamed it to 01_acpi and chmod +x it; then run update-grub2 to rebuild your Grub config. It originally came from ubuntuforums.org and I removed the -e flag in the acpi line, which caused the new DSDT only to be visible to Grub, but not to the OS.

I believe the acpi command in Grub2 originally came from the Hackintosh community – messing around with DSDTs is a lot more common there because Mac OS X is rather picky.

For those of you who don’t know what I’m talking about: the DSDT describes certain hardware features your PC has, such as buttons, CPU power save modes, and lots of other things. Some mainboards have very poorly done BIOSes that have equally messed up DSDT tables. I won’t go into a lot of detail regarding how to fix those here though (it’s as simple as cat /proc/acpi/dsdt > dsdt.dat; iasl -d dsdt.dat; editing dsdt.dsl to your liking; iasl -tc dsdt.dsl (this last step will probably produce a number of errors that can be solved by googling for the error number and making the appropriate changes in dsdt.dsl)).

Another valuable hint for people messing around with DSDTs: the DSDT is not the only place that can contain this kind of information, the other place would be the SSDT and possibly additional SSDTs. You can find them in /sys/firmware/acpi/tables and decompile them just like the DSDT as described above. Instead of recompiling the SSDT by itself, you could probably also consider merging it into your custom DSDT at the appropriate places.

Using motd to warn you of upcoming fsck

Has it ever happened to you that you remotely rebooted a Linux server and wondered why after 10 minutes you still couldn’t Ping or SSH into it? Quite often, this is caused by a lengthy run of fsck on one of your ext3 file systems (on current multi-TB disks, they can take an hour or longer). They usually get triggered automatically around every six months or 30 mounts, and there’s no easy way of knowing when it will happen.

My solution to this is rather simple: tune2fs -l tells you when the next fsck is coming up, so why not run a script upon every boot and have it write that information into /etc/motd so that you can see it every time you SSH in? The script is below (as a bonus, drives being fscked in less than 5 mounts or a week are printed in red), so all you need to do is put it somewhere on your hard drive and add something like
# Add FSCK status to MOTD
[ -f /root/fsck_stats.sh ] && bash /root/fsck_stats.sh >> /var/run/motd

to the init script that generates your motd (on Debian Squeeze, that would be /etc/init.d/bootlogs).

#!/bin/bash

echo

for disk in /dev/sda1 /dev/mapper/vm--storage-vms /dev/sdb1
do
cur_mounts=$(tune2fs -l $disk | grep "Mount count:" | awk '{print $3}')
max_mounts=$(tune2fs -l $disk | grep "Maximum mount count:" | awk '{print $4}')
diff_mounts=$(echo $max_mounts-$cur_mounts | bc)
last_check=$(tune2fs -l $disk | grep "Last checked:" | awk '{print $3}')
next_check=$(tune2fs -l $disk | grep "Next check after:" | awk '{print $4" "$5" "$6" "$7" "$8}')
next_check_timestamp=$(date -d "$next_check" "+%s")
cur_timestamp=$(date "+%s")
diff_next=$(echo $next_check_timestamp-$cur_timestamp | bc)

color=""
[ $diff_mounts -lt 5 ] && color="\033[31m"
[ $diff_next -lt 604800 ] && color="\033[31m"

echo -e "$color Next FSCK on $(basename $disk): $next_check or in $diff_mounts mounts\033[0m"