Yearly Archives: 2011

Using motd to warn you of upcoming fsck

Has it ever happened to you that you remotely rebooted a Linux server and wondered why after 10 minutes you still couldn’t Ping or SSH into it? Quite often, this is caused by a lengthy run of fsck on one of your ext3 file systems (on current multi-TB disks, they can take an hour or longer). They usually get triggered automatically around every six months or 30 mounts, and there’s no easy way of knowing when it will happen.

My solution to this is rather simple: tune2fs -l tells you when the next fsck is coming up, so why not run a script upon every boot and have it write that information into /etc/motd so that you can see it every time you SSH in? The script is below (as a bonus, drives being fscked in less than 5 mounts or a week are printed in red), so all you need to do is put it somewhere on your hard drive and add something like
# Add FSCK status to MOTD
[ -f /root/ ] && bash /root/ >> /var/run/motd

to the init script that generates your motd (on Debian Squeeze, that would be /etc/init.d/bootlogs).



for disk in /dev/sda1 /dev/mapper/vm--storage-vms /dev/sdb1
cur_mounts=$(tune2fs -l $disk | grep "Mount count:" | awk '{print $3}')
max_mounts=$(tune2fs -l $disk | grep "Maximum mount count:" | awk '{print $4}')
diff_mounts=$(echo $max_mounts-$cur_mounts | bc)
last_check=$(tune2fs -l $disk | grep "Last checked:" | awk '{print $3}')
next_check=$(tune2fs -l $disk | grep "Next check after:" | awk '{print $4" "$5" "$6" "$7" "$8}')
next_check_timestamp=$(date -d "$next_check" "+%s")
cur_timestamp=$(date "+%s")
diff_next=$(echo $next_check_timestamp-$cur_timestamp | bc)

[ $diff_mounts -lt 5 ] && color="\033[31m"
[ $diff_next -lt 604800 ] && color="\033[31m"

echo -e "$color Next FSCK on $(basename $disk): $next_check or in $diff_mounts mounts\033[0m"

IPv6 router on Linux

Setting up Linux as an IPv6 router is really easy. Even if your ISP doesn’t do IPv6 yet (like mine), there’s no reason not to get an IPv6 tunnel from and be IPv6-ready within minutes.

  1. Do a basic install of your favorite Linux distribution.
    Since my server runs Xen, I just did xen-create-image –ip= –netmask= –gateway= –nameserver= –mirror= –passwd –hostname=router-ipv6 –dist=squeeze –arch=i386 –size=4G –swap=1G –dir=/data/vms/router-ipv6 –memory=64M –role=udev –pygrub and ran ln -s /etc/xen/router-ipv6.cfg /etc/xen/auto to have it automatically start upon reboot. To start the VM, do xm create router-ipv6.cfg
  2. SSH into the virtual machine and configure the LAN and the WAN interface. Since I’m using a tunnel, my WAN interface is a 6in4 interface; if you’re using a physical one you’ll need to manually edit the Xen VM config file to add the physical interface to the VM. So we’re adding the following lines to /etc/network/interfaces
    iface eth0 inet6 static
    address 2001:470:xxxb:xxxx::1
    netmask 64

    auto 6in4
    iface 6in4 inet6 v4tunnel
    address 2001:470:xxxa:xxxx::2
    netmask 64
    gateway 2001:470:xxxa:xxxx::1
    up ip route add ::/0 dev 6in4

  3. Next, edit /etc/sysctl.conf and set net.ipv6.conf.all.forwarding=1 by removing the comment sign from the beginning of the line.
  4. apt-get install radvd and then edit /etc/radvd.conf to look like this:
    interface eth0
    AdvSendAdvert on;
    AdvLinkMTU 1280;
    prefix 2001:470:xxxb:xxxx::1/64
    AdvOnLink on;
    AdvAutonomous on;
    RDNSS 2001:470:xxxb:xxx:yyyy:yyyy:yyyy:yyyy

    Most of this is pretty self-explanatory (the prefix line should contain the address of the router’s network interface and everything else just enables router advertisements), however the RDNSS line needs to point to the IPv6 address (it will automatically get one after you finish step 6) of your local DNS forwarder.
  5. Next, you’ll probably want to configure the firewall so that your computers can’t be accessed from outside (remember, with IPv6 every device gets a publicly routable address). apt-get install shorewall6 and then edit the following files to configure it:
    In /etc/default/shorewall6: startup=1 (enables the firewall) and wait_interface="6in4" (your WAN interface)
    In /etc/shorewall6/zones: Add the lines fw firewall, net ipv6 and loc ipv6
    In /etc/shorewall6/interfaces: Add the lines net 6in4 detect and loc eth0 detect
    In /etc/shorewall6/policy: Add the lines net all REJECT notice, loc all ACCEPT, fw all ACCEPT and all all REJECT notice
    In /etc/shorewall6/rules: Configure the firewall rules to your liking. I added Ping(ACCEPT) all all to allow incoming pings (I don’t believe in this security-by-obscurity stuff). I also added ACCEPT all loc:2001:470:xxxb:xxxx:zzzz:zzzz:zzzz:zzzz because that machine has its own IPv6-configured firewall.
  6. Reboot the VM.

All your IPv6-ready clients should start picking up addresses automatically. Linux, Mac OS X and iPhones do as expected and base their IP on the MAC address. Windows 7 does the same, but also makes up a random IP which gets used by default for all outgoing connections due to privacy reasons. On Windows XP, you need to manually add IPv6 to the network protocols in the network connection properties, after which it’ll behave similarly to Windows 7.
The DNS server announced by radvd however only gets picked up by the iPhone. Mac OS X only supports manually-configured IPv6 DNS servers as far as I can tell. Windows automatically configures fec0:0:0:ffff::1, fec0:0:0:ffff::2 and fec0:0:0:ffff::3 as its DNS servers; you could add one of these addresses to your DNS server (and add some other address in the fec0:0:0:ffff::/64 range to your IPv6 router VM’s LAN interface so that clients can actually find a route to it), but unfortunately the site-local prefix fec0::/10 has been deprecated for more than half a decade and should no longer be used. But fear not, it’s perfectly fine to talk to your DNS server using IPv4 – it will still resolve AAAA (IPv6 A) queries without issues. And I expect IPv4 to stay around for at least another decade, so you’re not likely to run into trouble for a long time.

    ISC DHCPd: Dynamic DNS updates against secure Microsoft DNS

    UPDATE 2016: I have posted a much simpler way that works with DNS delegations so that you can have your domain controllers maintain the records necessary for their discovery in Microsoft DNS, while all your clients are in a BIND DNS server which can be easily interfaced with ISC DHCPd.

    ISC DHCPd is capable of Dynamic DNS updates against servers like BIND that support shared-key authentication or any other server that supports unauthenticated updates (such as BIND or Microsoft DNS with secure updates disabled).

    So, what to do if you want to run ISC DHCPd on your Windows network, which is obviously running Microsoft’s DNS server? BIND’s nsupdate tool supports Microsoft’s Kerberos authentication scheme when using the -g flag (the -o flag is only necessary for Windows 2000 Server, but not anymore for Windows Server 2008 R2), and DHCPd supports on commit/release/expiry blocks that let you run scripts upon these events. So here is my script:




    export KRB5CCNAME="/tmp/"

    keytab can be generated using

    $ ktutil

    ktutil: addent -password -p dhcpduser@EXAMPLE.COM -k 1 -e aes256-cts-hmac-sha1-96

    Password for dhcpduser@EXAMPLE.COM:

    ktutil: wkt dhcpduser.keytab

    ktutil: quit


    name=$(echo $3 | awk -F '.' '{print $1}')

    echo "USAGE:"
    echo $0 add testhost 00:11:22:33:44:55
    echo $0 add "" 00:11:22:44:33:55
    echo $0 delete testhost 00:11:22:33:44:55
    echo $0 delete "" 00:11:22:44:33:55

    if [ "$ip" = "" ]; then
    echo "IP missing"
    exit 101
    if [ "$name" = "" ]; then
    #echo "name missing"
    #exit 102
    name=$(echo $ip | awk -F '.' '{print "dhcp-"$1"-"$2"-"$3"-"$4}')

    if [ "$action" = "delete" ]; then
    name=$(host $ip | awk '{print $5}' | awk -F '.' '{print $1}')

    echo $name | grep NXDOMAIN 2>$1 >/dev/null
    if [ "$?" = "0" ]; then
    exit 0;

    ptr=$(echo $ip | awk -F '.' '{print $4"."$3"."$2"."$1""}')


    #export LD_LIBRARY_PATH=/usr/local/krb5-1.7/lib
    #export PATH=/usr/local/krb5-1.7/bin:$PATH

    klist 2>&1 | grep $realm | grep '/' > /dev/null
    if [ "$?" = 1 ]; then
    expiration=$(klist | grep $realm | grep '/' | awk -F ' ' '{system ("date -d \""$2"\" +%s")}' | sort | head -n 1)

    now=$(date +%s)
    if [ "$now" -ge "$expiration" ]; then
    echo "Getting new ticket, old one expired $expiration, now is $now"
    kinit -F -k -t $keytab $principal


    case "$action" in
    echo "Setting $name.$domain to $ip on $ns"

    oldname=$(host $ip $ns | grep "domain name pointer" | awk '{print $5}' | awk -F '.' '{print $1}')
    if [ "$oldname" = "" ]; then
    elif [ "$oldname" = "$name" ]; then
    echo "Also deleting $oldname A record"

    nsupdate -g <
    server $ns
    realm $realm
    update delete $oldname.$domain 3600 A
    update delete $name.$domain 3600 A
    update add $name.$domain 3600 A $ip
    nsupdate -g <
    server $ns
    realm $realm
    update delete $ptr 3600 PTR
    update add $ptr 3600 PTR $name.$domain

    echo "Deleting $name.$domain to $ip on $ns"
    nsupdate -g <
    server $ns
    realm $realm
    update delete $name.$domain 3600 A
    nsupdate -g <
    server $ns
    realm $realm
    update delete $ptr 3600 PTR
    echo "Invalid action specified"
    exit 103

    if [ "$result" != "00" ]; then
    echo "DHCP-DNS Update failed: $result"
    logger "DHCP-DNS Update failed: $result"

    exit $result

    and here is the relevant part of my dhcpd.conf:

    on commit {
    set noname = concat("dhcp-", binary-to-ascii(10, 8, "-", leased-address));
    set ClientIP = binary-to-ascii(10, 8, ".", leased-address);
    set ClientMac = binary-to-ascii(16, 8, ":", substring(hardware, 1, 6));
    set ClientName = pick-first-value(option host-name, host-decl-name, config-option host-name, noname);
    log(concat("Commit: IP: ", ClientIP, " Mac: ", ClientMac, " Name: ", ClientName));

    execute("/root/", "add", ClientIP, ClientName, ClientMac);
    on release {
    set ClientIP = binary-to-ascii(10, 8, ".", leased-address);
    set ClientMac = binary-to-ascii(16, 8, ":", substring(hardware, 1, 6));
    log(concat("Release: IP: ", ClientIP, " Mac: ", ClientMac));

    cannot get a ClientName here, for some reason that always fails

    execute("/root/", "delete", ClientIP, "", ClientMac);
    on expiry {
    set ClientIP = binary-to-ascii(10, 8, ".", leased-address);

    cannot get a ClientMac here, apparently this only works when actually receiving a packet

    log(concat("Expired: IP: ", ClientIP));

    cannot get a ClientName here, for some reason that always fails

    execute("/root/", "delete", ClientIP, "", "0");

    Figuring this all out took me several afternoons because Kerberos 5 1.8 has a bug where forwardable tickets (which is the default on Debian) are incompatible with nsupdate. Manually compiling 1.7 or getting 1.9 from the experimental Debian branch helps, as does adding the -F flag to kinit (which I did in the script above) to make the ticket non-forwardable.
    I filed a bug with Debian (#611906) and Sam Hartman (thanks!) helped me track it down.

    EDIT 2011-11-17:
    I recently ran into the issue that if the AD server could not be reached, dhcpd would stall (and not respond to DHCP requests during that time) until nsupdate reached its timeout. The fix is simple: rename to and create with the following contents to fork off the real script into the background:

    $(dirname $0)/ $@ 2>&1 | logger &

    Also, I updated the on commit section in the dhcpd.conf excerpt above to compose a fallback name from the IP address if the client provides no hostname. This fixes the issue that nsupdate tries to register a record based on the name and fails.

    Extending Active Directory for Mac OS X clients

    After I wrote about building your own OpenDirectory server on Linux a while back, I decided to do the same thing on Windows Server 2008 R2. The process of extending the AD schema to include Apple classes and attributes is documented by Apple (this is the Leopard version of the document – if you don’t plan on having exclusively Snow Leopard clients, you can follow the newer version of the document that skips a couple of things that Snow Leopard no longer needs).

    But since schema extensions are generally frowned upon in the Windows world because they’re irreversible (why the heck, Microsoft…?), I initially tried a dual-directory (golden triangle, magic triangle) type approach where I’d be augmenting my AD with Apple records coming from an AD LDS (Active Directory Lightweight Directory Services, previously called ADAM, Active Directory User Mode, which is basically a plain LDAP server from Microsoft). While this may sound like a great idea, I just couldn’t get it to work. After dozens of manual schema extensions to AD LDS (Microsoft doesn’t include many standard LDAP attributes, so I had to dig through the dependencies of apple.schema and even tried importing a complete OD schema), I gave up because I could not get Workgroup Manager to authenticate against it to allow me to make changes.

    So the next thing to do was follow Apple’s AD schema extension guide (linked above) and do what everybody else did. This was rather straight-forward (managed preferences for users, groups and computers worked right away), but when I tried to create a computer list (which is not possible using Snow Leopard’s Server Admin Tools, but requires Tiger’s (which throw loads of errors on Snow Leopard but still get the job done) since Leopard introduced computer groups which however are not supported by the AD plugin), it just said I didn’t have permission to do that. After enabling DirectoryService debug logging (killall -USR1 DirectoryService && killall -USR2 DirectoryService), I traced it down to Active Directory: Add record CN=Untitled_1,CN=Mac OS X,DC=xxx,DC=zz with FAILED – LDAP Error 19 in /Library/Logs/DirectoryService/*. Apparently, that’s caused by some versions of ADSchemaAnalyzer setting objectClassCategory to 0 instead of 1 on all exported classes. Too bad AD schema extensions are irreversible and that’s one of the attributes you can’t change later on… 🙁 Well, with AD Schema Management MMC snap-in, I was able to rename the botched apple-computer-list class, defunct it and add a new one using ldifde. With some really wild hacking in the AD Schema using ADSI Editor, I was then able to  eventually get OS X to no longer look at the renamed attribute, but instead at the new one. To see whether you have been successful, killall DirectoryService, wait a few seconds and grep -H computer-list /Library/Preferences/DirectoryService/ActiveDirectory* will show a line indicating which class in the schema it’s using.

    Once you’re there, everything should work as expected. If you don’t want to use Tiger’s Workgroup Manager to create old-style computer lists, you can do that in ADSI Editor and create apple-computer-list objects in the CN=Mac OS X branch by hand.

    So, attached is the schema ldif that’s exactly the way it should be. I really wonder why Apple doesn’t provide it themselves – it’s going to turn out exactly like that every time you follow their guide on any Windows server… Apple Schema for Active Directory

    I guess that the overall conclusion of this should be that AD schema extensions in general and specifically Mac OS X managed clients in AD environments are a nasty hack. I suppose the dual directory/magic triangle/golden triangle approach with a Microsoft AD and an Apple OD would work, but it requires maintaining two separate directories, which may not be that great in a larger environment either.

    If Apple discontinues Mac OS X Server at some point in the near future (which the demise of the Xserve and the lack of announcements regarding Mac OS X 10.7 Server alongside Mac OS X Lion suggest), this is definitely something they need to improve. There are some third-party solutions that store MCX settings outside of AD (similar to Windows GPOs, which are stored on the SYSVOL share) such Thursby ADmitMac – however that’s a rather expensive solution (a dozen client licenses costs about as much as two Mac mini servers) and might break after OS updates (though from what I’ve heard, they’re rather quick at providing updates). If Apple does discontinue Mac OS X Server, they should definitely improve Lion’s AD integration to replicate ADmitMac’s features.