Using a BIND DNS server in an Active Directory Environment

Years ago, I posted a script that allowed ISC DHCPd to update a Microsoft DNS server with dynamic records for DHCP clients. I haven’t used that method in a long time and there is a much simpler method: use ISC DHCPd together with the BIND DNS server like everybody else does, and only delegate the _mscds and _sites zones from the BIND server to the Microsoft DNS servers:

_msdcs.example.com. 86400 IN NS dc01.example.com.
_msdcs.example.com. 86400 IN NS dc02.example.com.
_sites.example.com. 86400 IN NS dc01.example.com.
_sites.example.com. 86400 IN NS dc02.example.com.

Then on all your machines, use the BIND server as DNS server (typically set via DHCP option 23). For Windows Domain matters, only records below _msdcs and _sites are ever looked up. I believe you should even be able to point your domain controllers to the BIND DNS server — they should be able to follow the NS record so that whenever they try to update their own records, they do so on the Microsoft DNS server.

For compatibility with Unix clients, you’ll want to add a couple of CNAME records for the SRV records:

_ldap._tcp.example.com. 86400 IN CNAME _ldap._tcp.dc._msdcs.example.com.
_gc._tcp.example.com. 86400 IN CNAME _ldap._tcp.gc._msdcs.example.com.
_kerberos._tcp.example.com. 86400 IN CNAME _kerberos._tcp.dc._msdcs.example.com.
_kerberos._udp.example.com. 86400 IN CNAME _kerberos._tcp.dc._msdcs.example.com.
_kpasswd._tcp.example.com. 3600 IN SRV 0 100 464 dc01.example.com
_kpasswd._tcp.example.com. 3600 IN SRV 0 100 464 dc02.example.com
_kpasswd._udp.example.com. 3600 IN SRV 0 100 464 dc01.example.com
_kpasswd._udp.example.com. 3600 IN SRV 0 100 464 dc02.example.com

The _kpasswd records unfortunately can’t be CNAMEs because they don’t exist in the _msdcs branch, so you manually need to keep them up-to-date when you add and remove domain controllers.

Boot a Windows install disc from the network using iPXE and wimboot

A while ago, I showed how you can use a Linux PXE server along with a tool called Serva to PXE boot a Windows installer DVD. By now, there is a much nicer solution available that doesn’t require any Windows tools: iPXE with wimboot. So go ahead and replace your PXELinux setup with iPXE first. Then copy the contents of a Windows installer DVD to your TFTP server and make sure that the folder is also shared read-only via SMB. Now copy the wimboot binary to your TFTP server and add something like the following to your iPXE config file:
set serverip 192.168.200.29
set tftpboot tftp://${serverip}/
set tftpbootpath /mnt/Daten/tftpboot

:menu
menu iPXE boot menu
item --key w win10de Windows 10 16.07 x64 German
choose os
goto ${os}

:win10de
echo Booting Windows Installer...
set root-path ${tftpboot}/ipxe
kernel ${root-path}/wimboot gui
set root-path ${tftpboot}/Win10_1607_German_x64
initrd ${root-path}/boot/bcd BCD
initrd ${root-path}/boot/boot.sdi boot.sdi
initrd ${root-path}/sources/boot.wim boot.wim
initrd ${root-path}/boot/fonts/segmono_boot.ttf segmono_boot.ttf
initrd ${root-path}/boot/fonts/segoe_slboot.ttf segoe_slboot.ttf
initrd ${root-path}/boot/fonts/segoen_slboot.ttf segoen_slboot.ttf
initrd ${root-path}/boot/fonts/wgl4_boot.ttf wgl4_boot.ttf
boot || goto failed

That’s it. When you boot this boot menu entry, you’ll be presented with the Windows installer, but if you click through it, it will at some point ask you for a driver because it can’t find its installer packages. So before you click through, hit Shift-F10 and execute the following commands to set up the network, mount the SMB share and re-execute the installer:
wpeinit
net use s: \\192.168.200.29\tftpboot\Win10_1607_German_x64 bar /user:foo
s:\sources\setup.exe

If your SMB server is running Samba, the user you specify (foo) must not exist on the server so you force it to use anonymous authentication. With a Windows server, things might be different.

That should give you a working installer that will get your Windows running within a few minutes because Gigabit Ethernet has a much bigger bandwidth than a spinning DVD or a cheap USB flash drive.

Serva did have one advantage, however: when you set it up, you could inject network drivers into the boot image. With Windows 10, luckily, that has become a non-issue: Microsoft releases a new installer ISO for it about once a year, which you can directly download and which should contain all drivers for the latest hardware available when it was released.

Wiring Fibre Channel for Arbitrated Loop

We have a small Fibre Channel SAN with three servers, a switch and a dual-controller RAID enclosure. With only a single switch, we obviously couldn’t connect all servers redundantly to the RAID system. That meant, for example, that firmware updates to it could only be applied after shutting down the servers. Buying a second switch was hard to justify for this simple setup, so we decided to hook the switch up to the first RAID controller and wire a loop off the second RAID controller. Each server would have one port connected to the switch and another one to the loop.

Back in the old days, Fibre Channel hubs existed for exactly this purpose, but nowadays you simply can’t get them anymore. However, in a redundant setup, you don’t need a hub, you can simply run cables appropriately, i.e. in a daisy-chain fashion. The only downside of not having a hub, the loop going down if one server goes down, is irrelevant here because you have a second path via the switch. For technical details on the loop topology, you can have a look at documentation available from EMC.

To wire your servers and RAID controllers in a fashion resembling a hub (only without the automatic bypass if a server goes down), you need simplex patch cables. These consist of a single fibre (instead of two, like you are used to) and have a connector that looks like half of a regular LC connector. You can get them in the same qualities as regular (duplex) patch cables and in single- and multi-mode as needed. For pictures, look at Fibertronics. They are somewhat exotic, so your usual cable dealer might not have them, but they exist and are available from specialized fiber cable dealers. As you are wiring in a daisy-chain fashion, you need one simplex cable per node you want to connect.

Once you have the necessary cables, you wire everything by connecting a cable from the output side of the FC port on the first device and connecting it to the input side of the FC port on the second device. The second device’s output side connects to the third device’s input side and so on, until you have made a full loop by connecting the last device’s output side to the first device’s input side. How can you tell the output side from the input side? Some transceivers have little arrows or have labels “TX” (output) and “RX” (input). If yours don’t, you can recognize the output side by the red laser light coming from it. To protect your eyes, never look directly into the laser light. Never connect two output sides together, otherwise you may damage the laser diodes in both of them. Therefore, be careful and always double-check.

That’s it, you’re done. All servers on the loop should immediately see the LUNs exported to the by the RAID controller.

In my setup however, further troubleshooting was required. The loop simply did not come up. As it turns out, someone had set the ports on one FC HBA being to point-to-point-only and 2 Gbit/s mode. After switching both these settings to their default automatic mode, the loop went up and data started flowing. My HBAs are QLogic QLE2462 (4 Gbit/s generation) and QLE2562 (8 Gbit/s generation), and they automatically negotiated the fastest common speed they could handle, which is 4 Gbit/s. Configuring these parameters on the HBA usually requires hitting a key at the HBA’s pre-boot screen to enter the configuration menu and doing it there, or via vendor-specific software. I didn’t have access to the pre-boot configuration menu and was running VMware ESXi 6.0 on the servers. The QConvergeConsole for my QLogic adapters luckily is also available for VMware. It is not as easy to install as on Windows or Linux, unfortunately. You first install a CIM provider via the command line on the ESXi host, reboot the host and then install a plugin into VMware vCenter server.

Snom VoIP phones: OpenVPN and multicast audio

Snom voice-over-IP phones have a built-in OpenVPN client, and they can also have audio transmitted to them via multicast. However, in my case a phone logged into OpenVPN did not play back audio received from the local ethernet network

I had multicast reception configured correctly on the phone:

multicast_listen=on
mc_address=239.255.255.245:5555

(Remember that you need to push the Apply button at the bottom of the page and then the Save button at the top of the page for the setting to actually take effect.)

To diagnose the problem, you can use multicast pings. Running

ping -I eth0 -t 2 239.255.255.245

on the local network showed no reactions from the phone. However, running

ping -I tun2 -t 2 239.255.255.245

on the VPN server showed responses from the phone. So clearly the phone wanted to do multicast exclusively via VPN. The cause of this was the following line in my OpenVPN config on the server:

push "redirect-gateway def1"

The obvious solution was to replace this line with routes that excluded the multicast IP addresses:

push "route 0.0.0.0 128.0.0.0"
push "route 128.0.0.0 192.0.0.0"
push "route 192.0.0.0 224.0.0.0"

While this worked, the phone wouldn’t complete booting up the next time I power-cycled it. So I switched back to the old OpenVPN config and instead turned on multicast routing on the VPN server:

apt-get install smcroute
echo 'mgroup from eth0 group 239.255.255.245' > /etc/smcroute.conf
echo 'mroute from eth0 group 239.255.255.245 to tun0 tun1 tun2' > /etc/smcroute.conf
service smcroute restart

Multicast audio now works and it even goes across the VPN.

Below you can find some commands that use FFMPEG to send the multicast audio:

ffmpeg -re -i song.mp3 -filter_complex 'aresample=16000,asetnsamples=n=160' -acodec g722 -ac 1 -vn -f rtp udp://239.255.255.245:5555
ffmpeg -re -i song.mp3 -filter_complex 'aresample=8000,asetnsamples=n=160' -acodec pcm_mulaw -ac 1 -vn -f rtp udp://239.255.255.245:5555
ffmpeg -re -i song.mp3 -filter_complex 'aresample=8000,asetnsamples=n=160' -acodec pcm_alaw -ac 1 -vn -f rtp udp://239.255.255.245:5555

Interestingly, you don’t even need to set the rtp_codec_type setting — the phone automatically determines the codec from the stream. I wasn’t able to get the Opus codec working though, the phone just makes crackling noises when I tried.

OpenMPI and SLURM CPU pinning

The SLURM scheduling and queueing system pins processes to CPUs. This worked great with OpenMPI 1.6, but OpenMPI 1.8 introduced its own pinning policy. As a result, our jobs ended up running very slowly because all jobs that used less than a whole machine ended up getting pinned to cores 0 through n, where n is the number of cores requested by a job.

Fixing this is pretty simple: just disable OpenMPI’s pinning so that the MPI processes inherit the scheduler’s pinning. To do that, add the following to your job script:

export OMPI_MCA_hwloc_base_binding_policy=none

Obviously, as a more permanent and more fool-proof solution, set that setting in the OpenMPI config file on all of your compute nodes, have the scheduler set that environment variable, or put it into your module file for OpenMPI.

Leserbrief “Bargeld”

Im Februar 2015 forderten eine Reihe von Politikern, Bargeld-Transaktionen oberhalb von 5000 Euro zu verbieten, um Geldwäsche und Finanzierung krimineller Aktivitäten zu verhindern. Um auf die Probleme verdachtsunabhängiger Datenspeicherung hinzuweisen, verfasste ich einen Leserbrief, den die Süddeutsche Zeitung am 15. Februar 2016 abdruckte:

Weiterer Baustein

[Wie bei der Vorratsdatenspeicherung für Telekommunikation belegen Studien auch der Einschränkung des anonymen Bargeldverkehrs keinen nennenswerten Nutzen für die Verhinderung von Straftaten.] Auch das organisierte Verbrechen arbeitet heute weitgehend bargeldlos. Beim Bargeld dürfte die Obergrenze nur ein erster Schritt zur vollständigen Abschaffung sein, ein weiterer Baustein der umfassenden Überwachung aller Bürger.
Datensammlungen durch und für den Staat sind grundsätzlich kritisch zu sehen: wer weiß schon, was eine Regierung in 15 Jahren mit den heute festgehaltenen Daten anstellt? [Als dystopische Beispiel sei an die Volkszählung 1930 in den Niederlanden erinnert: sie erfolgte erstmalig auf Lochkarten, weshalb die Deutschen 11 Jahre später den Lochkartensortierer nur auf die “Religionszugehörigkeit”-Spalte einstellen mussten, um systematisch mit ihren grausamen Taten zu beginnen.]

Michael Kuron, Frickenhausen

Der erste und letzte Satz sind leider der redaktionellen Kürzung zum Opfer gefallen, wofür ich Verständnis habe. Dennoch möchte ich sie hier samt Quellen erwähnen.

Triple-booting a Clamshell iBook

I have a first-generation Clamshell iBook, manufactured in January 2000. These shipped with 64 MB of on-board RAM, a 6 GB hard drive and a CD drive.
I got it in summer 2005 when my high school decommissioned these computers. Running Mac OS X 10.3.9 at the time, it was still a very usable computer, especially after I upgraded the RAM to 320 MB. I used it as my main computer for a year and a half, until I replaced it with a new Intel-based Mac.

Now, in 2016, I found a 40 GB IDE laptop hard drive and decided to put that into the Clamshell and try to make it usable again.
As usual, iFixit has a good tutorial. You have to take the entire machine apart and end up with around 50 screws, but it’s surprisingly easy. You only need a Torx T8, a Philips #1 and a 5mm nut driver.
Now, on to putting some operating systems on the hard drive.

Installing Mac OS X 10.4

Here is what you need:

  • Mac OS X 10.4 retail install DVD
  • Mac OS X 10.3 retail install CD (the first one is sufficient). Any other bootable Mac OS X 10.3 CD might also work.
  • USB hard drive

Officially, Tiger is only supported on the latest revision of the Clamshell iBook, which has FireWire and a DVD drive. Mine only has a CD drive and does not support USB booting, which makes things exceedingly difficult.

  1. Take an image of the 10.4 install DVD and save it onto the USB hard drive using a different Mac.
  2. Boot the Clamshell off the 10.3 install CD
  3. Open up Disk Utility. Partition the disk with the following sizes and names: 16 MB linux-boot, 2 GB linux-swap, 12 GB linux, 20 GB Macintosh HD, 6 GB Install. Make sure to select the Install OS 9 drivers checkbox.
  4. Plug in the USB hard drive.
  5. Clone the 10.4 install image onto the Install partition on the internal drive.
  6. Open a Terminal and bless the hard drive: bless --folder "/Volumes/Install/System/Library/CoreServices" --bootinfo "/Volumes/Install/usr/standalone/ppc/bootx.bootinfo".
  7. Using the Terminal, pico /Volumes/Install/System/Installation/Packages/OSInstall.mpkg/Contents/OSInstall.dist and remove the PowerBook2,1 from the badMachines list.
  8. Open Startup Disk and set the iBook to boot from the Install partition.
  9. Reboot
  10. Install 10.4 onto the Macintosh HD partition, deselecting all printer drivers and languages you don’t need in order to save some space.
  11. After the reboot, run Software Update a few times.

Installing Mac OS 9.2.2

Here is what you need:

Mac OS 9 is really easy to install, you just copy it to the hard drive.

  1. Mount the CD or image and copy System Folder and Applications (Mac OS 9) from the root directory to your Macintosh HD.
  2. Open the Startup Disk preference pane and select Mac OS 9.2.2.
  3. Reboot.
  4. If you used the NetBoot image, you will be asked for user name and password. Use NBUser and netboot.
  5. In order to disable the password prompt, go to the hard drive, System Folder and move the contents of Control Panels disabled to Control Panels. Now open the Multiple Users control panel, go to options and set it to local authentication. Then, disable multiple users entirely.
  6. Download and install QuickTime 6.0.3 for Mac.
  7. Open the Startup Disk control panel and select Mac OS 10.4.
  8. After the reboot, open the Classic preference pane and select the system folder on Macintosh HD.

Now you can use Classic and you can also do a native boot.

Installing Debian Linux 8 “Jessie”

Mac OS 9.2.2 hasn’t received updates in almost 15 years, and Mac OS X 10.4 is also 7 years beyond its update cycle. So how about a current operating system?

Here is what you need:

Except for the partitioning, the installation is quite straight-forward.

  1. Insert the CD and boot it by holding the C key while turning on the computer.
  2. Proceed through the installer.
  3. When asked for the partitioning, set the 16 MB partition to be NewWorld Boot and set the bootable flag. Set the 2 GB partition to be swap. Set the 12 GB partition to be ext3 and mounted at /.
  4. When asked for the packages to install, choose the Xfce desktop. It’s lightweight enough to run on this old hardware.
  5. After rebooting, run nano /etc/apt/sources.list and add the non-free repository to all entries in the file. Then, install the AirPort firmware: apt-get install linux-firmware-nonfree.

The resulting Debian install mostly worked for me, except for two things:

  • the AirPort card does not show up in NetworkManager
  • Fonts are a bit messed up. Bitmaps are drawn to the screen just fine.

I’m still looking for solutions to these two things.

Configuring the yaboot bootloader

Right now, your computer will boot into Linux all the time.
While the yaboot loader prompts you to hit x to boot into Mac OS X and m to boot into Mac OS 9, that doesn’t work as expected.
Instead of booting into Mac OS X, my iBook booted into the install partition.
And instead of booting into Mac OS 9, my iBook booted into Mac OS X.

So run nano /etc/yaboot.conf and adjust the macosx= line so that it refers to the same partition as the macos9= line. Exit the editor and run ybin -v to apply the changes. That fixes the first problem.

Now, reboot into Mac OS X and open the Startup Disk preference pane. Select Mac OS 9.2.2 and reboot.
While rebooting, hold down the option key and select the Linux partition at the boot picker.
Run nano /etc/yaboot.conf again and add brokenosx to a new line. Once more, run ybin -v.

Now, each boot loader entry does what you’d expect. The trick here is that brokenosx causes yaboot to directly load the Mac OS X booter for the macosx= entry. The macos= entry, on the other hand, will still cause the blessed system folder to be booted.

Fixing Mac OS 9 after installing Linux

One problem is still there: When you try to boot Mac OS 9, you are greeted by a blinking floppy with a question mark. This happens because the Debian partitioner destroys the Mac OS 9 drivers for the HFS+ partitions. However, the drivers can be reinstalled.

You need:

  • Mac OS 9 install CD. Any other bootable Mac OS 9 CD might also work.

Now,

  1. Insert the CD and reboot the iBook while holding down the C key.
  2. Open Drive Setup, highlight the internal hard drive, go to the Functions menu and click Update Drivers.
  3. Reboot

Resetting the Startup Disk

Whenever you touch the Startup Disk preference pane in Mac OS X or the Startup Disk control panel in Mac OS 9, your system will no longer show the yaboot prompt when you turn it on. To fix that, do the following:

  1. Hold down the option key while booting and select the Linux partition
  2. Run ybin -v

Fixing Linux problems

http://ppcluddite.blogspot.de/2012/03/installing-debian-linux-on-ppc-part-iv.html is an excellent article that explains how to fix most issues that Debian has on PowerPC. For example, to fix the font rendering troubles, create an xorg.conf file (switch to a text terminal, run init 3, Xorg -configure, cp /root/xorg.conf.new /etc/X11/xorg.conf) and insert Option "RenderAccel" "false" into its Device section.

Migrating JIRA to osTicket

We were running JIRA as a ticket system for tech support and as a knowledge base. While this works, it’s not ideal, so we’ve decided to switch to osTicket. Of course, we wanted to migrate all issues, comments, attachments and issue links. JIRA has a good REST API, and osTicket has a rather simple database schema, so I’ve created a PHP script to do the migration, which you can obtain at https://gist.github.com/mkuron/3e1c92f9c4e993c65857.

This script is not meant as a black-box solution. It can only get you started in creating your own migration script, as every JIRA setup is different. Before you start, create all staff accounts in osTicket and make sure they have the same user name as in JIRA. Then, set up OAuth for JIRA as explained here: Two-legged OAuth between PHP and JIRA. Add your MySQL password and the mail domain you use to identify staff accounts to the scripts (if you don’t have a common mail domain for staff accounts, you’ll need to edit the logic to identify staff accounts based on group membership). Specify your tech support project key and give the map of knowledge base project keys to osTicket KB category IDs. Run the scripts on the command-line and copy the resulting attachments folder to your osTicket server (note that you’ll need the Attachments on Filesystem plugin and configure it to use the attachments folder).

The scripts are rerun-safe and transactional. If you want, you can remove the jira_id columns from the ost_faq, ost_ticket_thread and ost_file tables after you finish the migration.

To ensure that links continue working as much as possible, add the following redirects to your web server config:

# Redirect JIRA issues
RedirectMatch permanent ^/jira/browse/(TECH-[0-9]+) /osticket/jira-redir.php?key=$1
# Redirect all other JIRA pages
RedirectMatch permanent ^/jira/ /osticket
# prevent access to attachments folder
RedirectMatch ^/osticket/attachments/ /osticket/file.php

Put the following file into your osTicket folder and call it jira-redir.php:

<?php
require_once 'bootstrap.php';
require_once INCLUDE_DIR . 'ost-config.php';
if (substr(DBHOST,0,1) == ':')
 $mysqli = new mysqli("localhost", DBUSER, DBPASS, DBNAME, 3306, substr(DBHOST, 1));
else
 $mysqli = new mysqli(DBHOST, DBUSER, DBPASS, DBNAME);

$stmt = $mysqli->prepare('SELECT ticket_id FROM ' . TABLE_PREFIX . 'ticket WHERE number=?');
$stmt->bind_param("s", $_GET['key']);
$stmt->execute();
$stmt->bind_result($ticket_id);
$stmt->fetch();
$stmt->close();

$url = 'http://';
if ($_SERVER['HTTPS'])
 $url = 'https://';
$url .= $_SERVER['HTTP_HOST'];
$url .= dirname($_SERVER['SCRIPT_NAME']);
$url .= '/scp/tickets.php?id=';
$url .= $ticket_id;
header("Location: " . $url, true, 301);
?>

Digest authentication freezes Apache on OS X Server 5.0

When running a web site on OS X Server 5.0 for a while (on OS X 10.10.5 in my case), eventually you’ll notice hundreds of httpd processes in Activity Monitor. One or two might cause a bit of CPU load, while the others don’t do anything. When you try to load the web page, it is insanely slow and often throws a HTTP 500 error, a HTTP 502 Proxy Error, or the connection just times out. /var/log/apache2/error_log reports errors like

[Thu Nov 05 13:15:24.435549 2015] [mpm_prefork:error] [pid 60920] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting.

but that’s the only hint you get. To find out more, add the following lines inside the VirtualHost section of /Library/Server/Web/Config/apache2/sites/0000_127.0.0.1_34580_.conf and restart the Websites service in Server.app:

<Location /server-status>
SetHandler server-status
</Location>

Now, got to http://localhost/server-status and refresh it occasionally while traffic hits your web site. You’ll eventually see dozens of lines like the one below (starting with 52-0 in my case):

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv PID Acc M   CPU SS  Req Conn    Child   Slot    Client  VHost   Request

52-0    80825   0/2/2   C   0.45    51  0   0.0 0.02    0.02    ::1

All httpd processes you’re seeing in Activity Monitor are stuck in “Closing connection”, except for those that cause considerable CPU load. If the server were behaving correctly, you wouldn’t have as many processes and those that aren’t currently handling requests would either be “Waiting for Connection” or “Open slot with no current process”.

Let’s fire up a debugger to see what’s causing the processes to get stuck:

$ sudo lldb -p 80825
(lldb) process attach --pid 80825
Process 80825 stopped
* thread #1: tid = 0x3a69d4, 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10
libsystem_kernel.dylib`__wait4:
->  0x7fff8fe35902 <+10>: jae    0x7fff8fe3590c            ; <+20>
    0x7fff8fe35904 <+12>: movq   %rax, %rdi
    0x7fff8fe35907 <+15>: jmp    0x7fff8fe30c78            ; cerror
    0x7fff8fe3590c <+20>: retq   

Executable module set to "/usr/sbin/httpd".
Architecture set to: x86_64-apple-macosx.
(lldb) bt
* thread #1: tid = 0x3a69d4, 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10
    frame #1: 0x0000000109b3ee95 libapr-1.0.dylib`apr_proc_wait + 70
    frame #2: 0x000000010aa4290c mod_auth_digest_apple.so`cleanup_server_event + 73
    frame #3: 0x0000000109b3627a libapr-1.0.dylib`apr_pool_destroy + 82
    frame #4: 0x0000000109a74ce2 httpd`clean_child_exit + 50
    frame #5: 0x0000000109a74c59 httpd`child_main + 2393
    frame #6: 0x0000000109a73b2e httpd`make_child + 510
    frame #7: 0x0000000109a74181 httpd`perform_idle_server_maintenance + 1265
    frame #8: 0x0000000109a72887 httpd`prefork_run + 2471
    frame #9: 0x0000000109a26328 httpd`ap_run_mpm + 120
    frame #10: 0x0000000109a1185f httpd`main + 4687
    frame #11: 0x00007fff8d7435c9 libdyld.dylib`start + 1
(lldb) continue
Process 82070 resuming
(lldb) exit

Ah, so mod_auth_digest_apple.so is the culprit. So in all *.conf files in /Library/Server/Web/Config/apache2 and its subdirectories, replace every occurrence of AuthType Digest with AuthType Basic and comment out all lines containing mod_auth_digest_apple.so by prepending a # character. Restart the Websites service in Server.app. Watch http://localhost/server-status: everything should be fine now and no more connections stuck on “Closing connection”!

Printer Sharing randomly stops working due to memorystatus_thread killing cupsd

Printers shared in OS X 10.9, 10.10 or 10.11 randomly stop being accessible from remote computers. The system log reports that the cupsd process was terminated by memorystatus_thread:

Oct 26 07:14:33 robert kernel[0]: memorystatus_thread: idle exiting pid 4901 [cupsd]

This can also manually be triggered by executing sudo launchctl stop org.cups.cupsd.plist and manually and temporarily fixed by executing sudo launchctl start org.cups.cupsd.plist.

As a workaround, you can edit the CUPS LaunchDaemon to restart whenever a remote computer attempts to connect. On 10.9 or 10.10:

sudo /usr/libexec/PlistBuddy -c "Delete Sockets:Listeners:0:SockNodeName" /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo /usr/libexec/PlistBuddy -c "Delete Sockets:Listeners:1:SockNodeName" /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo launchctl unload /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo launchctl load /System/Library/LaunchDaemons/org.cups.cupsd.plist

On 10.11, first disable System Integrity Protection in Recovery mode, then run

sudo /usr/libexec/PlistBuddy -c "Add Sockets:Listeners:1 Dict" /System/Library/LaunchDaemons/org.cups.cupsd.plist/org.cups.cupsd.plist
sudo /usr/libexec/PlistBuddy -c "Add Sockets:Listeners:1:SockServiceName String" /System/Library/LaunchDaemons/org.cups.cupsd.plist/org.cups.cupsd.plist
sudo /usr/libexec/PlistBuddy -c "Set Sockets:Listeners:1:SockServiceName ipp "/System/Library/LaunchDaemons/org.cups.cupsd.plist/org.cups.cupsd.plist
sudo launchctl unload /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo launchctl load /System/Library/LaunchDaemons/org.cups.cupsd.plist