Category Archives: Mac OS X Server

Disabling “secured” IPv6 addresses is macOS 10.12 Sierra

On older macOS versions, every network interface would have one IPv6 address autogenerated from its MAC address, easily identified by the characteristic “ff:fe” bytes in the middle of the host part:
$ ifconfig en0
[...]
ether 10:dd:b1:9f:6b:ba
inet6 fe80::12dd:b1ff:fe9f:6bba%en0 prefixlen 64 scopeid 0x4
inet6 2001:7c0:2012:4a:12dd:b1ff:fe9f:6bba prefixlen 64 autoconf
[...]

Since macOS 10.12 however, these were replaced with randomly-generated “secured” addresses:
$ ifconfig en0
[...]
ether 10:dd:b1:9b:d0:67
inet6 fe80::46:3b36:146:9857%en0 prefixlen 64 secured scopeid 0x4
inet6 2001:7c0:2012:4a:4e6:f1d1:dd90:c6b4 prefixlen 64 autoconf secured
[...]

Very little is known about these, besides a single mailing list post that discovered them. If you are running a server, you’ll want your IPv6 address to be deterministic so you can register it in DNS. Therefore, we need to revert to pre-10.12 behavior:

$ echo net.inet6.send.opmode=0 >> /etc/sysctl.conf
$ reboot

If you look at the source code of the XNU kernel (Search for the IN6_IFF_SECURED flag) and the IPConfiguration service in macOS 10.11 (the 10.12 source code hasn’t been released yet), you can see that the new behavior was already there, just not enabled by default like it is now. Also, we now know that the change wasn’t made to reflect RFC 7217 (Semantically Opaque Interface Identifiers) behavior, but rather implements RFC 3972 (Cryptographically Generated Addresses).

Digest authentication freezes Apache on OS X Server 5.0

When running a web site on OS X Server 5.0 for a while (on OS X 10.10.5 in my case), eventually you’ll notice hundreds of httpd processes in Activity Monitor. One or two might cause a bit of CPU load, while the others don’t do anything. When you try to load the web page, it is insanely slow and often throws a HTTP 500 error, a HTTP 502 Proxy Error, or the connection just times out. /var/log/apache2/error_log reports errors like

[Thu Nov 05 13:15:24.435549 2015] [mpm_prefork:error] [pid 60920] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting.

but that’s the only hint you get. To find out more, add the following lines inside the VirtualHost section of /Library/Server/Web/Config/apache2/sites/0000_127.0.0.1_34580_.conf and restart the Websites service in Server.app:

<Location /server-status>
SetHandler server-status
</Location>

Now, got to http://localhost/server-status and refresh it occasionally while traffic hits your web site. You’ll eventually see dozens of lines like the one below (starting with 52-0 in my case):

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv PID Acc M   CPU SS  Req Conn    Child   Slot    Client  VHost   Request

52-0    80825   0/2/2   C   0.45    51  0   0.0 0.02    0.02    ::1

All httpd processes you’re seeing in Activity Monitor are stuck in “Closing connection”, except for those that cause considerable CPU load. If the server were behaving correctly, you wouldn’t have as many processes and those that aren’t currently handling requests would either be “Waiting for Connection” or “Open slot with no current process”.

Let’s fire up a debugger to see what’s causing the processes to get stuck:

$ sudo lldb -p 80825
(lldb) process attach --pid 80825
Process 80825 stopped
* thread #1: tid = 0x3a69d4, 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10
libsystem_kernel.dylib`__wait4:
->  0x7fff8fe35902 <+10>: jae    0x7fff8fe3590c            ; <+20>
    0x7fff8fe35904 <+12>: movq   %rax, %rdi
    0x7fff8fe35907 <+15>: jmp    0x7fff8fe30c78            ; cerror
    0x7fff8fe3590c <+20>: retq   

Executable module set to "/usr/sbin/httpd".
Architecture set to: x86_64-apple-macosx.
(lldb) bt
* thread #1: tid = 0x3a69d4, 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10
    frame #1: 0x0000000109b3ee95 libapr-1.0.dylib`apr_proc_wait + 70
    frame #2: 0x000000010aa4290c mod_auth_digest_apple.so`cleanup_server_event + 73
    frame #3: 0x0000000109b3627a libapr-1.0.dylib`apr_pool_destroy + 82
    frame #4: 0x0000000109a74ce2 httpd`clean_child_exit + 50
    frame #5: 0x0000000109a74c59 httpd`child_main + 2393
    frame #6: 0x0000000109a73b2e httpd`make_child + 510
    frame #7: 0x0000000109a74181 httpd`perform_idle_server_maintenance + 1265
    frame #8: 0x0000000109a72887 httpd`prefork_run + 2471
    frame #9: 0x0000000109a26328 httpd`ap_run_mpm + 120
    frame #10: 0x0000000109a1185f httpd`main + 4687
    frame #11: 0x00007fff8d7435c9 libdyld.dylib`start + 1
(lldb) continue
Process 82070 resuming
(lldb) exit

Ah, so mod_auth_digest_apple.so is the culprit. So in all *.conf files in /Library/Server/Web/Config/apache2 and its subdirectories, replace every occurrence of AuthType Digest with AuthType Basic and comment out all lines containing mod_auth_digest_apple.so by prepending a # character. Restart the Websites service in Server.app. Watch http://localhost/server-status: everything should be fine now and no more connections stuck on “Closing connection”!

Printer Sharing randomly stops working due to memorystatus_thread killing cupsd

Printers shared in OS X 10.9, 10.10 or 10.11 randomly stop being accessible from remote computers. The system log reports that the cupsd process was terminated by memorystatus_thread:

Oct 26 07:14:33 robert kernel[0]: memorystatus_thread: idle exiting pid 4901 [cupsd]

This can also manually be triggered by executing sudo launchctl stop org.cups.cupsd.plist and manually and temporarily fixed by executing sudo launchctl start org.cups.cupsd.plist.

As a workaround, you can edit the CUPS LaunchDaemon to restart whenever a remote computer attempts to connect. On 10.9 or 10.10:

sudo /usr/libexec/PlistBuddy -c "Delete Sockets:Listeners:0:SockNodeName" /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo /usr/libexec/PlistBuddy -c "Delete Sockets:Listeners:1:SockNodeName" /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo launchctl unload /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo launchctl load /System/Library/LaunchDaemons/org.cups.cupsd.plist

On 10.11, first disable System Integrity Protection in Recovery mode, then run

sudo /usr/libexec/PlistBuddy -c "Add Sockets:Listeners:1 Dict" /System/Library/LaunchDaemons/org.cups.cupsd.plist/org.cups.cupsd.plist
sudo /usr/libexec/PlistBuddy -c "Add Sockets:Listeners:1:SockServiceName String" /System/Library/LaunchDaemons/org.cups.cupsd.plist/org.cups.cupsd.plist
sudo /usr/libexec/PlistBuddy -c "Set Sockets:Listeners:1:SockServiceName ipp "/System/Library/LaunchDaemons/org.cups.cupsd.plist/org.cups.cupsd.plist
sudo launchctl unload /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo launchctl load /System/Library/LaunchDaemons/org.cups.cupsd.plist

Fixing OS X Server Push Mail

OS X Server 10.7 and later support push mail for iOS devices. This mechanism is neither based on IMAP IDLE (which iOS doesn’t support) nor Exchange ActiveSync (EAS), but on Apple’s Push Notification Service (APNS) infrastructure.

After setting up Mail using the GUI in OS X Server 10.10 Yosemite, I wondered why push didn’t work. From my understanding, it should happen automatically. The only indications something was wrong were the following lines in /Library/Logs/Mail/push_notify.log:

Feb 21 20:13:27 server.example.com push_notify[22848]: ApplePushServiceProvider: Warning: no device map found for 3F2504E0-4F89-41D3-9A0C-0305E82C3301

as well as XAPPLEPUSHSERVICE missing from the IMAP capabilities list:

$ openssl s_client -quiet -connect localhost:993
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot ready.

This is often the point where you have to break out the disassembler to find out what is wrong. Luckily however, Dovecot is open source, including the modifications Apple made to support APNS. Tracing through the code, the message above is logged if /Library/Server/Mail/Data/mta/guid_device_maps.plist does not contain a section for the user to which the incoming email is addressed. This section is written when Dovecot receives an XAPPLEPUSHSERVICE command. This command is probably only sent by a client when the XAPPLEPUSHSERVICE capability is reported by the server. The reason why the server didn’t report the capability was a simple incorrect (default) setting, easily fixable using

sudo serveradmin settings mail:imap:aps_topic_enabled = yes

Push mail immediately started working for me after this command, and the capability is correctly reported:

$ openssl s_client -quiet -connect localhost:993
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE XAPPLEPUSHSERVICE AUTH=PLAIN AUTH=LOGIN] Dovecot ready.

PHP 5: ldap_search never returns when searching Active Directory

I recently moved a PHP web application from a server running PHP 5.3 on Mac OS X 10.6 to a newer one with PHP 5.4 on Mac OS X 10.9. This caused the following code sample, run against an Active Directory server, to hang at the ldap_search() call:

$conn = ldap_connect('ldaps://' . $LDAPSERVER);
ldap_set_option($conn, LDAP_OPT_PROTOCOL_VERSION, 3);
$bind = @ldap_bind($conn, $LDAPUSER, $LDAPPW);
$result = ldap_search($conn, $LDAPSEARCHBASE, '(&(samaccountname=' . $searchuser . '))');
$info = ldap_get_entries($conn, $result);
ldap_close($conn);

Wiresharking the connection between web server and LDAP server (after replacing ldaps:// with ldap://) showed:

bindRequest(1) "$LDAPUSER" simplebindResponse(1) success searchRequest82) "$LDAPSEARCHBASE" wholeSubtree
searchResEntry(2) "CN=$searchuser,...,$LDAPSEARCHBASE" | searchResRef(2) | searchResDone(2) success [1 result]
bindRequest(4) "" simple
bindResponse(4) success
searchRequest(3) "DC=DomainDnsZones,$LDAPSEARCHBASE" wholeSubtree
searchResDone(3) operationsError (000004DC: LdapErr: DSID-0C0906E8, comment: In order to perform this operation a successful bind must be complete on the connection., data0,

So it’s binding, receiving a success response, searching and then receiving a response and a referrer to DC=DomainDnsZones,$LDAPSEARCHBASE. Next, it opens a new TCP connection and follows the referrer, but does an anonymous bind.

The solution is simple: just add

ldap_set_option($conn, LDAP_OPT_REFERRALS, FALSE);

after line 2. If for some reason you actually need to follow the referrer, have a look at ldap_set_rebind_proc, which lets you specify a callback which then does the authentication upon rebind.

Update August 2015: Same goes when using Net_LDAP3, which is used e.g. by Roundcube’s LDAP integration. Here you need to add the following:

$config['ldap_public']['public'] = array(
[...]
 'referrals' => false,
);

HP StorageWorks P2000 G3

Hardware

To replace a 2006 Xserve and a 7TB Xserve RAID at the university, we recently got a Mac mini server, an ATTO ThunderLink FC 1082 Thunderbolt to 8Gbit Fibre Channel adapter, and a HP StorageWorks P2000 G3 MSA FC Dual Controller LFF (specifically, model number AP845B).

The P2000 is not explicitly on ATTO’s compatibility matrix, but when I asked their tech support about it, they said it was compatible and provided me with a pre-release version of their Multi Path Director driver for the Thunderlink which is officially compatible.

Evidently, the P2000 G3 is an OEM’d version of the Dot Hill AssuredSAN 3000 Series (specifically, the 3730), which is on ATTO’s compatibility list, so I assume the standard driver would work just as well. Update 2018: Since Dot Hill has in the meantime been sold, their support page has moved to Seagate.

We chose the Thunderlink/P2000 combo over a Promise solution because it was cheaper, fully 8Gbit capable and had four host ports. Also, I know that HP’s tech support is good and they’ll have spare parts around for many years. Plus, the P2000 is VMWare ESXi certified.

The obvious downside to the P2000 is that the disk bays do not have standard SAS connectors but require an interposer board to convert to a SCA-2/SCA-40 connector. The included slot blinds are in fact blinds and cannot be used to mount an actual drive. You can get empty caddies/trays for the P2000 on eBay or from some used SAN equipment dealer for around 100 euros, or buy your hard drives from HP for a premium of around 100-150 euros over the plain drives. (The interposer board itself appears to get sold under the model numbers 371595-001 or 60-272-02 on eBay, but I haven’t found a model number for the caddy frame yet.) If you’re buying plain drives, you can check HP’s hard drive model matrix to see what model of drive an HP part number corresponds to. For example, the 3TB SAS drive QK703A is a Seagate Constellation ES.2 ST33000650SS and the 2TB SAS drive AW555A, which we ordered, is a Seagate Constellation ES ST2000NM0001).

Firmware

I have verified that the firmwares are interchangeable between the AssuredSAN 3000 and the P2000 G3: I downloaded and extracted the TS250R023 from both Dot Hill and HP and both contain a file named TS250R023.bin with an MD5 sum of 7b267cc4178aef53f7d3487e356f8435. I assume that’s the file that can be uploaded through the web interface.

To extract the HP firmware, download the Linux updater (e.g. CP020030.scexe) and use a hex editor to find the offset of the line break after the end of the shell script at the beginning, then use dd to skip the plain text: dd if=CP020030.scexe bs=1 skip=8602 of=scexe_tmp24664.tar.gz. Now you can tar zxf scexe_tmp24664.tar.gz and pull out the TS250R023.bin.

To extract the TS250R023.bin, simply tar xf TS250R023.bin. If you want to poke around the root filesystem of the Management Controller, unsquashfs mc/components/app.squashfs. You may need to compile squashfs-tools yourself to get LZMA support (edit squashfs-tools/Makefile, set LZMA_SUPPORT=1 or LZMA_XZ_SUPPORT=1 and apt-get install liblzma-dev zlib1g-dev liblz-dev).

Setup and configuration

After unpacking the device, I first updated the firmware to the most recent version available from HP. Before you do that (I used the Windows utility), make sure to set static IP addresses or DHCP static mappings (otherwise the update might fail due to changing addresses). After you set the password for the manage user, you’ll need to SSH into the device to change the password on a hidden admin account about which HP issued a security advisory back in December 2010 (but still hasn’t fixed it in the firmware).

I created a RAID5 out of 4x 2TB drives and dedicated a fifth one as a global spare. In the global disk settings, I enabled spindown so the spare would not be running unnecessarily. The RAID initialization took close to two days, but as that runs in the background, you can already start using it.

Then I created a couple volumes (setting the default mapping to not mapped) and mapped two of them to our Mac mini server (on the Thunderlink) and a third to our two VMWare ESXi servers (on Qlogic QLE2460 HBAs). This was much easier to do than on our old Xserve RAID and I love that I can start out with smaller volumes (sized appropriately that they’ll last for the next year) and expand them later on. The P2000 does not do thin provisioning, but you can’t really expect that at this price point.

 

Xserve RAID and Atto Thunderlink FC 1082 are incompatible if used without an FC switch

We’re running a 2006 Xserve RAID at the university. Our old server was a 2006 Xserve with an Apple 2 Gbit Fibre Channel card. When we recently got a new Mac mini server to replace, we ordered an Atto Thunderlink FC 1082 to interface with the RAID. The Promise SANLink would have been a possible alternative, but the Thunderlink is capable of 8 Gbit/s, thus future-proofing our investment.

Unfortunately, when I hooked up the Thunderlink straight to the Xserve RAID using an Apple Fibre Channel Copper Cable, neither the Xserve RAID Admin utility nor the Mac mini showed a connection. After some googling around, it appears as if the Xserve RAID is not capable of negotiating links with HBAs that are capable of more than 2 Gbit/s. Turns out also says that you shouldn’t use their 4 Gbit card with the Xserve RAID: HT1769.

Since the RAID has been working fine for quite a while with two HP servers running VMWare ESXi with Qlogic QLE2460 controllers connected through a Qlogic SANbox 5200 2 Gbit FC switch, and I knew the Thunderlink worked with that switch, I simply used an FC Copper Cable between the Thunderlink and the switch and one between the switch and the RAID, configured the zoning, et voilà, the array mounted on the Mac mini.

Installing Apple Remote Desktop 3.6.1 without first installing the boxed version

Apple Remote Desktop 3.0 was released in 2006. The current version is 3.6.1, and as such your six year old boxed copy license still entitles you to run it.

ARD has undergone some major changes during that time: most notably, Apple switched it from a Postgres database to an SQLite database, which greatly reduced its memory footprint and made it much more self-contained.

After reinstalling Mac OS X on my computer, I didn’t want to first install my boxed copy (version 3.2) because I didn’t want it to bloat my system with a copy of Postgresql that would get replaced by the update anyway.

After six years, it’s a safe bet that the downloadable Admin Update 3.6.1 does not require any previous version’s files. However, the installer checks for the existence of a previous version before allowing you to select the destination drive. This check can be satisfied by first running

defaults write /Applications/Remote\ Desktop.app/Contents/Info CFBundleShortVersionString -float 3.0

I’d like to point out that it still requires the license key that came in the box when you first run ARD.

Troubleshooting Apple Software Update Server

We are currently in the process of migrating towards a thin imaging approach for the Macs I manage at my university. One of the things we needed was an Apple Software Update Server, which is pretty straight forward to set up using Server Admin. I used Snow Leopard Server 10.6.8 with Lion updates enabled as described in Apple’s KB article.

A command I found extremely useful:

sudo serveradmin settings swupdate | grep "enable = no" | awk -F '=' '{print $1"= yes"}' | sudo serveradmin settings

This command enables all available updates. If you have set SUS to automatically mirror, but not automatically enable, this command saves you from having to click on every single one of the 600 updates to enable it.

A few days after I set up my SUS I was starting to see a bunch of error messages in /var/log/swupd/swupd_syncd_log after starting a sync run using the refresh button below the updates list in Server Admin. I’ll go over them here and explain my fixes:

“Product file URL contains possible security violation.” in log

*** Product file URL contains possible security violation.
*** Product ID: "11D2515_ServerEssentials"; file URL: "http://swcdn.apple.com/content/downloads/10/59/11D2515_ServerEssentials/xajda1v3ycqbtv75fiw5hvosaovu9to9hc/ServerEssentials.dst/041-5774.Italian.dist"
*** Reason: file download path cannot be reached / does not exist.
*** The suspect product file will not be downloaded.

A few Lion updates contain subfolders, which SUS does not create. Simply create those subfolders (mkdir /var/db/swupd/content/downloads/10/59/11D2515_ServerEssentials/xajda1v3ycqbtv75fiw5hvosaovu9to9hc/ServerEssentials.dst; chown _softwareupdate /var/db/swupd/content/downloads/10/59/11D2515_ServerEssentials/xajda1v3ycqbtv75fiw5hvosaovu9to9hc/ServerEssentials.dst) and have SUS recheck for available updates.

[EDIT: The original version of this blog post used curl to download the file, but forgot to chown the folder. Letting SUS download the file itself is the cleaner solution.]

[EDIT 2: This issue is really widespread with the ARD Client 3.6 update, where http://swcdn.apple.com/content/downloads/31/58/041-5433/xt9k9paj5zu8rx258rdccohk236ee77clh/RemoteDesktopClient.dst/041-5433.*.dist fails. This does solve this issue.]

“Product XXX-YYYY is no longer available and has no replacement” in log or “*** Missing version string for product XXX-YYYY” in log or “_productId Update” showing up in Server Admin

Your catalog is corrupted. Delete it to have it rebuilt automatically. The following commands help:

sudo serveradmin stop swupdate
cd $(sudo serveradmin settings swupdate:updatesDocRoot | awk -F '"' '{print $2}')/html/content
mv catalogs catalogs.old
sudo serveradmin start swupdate

Create a bootable Mac OS X Lion Server USB drive

  1. Download Lion from the Mac App Store.
  2. Use Disk Utility to restore /Applications/Install Mac OS X Lion.app/Contents/SharedSupport to a USB flash drive.
  3. Download Lion Server from the Mac App Store.
  4. Grab the correct version of ServerEssentials.pkg by running curl “http://swscan.apple.com/content/catalogs/others/index-lion-snowleopard-leopard.merged-1.sucatalog” | grep ServerEssentials in a Terminal, searching for the line that corresponds to your version of Lion (it can be identified by looking at ./System/Library/CoreServices/SystemVersion.plist on the flash drive), and downloading the file from the link referenced by that line (it will look like http://swcdn.apple.com/content/downloads/21/09/11E53_ServerEssentials/yfLkxRYr8GczRPHKBSpStgMYT3kjRnKQRf/ServerEssentials.pkg).
  5. Drop the downloaded ServerEssentials.pkg into ./Packages on the flash drive.
  6. If after installing from the Flash drive, you are missing /Applications/Server.app, that can easily be extracted from the Lion Server Combo Updater (use the version that corresponds to your Lion version) using Pacifist.

More details here.