Digest authentication freezes Apache on OS X Server 5.0

When running a web site on OS X Server 5.0 for a while (on OS X 10.10.5 in my case), eventually you’ll notice hundreds of httpd processes in Activity Monitor. One or two might cause a bit of CPU load, while the others don’t do anything. When you try to load the web page, it is insanely slow and often throws a HTTP 500 error, a HTTP 502 Proxy Error, or the connection just times out. /var/log/apache2/error_log reports errors like

[Thu Nov 05 13:15:24.435549 2015] [mpm_prefork:error] [pid 60920] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting.

but that’s the only hint you get. To find out more, add the following lines inside the VirtualHost section of /Library/Server/Web/Config/apache2/sites/0000_127.0.0.1_34580_.conf and restart the Websites service in Server.app:

<Location /server-status>
SetHandler server-status

Now, got to http://localhost/server-status and refresh it occasionally while traffic hits your web site. You’ll eventually see dozens of lines like the one below (starting with 52-0 in my case):

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv PID Acc M   CPU SS  Req Conn    Child   Slot    Client  VHost   Request

52-0    80825   0/2/2   C   0.45    51  0   0.0 0.02    0.02    ::1

All httpd processes you’re seeing in Activity Monitor are stuck in “Closing connection”, except for those that cause considerable CPU load. If the server were behaving correctly, you wouldn’t have as many processes and those that aren’t currently handling requests would either be “Waiting for Connection” or “Open slot with no current process”.

Let’s fire up a debugger to see what’s causing the processes to get stuck:

$ sudo lldb -p 80825
(lldb) process attach --pid 80825
Process 80825 stopped
* thread #1: tid = 0x3a69d4, 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10
->  0x7fff8fe35902 <+10>: jae    0x7fff8fe3590c            ; <+20>
    0x7fff8fe35904 <+12>: movq   %rax, %rdi
    0x7fff8fe35907 <+15>: jmp    0x7fff8fe30c78            ; cerror
    0x7fff8fe3590c <+20>: retq   

Executable module set to "/usr/sbin/httpd".
Architecture set to: x86_64-apple-macosx.
(lldb) bt
* thread #1: tid = 0x3a69d4, 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10
    frame #1: 0x0000000109b3ee95 libapr-1.0.dylib`apr_proc_wait + 70
    frame #2: 0x000000010aa4290c mod_auth_digest_apple.so`cleanup_server_event + 73
    frame #3: 0x0000000109b3627a libapr-1.0.dylib`apr_pool_destroy + 82
    frame #4: 0x0000000109a74ce2 httpd`clean_child_exit + 50
    frame #5: 0x0000000109a74c59 httpd`child_main + 2393
    frame #6: 0x0000000109a73b2e httpd`make_child + 510
    frame #7: 0x0000000109a74181 httpd`perform_idle_server_maintenance + 1265
    frame #8: 0x0000000109a72887 httpd`prefork_run + 2471
    frame #9: 0x0000000109a26328 httpd`ap_run_mpm + 120
    frame #10: 0x0000000109a1185f httpd`main + 4687
    frame #11: 0x00007fff8d7435c9 libdyld.dylib`start + 1
(lldb) continue
Process 82070 resuming
(lldb) exit

Ah, so mod_auth_digest_apple.so is the culprit. So in all /Library/Server/Web/Config, replace every occurrence of AuthType Digest with AuthType Basic and comment out all lines containing mod_auth_digest_apple.so by prepending a # character. Restart the Websites service in Server.app. Watch http://localhost/server-status: everything should be fine no and no more connections stuck on “Closing connection”!

Printer Sharing randomly stops working due to memorystatus_thread killing cupsd

Printers shared in OS X 10.9, 10.10 or 10.11 randomly stop being accessible from remote computers. The system log reports that the cupsd process was terminated by memorystatus_thread:

Oct 26 07:14:33 robert kernel[0]: memorystatus_thread: idle exiting pid 4901 [cupsd]

This can also manually be triggered by executing sudo launchctl stop org.cups.cupsd.plist and manually and temporarily fixed by executing sudo launchctl start org.cups.cupsd.plist.

As a workaround, you can edit the CUPS LaunchDaemon to restart whenever a remote computer attempts to connect. On 10.9 or 10.10:

sudo /usr/libexec/PlistBuddy -c "Delete Sockets:Listeners:0:SockNodeName" /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo /usr/libexec/PlistBuddy -c "Delete Sockets:Listeners:1:SockNodeName" /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo launchctl unload /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo launchctl load /System/Library/LaunchDaemons/org.cups.cupsd.plist

On 10.11, first disable System Integrity Protection in Recovery mode, then run

sudo /usr/libexec/PlistBuddy -c "Add Sockets:Listeners:1 Dict" /System/Library/LaunchDaemons/org.cups.cupsd.plist/org.cups.cupsd.plist
sudo /usr/libexec/PlistBuddy -c "Add Sockets:Listeners:1:SockServiceName String" /System/Library/LaunchDaemons/org.cups.cupsd.plist/org.cups.cupsd.plist
sudo /usr/libexec/PlistBuddy -c "Set Sockets:Listeners:1:SockServiceName ipp "/System/Library/LaunchDaemons/org.cups.cupsd.plist/org.cups.cupsd.plist
sudo launchctl unload /System/Library/LaunchDaemons/org.cups.cupsd.plist
sudo launchctl load /System/Library/LaunchDaemons/org.cups.cupsd.plist

ARP and multicast packets lost with OpenVPN in tap mode

After upgrading our OpenVPN server VM from Debian 7 to Debian 8 (moving us from OpenVPN 2.2 to OpenVPN 2.3 and Linux kernel 3.2 to Linux kernel 3.16) and upgrading our virtualization from VMware ESXi 5.5 to ESXi 6.0 and moving the VM to a different host, the VPN got really unreliable: the VPN connection itself worked fine, but any connections established across the VPN were very slow to get established. Once they were established, everything worked fine and you could even create new connections to the same host across the VPN and they would be established quickly.

I wasn’t sure which one of the many changes caused the issue, but luckily Wireshark quickly revealed the problem: As we are using OpenVPN in layer 2 mode (i.e. with tap interfaces), ARP packets are quite important. While I could see the ARP requests making it across the interface bridge from tap0 to eth0, I saw the ARP replies going into eth0 and not making it to tap0. The server-side fix is easy, just disable the MAC table on the bridge completely and simply lets all packets pass:

brctl setageing br0 0

Now that ARP was working, I noticed that VPN clients also did not get IPv6 addresses. Evidently, the ICMPv6 multicasts weren’t making it across the bridge either. To fix that, enable multicast snooping on the bridge:

echo 1 > /sys/devices/virtual/net/br0/bridge/multicast_querier

Fixing OpenMPI over InfiniBand on Rocks Cluster Linux

We recently got a new small compute cluster at the university, running Rocks Clusters Linux 6.1.1, a CentOS 6 derivative. The nodes are interconnected via an InfiniBand network. Unfortunately, the default configuration of OpenMPI 1.6.2 in the HPC roll wastes a significant amount of performance: it communicates using TCP, which is run over a load-balanced combination of IP over InfiniBand and IP over Ethernet.

Switching to DMA over InfiniBand is simple: just run the following command on all compute nodes and the head node:

sed -i 's/add rocks-openmpi/add rocks-openmpi_ib/g' /etc/profile.d/rocks-hpc.*sh

Now however, you get a message like this when you run an MPI job:

WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory.  This can cause MPI jobs to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered.  You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module


  Local host:              bee.icp.uni-stuttgart.de
  Registerable memory:     32768 MiB
  Total memory:            130967 MiB

Your MPI job will continue, but may be behave poorly and/or hang.

To fix that, run

echo "options mlx4_core log_num_mtt=24" >> /etc/modprobe.d/mlx4.conf

on all nodes and reboot. log_mtts_per_seg defaulted to 3 on our kernel and did not need tweaking. To check your current values, run

grep . /sys/module/mlx4_core/parameters/*mtt*

One warning message that still comes up when running an MPI job is the following:

WARNING: Failed to open "OpenIB-cma-1" [DAT_INVALID_ADDRESS:]. 
This may be a real error or it may be an invalid entry in the uDAPL
Registry which is contained in the dat.conf file. Contact your local
System Administrator to confirm the availability of the interfaces in
the dat.conf file.
bee.icp.uni-stuttgart.de:30104:  open_hca: getaddr_netdev ERROR: No such device. Is ib1 configured?
bee.icp.uni-stuttgart.de:30104:  open_hca: device mthca0 not found
bee.icp.uni-stuttgart.de:30104:  open_hca: device mthca0 not found
DAT: library load failure: libdaplscm.so.2: cannot open shared object file: No such file or directory
DAT: library load failure: libdaplscm.so.2: cannot open shared object file: No such file or directory

As UDAPL is removed in newer OpenMPI versions anyway, this is fixed by running

echo "btl = ^udapl" >> /opt/openmpi/etc/openmpi-mca-params.conf

on all compute nodes and the head node.

So all in all, you can simply add the following lines to /export/rocks/install/site-profiles/6.1.1/nodes/extend-compute.xml and rebuild your compute node image:

echo "btl = ^udapl" >> /opt/openmpi/etc/openmpi-mca-params.conf
sed -i 's/add rocks-openmpi/add rocks-openmpi_ib/g' /etc/profile.d/rocks-hpc.*sh
echo "options mlx4_core log_num_mtt=24" >> /etc/modprobe.d/mlx4.conf
dracut -f 2.6.32-504.16.2.el6.x86_64 # may need to rebuild the initrd so it picks up the modprobe parameters

Leserbrief “Vorratsdatenspeicherung”

Am 15. April 2015 kündigte Justizminister H. Maas an, die Vorratsdatenspeicherung wieder einführen zu wollen, die er selber jahrelang kritisiert hatte und die im Jahr 2010 vom Bundesverfassungsgericht und im Jahr 2014 vom Europäischen Gerichtshof verboten worden war. Am 27. April 2015 druckte die Süddeutsche Zeitung dazu einen von mir verfassten Leserbrief:

Elektronische Fußfessel für alle

Wie kann die Antwort auf die überbordende und ungesetzliche Überwachung durch westliche Geheimdienste wie NSA, BND und GCHQ sein, noch größere Datenmassen anzusammeln? In einem Rechtsstaat dürfen erst bei konkretem Verdacht Überwachungsmaßnahmen ergriffen werden, sonst besteht die Gefahr, dass die Bürger den Gebrauch ihrer Freiheiten einschränken, um bloß nicht als falsch-positiver Treffer bei einer Rasterfahndung aufzufallen.

Die Befürworter der Vorratsdatenspeicherung argumentieren unlauter: Es werden Einzelfälle verallgemeinert, um eine Notwendigkeit der Wiedereinführung zu belegen. Es werden seriöse Gutachten ignoriert, die der Vorratsdatenspeicherung keinen nennenswerten Nutzen für die Strafverfolgung attestieren. Es werden gar frei erfundene Beispiele genutzt, siehe Sigmar Gabriels Behauptung, die Vorratsdatenspeicherung habe bei der Aufklärung der Anschläge von Norwegen 2011 geholfen. Wenn die Vorratsdatenspeicherung aber gar nicht die Verfolgung schwerer Straftaten erleichtert, warum verlangt die Politik sie denn dann?

Wenn schon die Kommunikationsdaten nicht von Nutzen sind, liegt die Vermutung nahe, dass der Staat es auf die Mobilfunk-Standortdaten abgesehen hat: Fortwährende Speicherung der Aufenthaltsorte aller Mobiltelefone in Deutschland, quasi elektronische Fußfesseln für die gesamte Bevölkerung, ohne dass diese sie bemerkt – ein Traum für jeden Sicherheitspolitiker, ein Albtraum für die Väter und Mütter des Grundgesetzes.

Die nächsten beiden Jahre großer Koalition werden sehr schwer für die SPD: Sie hat alle ihre Wünsche aus dem Koalitionsvertrag bereits erfüllt bekommen, jetzt besteht die Union auf die Umsetzung ihrer eigenen Projekte. Vielleicht ist es an der Zeit für die SPD, die Koalition einfach aufzukündigen.

Michael Kuron, Frickenhausen

Scientific Article: Role of Geometrical Shape in Like-Charge Attraction of DNA

My first scientific article has been published. It is available at The European Physical Journal E.

Role of geometrical shape in like-charge attraction of DNA
Michael Kuron, Axel Arnold
Eur. Phys. J. E 38 (3) 20 (2015)
DOI: 10.1140/epje/i2015-15020-9

While the journal is not open-access, I am allowed to provide a PDF version of the author’s accepted manuscript for download on my own website below:


Multiple GPUs on unsupported Mac Pro

The first two generations of Apple’s Mac Pro, the MacPro1,1 and MacPro2,1, do not officially run OS X later than 10.7.5. However, there is a modified EFI bootloader available which emulates the EFI64 interface on EFI32 machines. The original version available at Google Code supports OS X 10.9, and there’s a newer one available at Github which also does OS X 10.10. You simply drop in a new boot.efi in two places and add your board ID to the list of supported systems.

The resulting system works perfectly fine, which makes me wonder why Apple didn’t come up with a solution like this themselves. In any case, it extends the life of 2006 and 2007 Mac Pros beyond last year’s end-of-support for OS X Lion.

Since OS X 10.7 and higher included graphics drivers that not only supported the official Apple-supplied GPUs with EFI-compatible firmwares, but pretty much any off-the-shelf Nvidia or AMD GPU, GPUs have become quite easy to upgrade in Mac Pros (the classic tower cheese grater Mac Pro, not the new black trash can Mac Pro). The only thing you lose is the boot screen, so you still need to keep around that original GPU to debug the machine if it doesn’t boot.

These upgraded GPUs work fine with the modified bootloader as well, however if you try to install multiple GPUs (e.g. if you want to drive more than two displays or develop CUDA code and would like to run it in the debugger), only one of them will actually output video.

The solution to make multiple GPUs work in Macs running OS X versions they don’t officially support is surprisingly simple:

sudo /usr/libexec/PlistBuddy -c "Add :IOKitPersonalities:AppleGraphicsDevicePolicy:ConfigMap:Mac-F4208DA9 string none" /System/Library/Extensions/AppleGraphicsControl.kext/Contents/PlugIns/AppleGraphicsDevicePolicy.kext/Contents/Info.plist
sudo /usr/libexec/PlistBuddy -c "Add :IOKitPersonalities:AppleGraphicsDevicePolicy:ConfigMap:Mac-F4208DC8 string none" /System/Library/Extensions/AppleGraphicsControl.kext/Contents/PlugIns/AppleGraphicsDevicePolicy.kext/Contents/Info.plist
sudo kextcache -update-volume /

So here we are, running a 2006 MacPro1,1 with two EVGA Nvidia GT610 1GB cards, driving three Apple Cinema Displays.

Fixing OS X Server Push Mail

OS X Server 10.7 and later support push mail for iOS devices. This mechanism is neither based on IMAP IDLE (which iOS doesn’t support) nor Exchange ActiveSync (EAS), but on Apple’s Push Notification Service (APNS) infrastructure.

After setting up Mail using the GUI in OS X Server 10.10 Yosemite, I wondered why push didn’t work. From my understanding, it should happen automatically. The only indications something was wrong were the following lines in /Library/Logs/Mail/push_notify.log:

Feb 21 20:13:27 server.example.com push_notify[22848]: ApplePushServiceProvider: Warning: no device map found for 3F2504E0-4F89-41D3-9A0C-0305E82C3301

as well as XAPPLEPUSHSERVICE missing from the IMAP capabilities list:

$ openssl s_client -quiet -connect localhost:993

This is often the point where you have to break out the disassembler to find out what is wrong. Luckily however, Dovecot is open source, including the modifications Apple made to support APNS. Tracing through the code, the message above is logged if /Library/Server/Mail/Data/mta/guid_device_maps.plist does not contain a section for the user to which the incoming email is addressed. This section is written when Dovecot receives an XAPPLEPUSHSERVICE command. This command is probably only sent by a client when the XAPPLEPUSHSERVICE capability is reported by the server. The reason why the server didn’t report the capability was a simple incorrect (default) setting, easily fixable using

sudo serveradmin settings mail:imap:aps_topic_enabled = yes

Push mail immediately started working for me after this command, and the capability is correctly reported:

$ openssl s_client -quiet -connect localhost:993

PHP 5: ldap_search never returns when searching Active Directory

I recently moved a PHP web application from a server running PHP 5.3 on Mac OS X 10.6 to a newer one with PHP 5.4 on Mac OS X 10.9. This caused the following code sample, run against an Active Directory server, to hang at the ldap_search() call:

$conn = ldap_connect('ldaps://' . $LDAPSERVER);
ldap_set_option($conn, LDAP_OPT_PROTOCOL_VERSION, 3);
$bind = @ldap_bind($conn, $LDAPUSER, $LDAPPW);
$result = ldap_search($conn, $LDAPSEARCHBASE, '(&(samaccountname=' . $searchuser . '))');
$info = ldap_get_entries($conn, $result);

Wiresharking the connection between web server and LDAP server (after replacing ldaps:// with ldap://) showed:

bindRequest(1) "$LDAPUSER" simplebindResponse(1) success searchRequest82) "$LDAPSEARCHBASE" wholeSubtree
searchResEntry(2) "CN=$searchuser,...,$LDAPSEARCHBASE" | searchResRef(2) | searchResDone(2) success [1 result]
bindRequest(4) "" simple
bindResponse(4) success
searchRequest(3) "DC=DomainDnsZones,$LDAPSEARCHBASE" wholeSubtree
searchResDone(3) operationsError (000004DC: LdapErr: DSID-0C0906E8, comment: In order to perform this operation a successful bind must be complete on the connection., data0,

So it’s binding, receiving a success response, searching and then receiving a response and a referrer to DC=DomainDnsZones,$LDAPSEARCHBASE. Next, it opens a new TCP connection and follows the referrer, but does an anonymous bind.

The solution is simple: just add

ldap_set_option($conn, LDAP_OPT_REFERRALS, FALSE);

after line 2. If for some reason you actually need to follow the referrer, have a look at ldap_set_rebind_proc, which lets you specify a callback which then does the authentication upon rebind.

Update August 2015: Same goes when using Net_LDAP3, which is used e.g. by Roundcube’s LDAP integration. Here you need to add the following:

$config['ldap_public']['public'] = array(
 'referrals' => false,

Crossflashing Dell PERC H200 to LSI 9211-8i

OEM version of the LSI SAS 9211-8i, such as the Dell H200, H310 or IBM M1015 are quite popular for use with FreeNAS. However, they need to be flashed with a regular LSI firmware to disable their RAID capabilities in order to passthrough the drives directly to the OS.

Here’s how I upgraded my Dell PERC H200, which came with Dell’s A10 firmware (equivalent to LSI SAS 2008 P07 firmware), to LSI P20 firmware. The newer version also has the advantage that drives larger than 2TB are supported. Also, re-flashing allowed me to not flash a boot ROM to the card, speeding up the boot process of my server as my boot disk is connected to the Intel AHCI controller on the mainboard.

Please note that this operation is not supported by Dell or LSI, may void your warranty and could potentially damage the controller. So proceed at your own risk.

Step 1: Downloading old firmware

  1. Download the firmware for the Dell 6Gbps SAS HBA (this is a variant of the H200 with 8 external ports instead of 8 internal ports) and extract 6GBPSAS.FW from SASHBA_Firmware_6GBPS-SAS-HBA_07.03.06.00_A10_ZPE.exe.
  2. Download the P07 firmware for the LSI SAS 9211-8i and extract the file 2118it.bin from 9211_8i_Package_For_P7_Firmware_BIOS_Upgrade_on_MSDOS_and_Windows.zip.
  3. Download the P05 UEFI flasher for the LSI SAS 9211-8i and extract the file sas2flash.efi from EFI_Installer_P5.zip.
  4. Place these three files into a directory named P07 on a FAT32-formatted USB flash drive.

Step 2: Downloading current firmware

  1. Download the current firmware for the LSI SAS 9211-8i and extract the file 2118it.bin from 9211_8i_Package_P20_IR_IT_Firmware_BIOS_for_MSDOS_Windows.
  2. Download the current UEFI flasher for the LSI SAS 9211-8i and extract the file sas2flash.efi from Installer_P20_for_UEFI.zip.
  3. Place these two files into a directory named P20 on the USB flash drive.

Step 3: Downloading UEFI shell

  1. Download an x86_64 UEFI shell. I had to use the v1 shell because my server would only show error messages (about failed assertions and files not found).
  2. Rename the shell to BOOTX64.efi and place it into a directory named BOOT inside a directory named EFI on the USB flash drive.

Step 4: Flashing

  1. UEFI boot your server off the flash drive
  2. Type map -b to find the flash drive
  3. Switch to it, e.g. by entering fs1:
  4. cd P07
  5. sas2flash.efi -listall should show you your controller
  6. sas2flash.efi -c 0 -list shows you details on the controller. Note down the SAS address in case something goes wrong and you need to reprogram the SAS address.
  7. Erase the old firmware and boot ROM: sas2flash.efi -o -e 6
  8. Write the Dell 6Gbps firmware: sas2flash.efi -o -f 6GBPSAS.FW
  9. Write the LSI P07 firmware: sas2flash.efi -o -f 2118it.bin
  10. cd ..\P20
  11. Write the LSI P20 firmware: sas2flash.efi -o -f 2118it.bin


At the end of every command (before I rebooted before step 10), I got the message “Failed Reconnecting the EFI Driver. (EFI Error: Not Found)”. It did not seem to affect anything.

Step 7 showed “Erasing Flash Region” and then after a while “ERROR: Erase Flash Operation Failed!”. I simply proceeded and the error did not appear to affect anything.

Step 8 looked like this:

Screen Shot 2014-11-29 at 10.02.09

and the controller details after the flash looked like this:

Screen Shot 2014-11-29 at 10.03.04

During step 9, I received the message “NVDATA Versions Compatible. NVDATA Product ID and Vendor ID do not match. Would you like to flash anyway [y/n]?”, where I simply hit y and it proceeded flashing. At the end, it said “Firmware Flash Successful! Resetting Adapter… Adapter Reset Failed!”. Looking at the controller details showed lots of errors:

Screen Shot 2014-11-29 at 10.06.34So at this point, I rebooted the machine. Now the details looked all right:

Screen Shot 2014-11-29 at 10.09.54

Step 11 worked without a hitch and afterwards the controller details looked like this:

Screen Shot 2014-11-29 at 10.11.38


Booting up the server, I now had a Dell H200 that behaved exactly like a LSI SAS 9211-8i. The only difference was that it still reported its name and PCI ID (1028:1f1c) as a Dell 6Gbps SAS Card. FreeNAS didn’t care about that though.

Addendum: Matching firmware and driver versions

I was using this controller with FreeNAS and kept on getting kernel messages like

Nov 30 19:49:55 file02 kernel: (da1:mps0:0:0:0): READ(10). CDB: 28 00 a1 ea 00 58 00 01 00 00 length 131072 SMID 609 terminated ioc 804b scsi 0 state 0 xfer 0
Nov 30 19:49:55 file02 kernel: (da1:mps0:0:0:0): READ(10). CDB: 28 00 a1 e9 ff 58 00 01 00 00 
Nov 30 19:49:55 file02 kernel: (da1:mps0:0:0:0): CAM status: SCSI Status Error
Nov 30 19:49:55 file02 kernel: (da1:mps0:0:0:0): SCSI status: Check Condition
Nov 30 19:49:55 file02 kernel: (da1:mps0:0:0:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Nov 30 19:49:55 file02 kernel: (da1:mps0:0:0:0): Retrying command (per sense data)

It turns out that these are due to a mismatch between firmware and driver version. FreeBSD 9 ships with driver version 16 and FreeBSD 10 includes version 19. Linux currently has version 18. So make sure that in step 11, you always flash the version that matches your operating system’s driver version, don’t blindly go with version P20 or the latest version. On FreeNAS, you can determine the driver version using dmesg | grep “mps0: Firmware”:

mps0: Firmware:, Driver:

In this case, I had no data loss whatsoever and the person who reported this on the FreeBSD mailing list didn’t either, but both LSI and FreeBSD recommend to keep driver and firmware in sync.

Downgrading firmware

As I had to downgrade the firmware, I needed to do sas2flash.efi -o -e 6 before the flashing. Of course, this means that the crossflashing is undone and you’ll have to start over, flashing Dell 6GPBSAS.FW, flashing LSI P07 IT, and finally flashing the version matching your driver.

6GBPSAS.FW and LSI P07 need to be flashed with the LSI P05 sas2flash.efi because later versions will simply refuse when they encounter a NVDATA mismatch. Once that’s done, you can use the latest (P20 at the moment) sas2flash.efi to flash your final firmware version. This also has the advantage that you don’t get these “Adapter Reset Failed” warnings.

Below is a screenshot of the details of the P16 firmware:

Screen Shot 2014-12-03 at 09.11.40