When running a web site on OS X Server 5.0 for a while (on OS X 10.10.5 in my case), eventually you’ll notice hundreds of httpd processes in Activity Monitor. One or two might cause a bit of CPU load, while the others don’t do anything. When you try to load the web page, it is insanely slow and often throws a HTTP 500 error, a HTTP 502 Proxy Error, or the connection just times out. /var/log/apache2/error_log reports errors like
[Thu Nov 05 13:15:24.435549 2015] [mpm_prefork:error] [pid 60920] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting.
but that’s the only hint you get. To find out more, add the following lines inside the VirtualHost section of /Library/Server/Web/Config/apache2/sites/0000_127.0.0.1_34580_.conf and restart the Websites service in Server.app:
<Location /server-status> SetHandler server-status </Location>
Now, got to http://localhost/server-status and refresh it occasionally while traffic hits your web site. You’ll eventually see dozens of lines like the one below (starting with 52-0 in my case):
Scoreboard Key: "_" Waiting for Connection, "S" Starting up, "R" Reading Request, "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup, "C" Closing connection, "L" Logging, "G" Gracefully finishing, "I" Idle cleanup of worker, "." Open slot with no current process Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request 52-0 80825 0/2/2 C 0.45 51 0 0.0 0.02 0.02 ::1
All httpd processes you’re seeing in Activity Monitor are stuck in “Closing connection”, except for those that cause considerable CPU load. If the server were behaving correctly, you wouldn’t have as many processes and those that aren’t currently handling requests would either be “Waiting for Connection” or “Open slot with no current process”.
Let’s fire up a debugger to see what’s causing the processes to get stuck:
$ sudo lldb -p 80825 (lldb) process attach --pid 80825 Process 80825 stopped * thread #1: tid = 0x3a69d4, 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP frame #0: 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10 libsystem_kernel.dylib`__wait4: -> 0x7fff8fe35902 <+10>: jae 0x7fff8fe3590c ; <+20> 0x7fff8fe35904 <+12>: movq %rax, %rdi 0x7fff8fe35907 <+15>: jmp 0x7fff8fe30c78 ; cerror 0x7fff8fe3590c <+20>: retq Executable module set to "/usr/sbin/httpd". Architecture set to: x86_64-apple-macosx. (lldb) bt * thread #1: tid = 0x3a69d4, 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP * frame #0: 0x00007fff8fe35902 libsystem_kernel.dylib`__wait4 + 10 frame #1: 0x0000000109b3ee95 libapr-1.0.dylib`apr_proc_wait + 70 frame #2: 0x000000010aa4290c mod_auth_digest_apple.so`cleanup_server_event + 73 frame #3: 0x0000000109b3627a libapr-1.0.dylib`apr_pool_destroy + 82 frame #4: 0x0000000109a74ce2 httpd`clean_child_exit + 50 frame #5: 0x0000000109a74c59 httpd`child_main + 2393 frame #6: 0x0000000109a73b2e httpd`make_child + 510 frame #7: 0x0000000109a74181 httpd`perform_idle_server_maintenance + 1265 frame #8: 0x0000000109a72887 httpd`prefork_run + 2471 frame #9: 0x0000000109a26328 httpd`ap_run_mpm + 120 frame #10: 0x0000000109a1185f httpd`main + 4687 frame #11: 0x00007fff8d7435c9 libdyld.dylib`start + 1 (lldb) continue Process 82070 resuming (lldb) exit
Ah, so mod_auth_digest_apple.so is the culprit. So in all /Library/Server/Web/Config, replace every occurrence of
AuthType Digest with
AuthType Basic and comment out all lines containing
mod_auth_digest_apple.so by prepending a
# character. Restart the Websites service in Server.app. Watch http://localhost/server-status: everything should be fine no and no more connections stuck on “Closing connection”!