Ubuntu 20.04: OpenMPI bind-to NUMA is broken when running without mpiexec

I tend to set the CPU pinning for my OpenMPI programs to the NUMA node. That way, they always access fast local memory without having to cross between processors. Some recent CPUs like the AMD Ryzen Threadripper have multiple NUMA nodes per socket, so pinning to the socket is not the same thing.

Since upgrading to Ubuntu 20.04, we were seeing error messages like this:

$ python3 -m mpi4py.bench helloworld
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

 Setting processor affinity failed failed
 --> Returned value Error (-1) instead of ORTE_SUCCESS

Launching through mpiexec/mpirun, even if it was with just one MPI rank, did not show the error:

$ mpiexec -n 1 python3 -m mpi4py.bench helloworld
Hello, World! I am process 0 of 1 on host1.
$ mpirun -n 1 python3 -m mpi4py.bench helloworld
Hello, World! I am process 0 of 1 on host1.
$ mpiexec -n 4 python3 -m mpi4py.bench helloworld
Hello, World! I am process 3 of 4 on host1.
Hello, World! I am process 0 of 4 on host1.
Hello, World! I am process 1 of 4 on host1.
Hello, World! I am process 2 of 4 on host1.

If you look through the OpenMPI code, you can see that CPU pinning is done by different code depending on whether you run standalone (called singleton mode) or through mpiexec. The relevant bit for the former is in ess_base_fns.c. It searches for a hwloc object of type HWLOC_OBJ_NODE (which is deprecated on the hwloc side and identical to the newer HWLOC_OBJ_NUMANODE). Since hwloc 2.0, NUMA nodes are no longer containers for CPU cores, but exist besides them inside a HWLOC_OBJ_GROUP.

$ lstopo --version
lstopo 1.11.9
$ lstopo --output-format console
Machine (31GB total) + Package L#0
  NUMANode L#0 (P#0 16GB)
    L3 L#0 (8192KB)
      L2 L#0 (512KB) + L1d L#0 (32KB) + L1i L#0 (64KB) + Core L#0
        PU L#0 (P#0)
        PU L#1 (P#12)
$ lstopo --version
lstopo 2.1.0
$ lstopo --output-format console
Machine (31GB total) + Package L#0
  Group0 L#0
    NUMANode L#0 (P#0 16GB)
    L3 L#0 (8192KB)
      L2 L#0 (512KB) + L1d L#0 (32KB) + L1i L#0 (64KB) + Core L#0
        PU L#0 (P#0)
        PU L#1 (P#12)

The current OpenMPI master (i.e. versions beyond the 4.1.x series) don’t bind through hwloc anymore, so the issue is fixed upstream (if only by accident). However, we’re stuck with Ubuntu 20.04 for the next two years, so let’s fix it ourselves. We load up the incriminating file, /usr/lib/x86_64-linux-gnu/openmpi/lib/libopen-rte.so.40.20.3, in Hopper and jump to orte_ess_base_proc_binding. Comparing it to its C code quickly reveals the instruction we need to change:

0x3 is OPAL_BIND_TO_NUMA and 0xd is HWLOC_OBJ_NODE. Looking at the hex code tells us that we need to make this change:

- 66 83 F8 03 0F 85 70 02 00 00 BA 0D 00 00 00
+ 66 83 F8 03 0F 85 70 02 00 00 BA 0C 00 00 00

Here’s a bit of Python code to do that:

import mmap
with open("/usr/lib/x86_64-linux-gnu/openmpi/lib/libopen-rte.so.40.20.3", 'r+b') as f:
m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE)
m.seek(m.find(bytes.fromhex("66 83 F8 03 0F 85 70 02 00 00 BA 0D 00 00 00")))
m.write( bytes.fromhex("66 83 F8 03 0F 85 70 02 00 00 BA 0C 00 00 00"))

Update 2021-02-26

The recent kernel update from to switched us from HWLOC_OBJ_GROUP to HWLOC_OBJ_DIE. lstopo now reports

$ lstopo --output-format console
 Machine (31GB total) + Package L#0
   Die L#0
     NUMANode L#0 (P#0 16GB)
     L3 L#0 (8192KB)
       L2 L#0 (512KB) + L1d L#0 (32KB) + L1i L#0 (64KB) + Core L#0
         PU L#0 (P#0)
         PU L#1 (P#16)

So the patch needs to be modified to have 0x13 in its fourth-to-last byte now.

What to do when Mathematica’s ParallelMap/ParallelTable takes a long time to start up

I have a Mathematica notebook that derives some rather massive expressions. I wanted to do some transformations on them in parallel using ParallelMap or ParallelTable, but noticed that these commands were only running on a single CPU core for hours before actually starting to run in parallel and occupy all CPU cores. While it was running on only that single CPU core, I could not even abort the evaluation using Alt-. like one usually can: it simply seemed stuck.

make_massive_expression[x_] := ...;
process[x_] := Simplify[x];
a1 = simple_expression;
a2 = make_massive_expression[a1];
a3 = make_massive_expression[a2];
as = {a1,a2,a3};

b = ParallelTable[process[as[[i]]], {i,Length[as]}];

As it turns out, during the startup phase Mathematica copies all definitions from the main kernel to the parallel kernels. And that seems to be a rather inefficient procedure. So let’s transfer the needed definitions manually.

make_massive_expression[x_] := ...;
process[x_] := Simplify[x];
a1 = simple_expression;
a2 = make_massive_expression[a1];
a3 = make_massive_expression[a2];
as = {a1,a2,a3};

DistributeDefinitions[as, process];
b = ParallelTable[process[as[[i]]], {i,Length[as]}, DistributedContexts -> None];

Now DistributeDefinitions is slow, but ParallelTable immediately starts running in parallel on multiple kernels. We haven’t gained anything by splitting things like this, but at least we can now tell exactly where the problem lies. So instead of transferring the massive expressions to the parallel kernels, let’s only transfer the simple expression and have the parallel kernels derive the massive expression themselves:

make_massive_expression[x_] := ...;
process[x_] := Simplify[x];
a1 = simple_expression;

DistributeDefinitions[a1, make_massive_expression, process];

   a2 = make_massive_expression[a1];
   a3 = make_massive_expression[a2];
   as = {a1,a2,a3}
), DistributedContexts -> None];

b = ParallelTable[process[as[[i]]], {i,Length[as]}, DistributedContexts -> None];

Leserbrief “Corona-Einschränkungen”

Im November 2020 beschlossen die Landesregierungen, einen großen Teil der Maßnahmen wiederherzustellen, die sie bereits im Frühjahr gegen die Ausbreitung des Coronavirus ergriffen hatten. Am 4. November 2020 druckte die Süddeutsche Zeitung dazu einen von mir verfassten Leserbrief:

Zu viel Optimismus

Der zweite Quasi-Lockdown zeigt, dass der erste keinen bleibenden Nutzen gestiftet hat, sondern lediglich das Unvermeidliche um einige Monate verzögerte. Auch der dritte oder vierte wird uns nicht nah genug an ein Heilmittel bringen, so sehr wir uns das auch wünschen mögen. Gleichzeitig setzt sich immer mehr die Erkenntnis durch, dass die in die Impfstoffentwicklung gesetzte Hoffnung viel zu optimistisch war und ein Impfstoff voraussichtlich die Eindämmungsmaßnahmen nicht obsolet machen wird. Man muss also durchaus die Frage stellen, ob das Ziel, das wir zu erreichen suchen, überhaupt erreichbar ist. Auch für die rechtliche Bewertung ist diese Frage elementar: Ist eine Maßnahme ungeeignet, ihr Ziel zu erreichen, so ist sie unverhältnismäßig. An einem übermächtigen Gegner wie einer Naturkatastrophe zu scheitern, ist jedenfalls keine Schande. Im Gegenteil, es zeigt, dass wir immer noch Menschen und keine Götter sind. Leider sind Politiker nicht bekannt dafür, eigene Fehler eingestehen zu können. Dies wird aber nötig sein, da es mit der aktuellen Strategie wohl kein „nach Corona“ geben wird – wenn man bloßes Wunschdenken überhaupt als Strategie bezeichnen kann.

Michael Kuron, Frickenhausen

Setting up BigBlueButton

Like so many other people, me and most of my two dozen colleagues are currently working from home full-time. While even before the current situation we have always had people work at home for individual days, we didn’t have the infrastructure to replace physical person-to-person communication. The first day was full of phone calls and emails, while our usual video conferencing system DFNconf, provided by the German research network, was struggling to keep up with growing demand. Microsoft Teams was also collapsing under the unexpected load, and I suspect other services like WebEx and Zoom had similar problems. As we might be stuck in this situation for months, we decided to take things into our own hands. For privacy reasons, we wouldn’t be able to use any of these commercial services anyway.

The first step was a chat system. We already have a self-hosted GitLab instance, so switching on Mattermost, an open-source competitor to Slack or Microsoft Teams, was a matter of minutes. Create a DNS record, wait for it to propagate, edit one GitLab config file, and restart GitLab twice.

Next step was video. This is what this article is going to be about. Unlike Microsoft Teams, Mattermost does not have a built-in video conferencing solution. It does have an API that allows third-party software and services to integrate with it. There is a list of video integrations. Our requirements were that it be self-hosted, free, straight forward to set up, and well maintained. That basically led us to BigBlueButton, which can interface to Mattermost via a plugin.


You need two Linux machines. We have access to an OpenStack cloud provided by the state (bwCloud), so that was easy. The first one, called turn.example.com, has 2 GB RAM, 2 CPU cores, and Ubuntu 18.04. The second one, called video.example.com, has 8 GB RAM, 4 CPU cores and Ubuntu 16.04 (no, that is not a typo). Once the machines are running, set up DNS records for IPv4 and IPv6 and wait until they propagate. Then, SSH into each of them and get them prepared:

sudo hostnamectl set-hostname XXX.example.com
sudo apt-get update
sudo apt-get install language-pack-en
sudo systemctl set-environment LANG=en_US.UTF-8
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo reboot

While that is running, configure your cloud provider’s firewall rules. turn.example.com needs incoming IPv4 and IPv6 access for tcp/80, tcpudp/3478, tcpudp/443, udp/49152-65535, while video.example.com needs tcp/80, tcp/443, udp/16384-32768.

Setting up the TURN server

SSH into turn.example.com and run

wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh | sudo bash -s -- \
  -c turn.example.com:YYYYYYYY -e you@example.com

Instead of the placeholder YYYYYYYY, you should use a random token. You’ll need it again in the next section to connect BigBlueButton to your TURN server.

Setting up BigBlueButton

SSH into video.example.com and run

wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh | sudo bash -s -- \
  -v xenial-22 -s video.example.com -e you@example.com \
  -c turn.example.com:YYYYYYYY
sudo apt-get install bbb-webhooks
sudo bbb-conf --stop
sudo sed -i 's/allowStartStopRecording=./allowStartStopRecording=false/g' \
sudo sed -i 's/disableRecordingDefault=./disableRecordingDefault=true/g' \
sudo bbb-conf --start
sudo bbb-conf --secret

The last command will print out a URL and a secret. Save these for later; you’ll need them to integrate with Mattermost. If you don’t have Mattermost, add -g to the third line and the installer will install the Greenlight management UI for you.

Integrating Mattermost and BigBlueButton

Download the latest release from the GitHub repo. Go to your Mattermost system console, go to Plugin Management and upload the file. Refresh the page and go to the plugin’s settings. Make sure the plugin is disabled, paste the URL (https://video.example.com/bigbluebutton/api) and secret from the previous step, and click Save. Then, enable the plugin and click Save again.

That’s it, you now have a video button at the top of every Mattermost conversation. Click it in a direct message or channel and it will post an invitation link. Everyone can click it to join. The plugin will always show the names of the people that have joined a conference. There’s an end meeting button, but the conference will automatically end a few minutes after the last person has left.

Configuring Phone Dial-in

On video.example.com, configure FreeSWITCH to route incoming calls by creating /opt/freeswitch/etc/freeswitch/dialplan/public/dialin.xml with the following contents:

<extension name="from_my_provider">
  <condition field="destination_number" expression="^ZZZZZZZZZZ">
    <action application="answer"/>
    <action application="sleep" data="500"/>
    <action application="play_and_get_digits" data="5 5 3 7000 # conference/conf-pin.wav ivr/ivr-that_was_an_invalid_entry.wav pin \d+"/>
    <action application="transfer" data="SEND_TO_CONFERENCE XML public"/>
 <extension name="check_if_conference_active">
  <condition field="${conference ${pin} list}" expression="/sofia/g" />
  <condition field="destination_number" expression="^SEND_TO_CONFERENCE$">
    <action application="set" data="bbb_authorized=true"/>
    <action application="transfer" data="${pin} XML default"/>

Configure your SIP PBX/provider to route calls for your number (assumed to be +49 711 12345678 in the following) to sip:ZZZZZZZZZZ@video.example.com;transport=tcp without registration. ZZZZZZZZZZ is a secret token and you should pick a random one.

In /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties on video.example.com, set

defaultWelcomeMessageFooter=<br><br>To join this meeting by phone, dial:<br>  %%DIALNUM%%<br>Then enter %%CONFNUM%% as the conference PIN number.<br>Note that this will only work once at least one person has joined the audio bridge from their computer.<br>You can mute and unmute yourself by pushing 0.

and restart BigBlueButton (sudo bbb-conf --stop && sudo bbb-conf --start). Open tcp/5060 in the firewall and you are ready.

Using Greenlight

If you are using Greenlight instead of Mattermost to manage your conferences, it will be available at https://video.example.com. I have not tried it, so I am leaving you to read the documentation yourself.


We decided to do this on Monday around noon (day one of the semi-lockdown) and I sent out the announcement email to my colleagues just four hours later. In other words, BigBlueButton is really easy to set up, thanks to bbb-install.

Today is day four. So far all our meetings were small (< 5 people), but BigBlueButton is extremely light on server resources. Audio quality is great (though the noise gate is a bit aggressive sometimes). Screen sharing and webcam streaming work well, even from networks with firewalls that block all UDP traffic. Firefox, Chrome and Safari work equally well, the only thing that is currently missing is screen sharing from Safari. The server mainly expends CPU time for mixing the audio conferences (extrapolating suggests we can handle at least 20 participants, probably more), while all video is just relayed to the other participants. That means that your server needs enough bandwidth for every participant to exchange ~500 Kbit/s with every other participant if everyone has their camera enabled. Your clients need 500 Kbit/s upstream total and 500 Kbit/s downstream for every other participant.

Week 2 Update (2020-03-24)

Today we had our first bigger video meeting. Ten people with audio, five with video and the server was operating at around 50% of one CPU core. Three problems were discovered:

  • Safari cannot send video if it is behind a firewall that blocks UDP, producing an error 1020. Judging from packet captures, it does not appear to fall back to the TURN server.
    Solution: use Chrome or Firefox.
  • Firefox cannot send audio if ICE is disabled. uBlock and some other privacy addons might cause that. Go to about:config and check whether any of the media.peerconnection.* settings have been modified from their defaults (are displayed in boldface).
    Solution: disable the addons and return these settings to their default. If it works after that, you might re-enable the addons and whitelist your server.
  • Some people don’t have headsets. Their microphones pick up ambient noise, overdrive, feed back, etc. and make audio a pain to listen to.
    Solution: get a USB headset. I have a Plantronics Blackwire C320 (mainly because it is compatible with my desk phone), which is a few years old and no longer sold, but you can buy its successor, the Plantronics Blackwire 3220. It’s cheap (around 30 Euros) and good enough for someone like me who only needs it for an hour or two per day. Of course, they are sold out everywhere, so be prepared to wait for multiple weeks to get yours delivered. Until then, use your smartphone headset as it’s still better than your computer’s built-in microphone, or dial into the conference via telephone.

Week 6 Update (2020-04-23)

We have had a few minor complaints about audio quality, mainly in direct comparison to Webex and Zoom. These seem to do better echo cancellation, apply some kind of magic audio processing that makes built-in microphones not sound as terrible, and have (better) packet loss concealment. Still, considering its price, privacy, and ease of use, I prefer BBB.

There is a more significant audio issue in BBB (#7007) though where you get occasional drops and crackles for no apparent reason. I set use-dtx=0, jitterbuffer=60, and energy-level=50 as suggested there and it gets a bit better, but there is still room for improvement. Hopefully that will be resolved by the BBB developers soon.

Today I updated to the latest version of BigBlueButton. It is as simple as running bbb-install again and only takes a minute or two.

Exploring the MoneyMoney database

MoneyMoney is a convenient online banking application for macOS, most useful if you have accounts with multiple different German banks. I wanted to export some of the data I had stored there in a format that the app didn’t support by itself, so I was wondering if I could pull it straight from the database. You can go to the Help menu and click Show Database in Finder and will be led to ~/Library/Containers/com.moneymoney-app.retail/Data/Library/Application Support/MoneyMoney/Database/MoneyMoney.sqlite. If you go to MoneyMoney’s About screen, you’ll see that it attributes SQLCipher.

So, download and compile SQLCipher like this:

git clone https://github.com/sqlcipher/sqlcipher.git
cd sqlcipher
./configure --enable-tempstore=yes CFLAGS="-DSQLITE_HAS_CODEC" LDFLAGS="-L/opt/local/lib -lcrypto" CC=clang CXX=clang++
make -j 4

Now you can open the database using

./sqlcipher "file://$HOME/Library/Containers/com.moneymoney-app.retail/Data/Library/Application Support/MoneyMoney/Database/MoneyMoney.sqlite?mode=ro"

and decrypt and dump it by typing the following commands at the SQLite prompt:

PRAGMA key = '<your database password>';
PRAGMA cipher_compatibility = 3;

The database password is just what you set in MoneyMoney and I figured out the compatibility level by trial and error. It seems like MoneyMoney didn’t yet get around to upgrading to SQLCipher 4, which was released in late 2018. It’s entirely possible that some older versions of MoneyMoney require a lower compatibility level and that some future version may be using level 4.

Disclaimer: You should not attempt to modify the database as that may cause MoneyMoney’s data to become inconsistent or modified in ways the developer did not anticipate. That’s why I open the file in read-only mode above. If you do modify it, you’re on your own.

Debugging code-signed applications on macOS

To debug a code-signed application on macOS, you need to turn off System Integrity Protection. Some applications however explicitly forbid debugging them. This manifests itself in lldb error messages like

Process 12345 exited with status = 45 (0x0000002d) 

If you google that message, you’ll find that this happens if the application disallows attaching a debugger by calling

ptrace(PT_DENY_ATTACH, 0, 0, 0)

There are many workarounds described around the internet. The one I like using is the following lldb command:

breakpoint set --name ptrace --command "thread return" --command "continue"

It just causes the ptrace call to be skipped. Of course, some applications take more involved measures to prevent attaching a debugger, but often this is sufficient.

Cherry MX Board 1.0 Review

After my old keyboard started randomly dropping characters as I typed them, I decided it was time to get a new one. Having used flat keyboards with scissor switches for over a decade, I was a bit fed up with crumbs getting caught in the mechanics and keys failing, so I decided it was time to upgrade to a mechanical keyboard. The Cherry MX switch series seems to be the most popular on this market, so my main difficulty was deciding on a color. Also, I found it slightly irritating that these keyboards are mainly marketed at gamers, even though they are great for typing too!

Cherry has three groups of MX switches: linear (Red, Black), tactile non-clicky (Brown, Clear), and tactile clicky (Blue). Since I always push my keys all the way through to the bottom, I don’t really need clickyness to get a sound. Also, I prefer my keyboard to be as silent as possible. I ended up ordering a Cherry MX Board 1.0 both with Brown switches and with Silent Red switches. The former are tactile, the latter are linear. So here is a list of those things I disliked on both keyboards.

MX Silent Red

  • If you catch a key by the edge while trying to hit another, it may still activate. That’s slightly annoying at first, but teaches you to type more precisely. The MX Black would require slightly more force and thus be less prone to this, but it’s not available in a silent variant.
  • When typing fast, the bottom metal plate has a faint high-pitched ring.
  • Lack of tactile response feels a bit weird during the first few thousand keystrokes, but you get used to it very quickly.

MX Brown

  • It is 6 dB louder than the Silent Red.

I ended up keeping the Silent Red because the difference in volume was just what crossed the line between bearable and annoying. I know you can put rubber O-rings between the key caps and the switches to silence them, but I just wanted something that I’m happy with out of the box. If Cherry decides to make a Silent Brown some day, I’m pretty sure I’ll buy it immediately though.

Snom D735 review

I’ve been using Snom Voice over IP telephones for about 10 years. Their software works reliably and provides all the features you might wish for, and the hardware is solid too. I know it’s 2018 and most people don’t use landline phones anymore, but the audio quality is still much better, you can’t comfortably hold a cell phone between your shoulder and your ear, and cellular reception isn’t great where I live.

I started with a 360, then had an 870 and later a 760. When it was time to get a new phone, my list of requirements was pretty short: it should have a USB port on the side for a headset and it should have a graphical display. That left only the D735, D765 and D785. The latter two are priced rather similarly, while the first one can occasionally be picked up for just around 100€.

This article isn’t going to be about the software running on the phones: it is and has always been great. Also, it’s the same across all of Snom’s models. So I’ll just write what I liked and didn’t like about the hardware.

I first tested the D785 for a few days. It’s rather bulky and while the large display looks great, the software doesn’t really make much use of all that extra space (yet). The self-labeling keys with the second display seem like a neat feature, but they are a bit hard to read when the backlight is off and not as useful as I had expected.

So I decided to settle for the D735. The one obvious downside is the tiny screen by comparison to the D765. The entire UI is sized down and even the phone number displayed while in a call scrolls horizontally because its width doesn’t fit. There is still quite a bit of whitespace on the call screen, so if Snom reduced the margins a bit, I think it could actually fit a lot more onto that screen. The downside is also an upside: the phone is smaller, more akin to the D715 than to the D765. While the D765 has two rows of six speed dial keys each above the keypad, the D735 has four of them on either side of the display. That allows it to display labels for them on the screen so you can immediately see what would happen if you pressed them. It also lets you have four pages of different speed dial keys. The labels are very narrow — just showing an icon and a few characters of text. However, they tie in with the proximity sensor. Snom has advertised that as a unique and highly innovative feature, which seems overblown — until you actually try it. Just move your hand close to the phone and it displays the full key label (across half the width of the screen). This allows you to put a lot more text into the label than you could on the paper-labeled keys of the D715 or D765 and even more than on the second screen of the D785.

Personally, I think the D735 has the potential to replace the D715 as Snom’s “default” phone. Supposedly the D715 is their best selling device. Since the D735 only costs a little more and has a color screen and more speed dial keys, it seems like a no-brainer to prefer it over the D715. I can also see it cutting into the D765’s sales a bit — if you don’t mind the smaller screen, you get basically the same feature set in a smaller case. The D785 still remains Snom’s top of the line model — if you want a gorgeous huge screen and self-labeling speed dial keys, it offers a great package. The D735 however may just provide the best value of any of the devices Snom currently has in its lineup.

OpenWRT on AVM Fritz!Box 3370

I was looking for a new DSL modem and router as I am switching from cable to VDSL2. I was eyeing the Ubiquiti EdgeRouter series for a while because they have a big feature set at a reasonable price. I was a bit reluctant about the X series as they seem to be a bit troubled by their small flash memory and the Lite doesn’t have an SFP slot, which would have been nice for the fibre-to-the-home future. Also, both the X and the Lite have been available for quite a few years now, so I’m not sure how long they would have remained in firmware support. The 4 is much more expensive however, and I’d still need a VDSL2 modem, which seems to cost around 100€ (e.g. Draytek Vigor 130 or Allnet ALL-BM200VDSL2V).

Of course, I could have gotten an off-the-shelf router with an integrated modem, like the AVM Fritz!Box series that’s very popular in Germany and probably paid less in total (standalone VDSL2 modems are rather expensive because not many people want/need them). I had a Fritz!Box on cable for the past few years and am not particularly happy with the quality of their firmware though. The hardware is great, however.

So I decided to go with OpenWRT. The only built-in DSL modems it supports are Lantiq chips, so it had to be one from that list. I wanted something that has at least 64 MB of flash memory (so I could install some extra packages) and Gigabit Ethernet on all four ports. Luckily, OpenWRT recently got full support for the AVM Fritz!Box 3370. AVM announced that model back in 2010 and dropped official support for it in 2015, so they are available for ~25€ on eBay nowadays. Other models that would have been nice but are not currently supported by OpenWRT are the 3390 (simultaneous dual-band WiFi), and 3490/7490 (USB 3.0, 802.11ac, 512 MB flash memory and 256 MB RAM; the 7490 additionally has phone ports which can’t be used with OpenWRT). The ZyXEL P-2812HNU-F1 and P-2812HNU-F3 are quite similar to the AVM 3370, but they are not as readily available on eBay and tend to cost about twice as much.

OpenWRT doesn’t provide too much information on how to install, but it’s quite straight-forward. First, you need to check if your device is at least revision 2 and that it doesn’t have a certain bad bootloader version that makes installation more difficult. Go to on the original firmware, log in and click “Support-Daten erstellen”. In the resulting file, you should see something like the following at the top:

HWRevision      175
HWSubRevision   5
ProductID       Fritz_Box_3370
urlader-version 2475

Also, we need to know what kind of flash memory chip the device has. Scroll down to ##### BEGIN SECTION dmesg and look for something like

[ 1.450000] [HSNAND] Hardware-ECC activated
[ 1.450000] NAND device: Manufacturer ID: 0x2c, Chip ID: 0xf1 (Micron NAND 128MiB 3,3V 8-bit)

Download the files corresponding to your flash chip. Set your IP address statically to and reboot the router. When the ethernet interface comes up after a few seconds, ftp and upload the firmware as documented by OpenWRT:

quote USER adam2
quote PASS adam2
quote SETENV linux_fs_start 0
put openwrt-lantiq-xrx200-avm_fritz3370-rev2-micron-squashfs-eva-kernel.bin mtd1
put openwrt-lantiq-xrx200-avm_fritz3370-rev2-micron-squashfs-eva-filesystem.bin mtd0
quote REBOOT

OpenWRT is now ready at after a few minutes.

Note that Fritz!Box 3370 support is not in the 18.06 release version, only in the current snapshot builds. This means that the luci web interface is not pre-installed and you should only install new packages within the first few days after flashing (so you don’t pick up newer packages incompatible with your firmware).

After using the 3370 for a little while, unfortunately I noticed that it doesn’t handle more than around 60 Mbit/s. So be warned that it might not exhaust a 100 Mbit/s down, 50 Mbit/s up line. This seems to be due to a slightly underpowered CPU.

Root disk spindown on Debian 9

I recently installed Debian 9 on a Seagate PersonalCloud. Because the device will only get backed up to once every day, I want its disk to be spun down when it’s not needed. Even on a minimal install, you’ll find quite a few background services that access the root disk every few minutes. Here is what I had to do to keep my disk spun down.


First, install hdparm (apt-get install hdparm) and configure /etc/hdparm.conf to spin down your disks after 10 minutes:

spindown_time = 120

Since hdparm isn’t available in the initrd, when the rule /lib/udev/rules.d/85-hdparm.rules fires, you need to add /etc/systemd/system/hdparm-sda.service:

Description=hdparm sda



Now systemctl daemon-reload && systemctl enable hdparm-sda && systemctl start hdparm-sda.


When you look into /var/log/syslog, you see messages like

CRON[393]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

Of course, we can’t have these messages written to a log if we want the disk to remain spun down, so edit /etc/crontab and comment out the hourly job. As long as /etc/cron.hourly is empty, this will not do any harm. If you have any hourly jobs, you might want to move them to daily jobs.

systemctl timers

Systemctl has its own cron-like timer mechanism. You can view active timers with systemctl list-timers --all and disable ones you don’t need, especially ones that run more than once a day. systemctl disable snapper-timeline.timer && systemctl stop snapper-timeline.timer.


If you have smartmontools installed (apt-get install smartmontools), you’ll see lines like the following appear in the syslog when the disk is spun down:

smartd[294]: Device: /dev/sda [SAT], is in STANDBY mode, suspending checks

Writing these messages causes the disk to spin up, so we need to disable smartd: systemctl stop smartd && systemctl disable smartd. To keep monitoring our disks, put the following into /etc/cron.daily/smart-check and then chmod +x /etc/cron.daily/smart-check:


/usr/sbin/smartctl -q errorsonly -A /dev/sda


When you run

inotifywait -m -r -e access -e modify -e create -e delete --format 'PATH:%w%f EVENTS:%,e' --exclude '/(dev/pts|proc|sys|run)' /

to see what is going on on your disk, you’ll see that your temporary directories are being cleaned every couple of minutes. We can reduce that to bi-weekly by running systemctl edit systemd-tmpfiles-clean.timer and pasting



If you have Postfix installed, inotify will also show you that it periodically checks its queue directories. So uninstall postfix (apt-get remove postfix) and configure a forwarding MTA that does not run as a daemon.


This last one was tricky to discover because it doesn’t appear in the logs and inotify doesn’t see it. You can logging of every disk access to the kernel log to see even more, but you need to disable syslog, otherwise you’ll get a self-amplifying write cascade.

systemctl stop syslog.socket
systemctl stop rsyslog.service
dmesg -C
echo 1 | sudo tee /proc/sys/vm/block_dump
dmesg -Tw

Here you’ll see that systemd-timesyncd stores its last sync date by touching a file. It only changes metadata, which is why inotify doesn’t see it happening. My solution was to put the following into /etc/tmpfiles.d/zz-clock.conf:

d /run/systemd/timesync 0755 systemd-timesync systemd-timesync - -
f /run/systemd/clock 0644 systemd-timesync systemd-timesync - -
f /run/systemd/timesync/clock 0644 systemd-timesync systemd-timesync - -
L+ /var/lib/systemd/timesync/clock - - - - /run/systemd/timesync/clock
L+ /var/lib/systemd/clock - - - - /run/systemd/clock

/var/lib/systemd/clock is used by systemd 234 and lower, while /var/lib/systemd/timesync/clock is used by systemd 235 and higher. So the latter will only be needed once you upgrade to Debian 10.