Like so many other people, me and most of my two dozen colleagues are currently working from home full-time. While even before the current situation we have always had people work at home for individual days, we didn’t have the infrastructure to replace physical person-to-person communication. The first day was full of phone calls and emails, while our usual video conferencing system DFNconf, provided by the German research network, was struggling to keep up with growing demand. Microsoft Teams was also collapsing under the unexpected load, and I suspect other services like WebEx and Zoom had similar problems. As we might be stuck in this situation for months, we decided to take things into our own hands. For privacy reasons, we wouldn’t be able to use any of these commercial services anyway.

The first step was a chat system. We already have a self-hosted GitLab instance, so switching on Mattermost, an open-source competitor to Slack or Microsoft Teams, was a matter of minutes. Create a DNS record, wait for it to propagate, edit one GitLab config file, and restart GitLab twice.

Next step was video. This is what this article is going to be about. Unlike Microsoft Teams, Mattermost does not have a built-in video conferencing solution. It does have an API that allows third-party software and services to integrate with it. There is a list of video integrations. Our requirements were that it be self-hosted, free, straight forward to set up, and well maintained. That basically led us to BigBlueButton, which can interface to Mattermost via a plugin.

Prerequisites

You need two Linux machines. We have access to an OpenStack cloud provided by the state (bwCloud), so that was easy. The first one, called turn.example.com, has 2 GB RAM, 2 CPU cores, and Ubuntu 18.04. The second one, called video.example.com, has 8 GB RAM, 4 CPU cores and Ubuntu 16.04 (no, that is not a typo). Once the machines are running, set up DNS records for IPv4 and IPv6 and wait until they propagate. Then, SSH into each of them and get them prepared:

sudo hostnamectl set-hostname XXX.example.com
sudo apt-get update
sudo apt-get install language-pack-en
sudo systemctl set-environment LANG=en_US.UTF-8
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo reboot

While that is running, configure your cloud provider’s firewall rules. turn.example.com needs incoming IPv4 and IPv6 access for tcp/80, tcpudp/3478, tcpudp/443, udp/49152-65535, while video.example.com needs tcp/80, tcp/443, udp/16384-32768.

Setting up the TURN server

SSH into turn.example.com and run

wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh | sudo bash -s -- \
  -c turn.example.com:YYYYYYYY -e you@example.com

Instead of the placeholder YYYYYYYY, you should use a random token. You’ll need it again in the next section to connect BigBlueButton to your TURN server.

Setting up BigBlueButton

SSH into video.example.com and run

wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh | sudo bash -s -- \
  -v xenial-22 -s video.example.com -e you@example.com \
  -c turn.example.com:YYYYYYYY
sudo apt-get install bbb-webhooks
sudo bbb-conf --stop
sudo sed -i 's/allowStartStopRecording=./allowStartStopRecording=false/g' \
  /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
sudo sed -i 's/disableRecordingDefault=./disableRecordingDefault=true/g' \
  /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
sudo bbb-conf --start
sudo bbb-conf --secret

The last command will print out a URL and a secret. Save these for later; you’ll need them to integrate with Mattermost. If you don’t have Mattermost, add -g to the third line and the installer will install the Greenlight management UI for you.

Integrating Mattermost and BigBlueButton

Download the latest release from the GitHub repo. Go to your Mattermost system console, go to Plugin Management and upload the file. Refresh the page and go to the plugin’s settings. Make sure the plugin is disabled, paste the URL (https://video.example.com/bigbluebutton/api) and secret from the previous step, and click Save. Then, enable the plugin and click Save again.

That’s it, you now have a video button at the top of every Mattermost conversation. Click it in a direct message or channel and it will post an invitation link. Everyone can click it to join. The plugin will always show the names of the people that have joined a conference. There’s an end meeting button, but the conference will automatically end a few minutes after the last person has left.

Configuring Phone Dial-in

On video.example.com, configure FreeSWITCH to route incoming calls by creating /opt/freeswitch/etc/freeswitch/dialplan/public/dialin.xml with the following contents:

<extension name="from_my_provider">
  <condition field="destination_number" expression="^ZZZZZZZZZZ">
    <action application="answer"/>
    <action application="sleep" data="500"/>
    <action application="play_and_get_digits" data="5 5 3 7000 # conference/conf-pin.wav ivr/ivr-that_was_an_invalid_entry.wav pin \d+"/>
    <action application="transfer" data="SEND_TO_CONFERENCE XML public"/>
  </condition>
 </extension>
 <extension name="check_if_conference_active">
  <condition field="${conference ${pin} list}" expression="/sofia/g" />
  <condition field="destination_number" expression="^SEND_TO_CONFERENCE$">
    <action application="set" data="bbb_authorized=true"/>
    <action application="transfer" data="${pin} XML default"/>
  </condition>
 </extension>

Configure your SIP PBX/provider to route calls for your number (assumed to be +49 711 12345678 in the following) to sip:ZZZZZZZZZZ@video.example.com;transport=tcp without registration. ZZZZZZZZZZ is a secret token and you should pick a random one.

In /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties on video.example.com, set

defaultDialAccessNumber=+49-711-12345678
defaultWelcomeMessageFooter=<br><br>To join this meeting by phone, dial:<br>  %%DIALNUM%%<br>Then enter %%CONFNUM%% as the conference PIN number.<br>Note that this will only work once at least one person has joined the audio bridge from their computer.<br>You can mute and unmute yourself by pushing 0.

and restart BigBlueButton (sudo bbb-conf --stop && sudo bbb-conf --start). Open tcp/5060 in the firewall and you are ready.

Using Greenlight

If you are using Greenlight instead of Mattermost to manage your conferences, it will be available at https://video.example.com. I have not tried it, so I am leaving you to read the documentation yourself.

Conclusion

We decided to do this on Monday around noon (day one of the semi-lockdown) and I sent out the announcement email to my colleagues just four hours later. In other words, BigBlueButton is really easy to set up, thanks to bbb-install.

Today is day four. So far all our meetings were small (< 5 people), but BigBlueButton is extremely light on server resources. Audio quality is great (though the noise gate is a bit aggressive sometimes). Screen sharing and webcam streaming work well, even from networks with firewalls that block all UDP traffic. Firefox, Chrome and Safari work equally well, the only thing that is currently missing is screen sharing from Safari. The server mainly expends CPU time for mixing the audio conferences (extrapolating suggests we can handle at least 20 participants, probably more), while all video is just relayed to the other participants. That means that your server needs enough bandwidth for every participant to exchange ~500 Kbit/s with every other participant if everyone has their camera enabled. Your clients need 500 Kbit/s upstream total and 500 Kbit/s downstream for every other participant.

Week 2 Update (2020-03-24)

Today we had our first bigger video meeting. Ten people with audio, five with video and the server was operating at around 50% of one CPU core. Three problems were discovered:

• Safari cannot send video if it is behind a firewall that blocks UDP, producing an error 1020. Judging from packet captures, it does not appear to fall back to the TURN server.
  Solution: use Chrome or Firefox.
• Firefox cannot send audio if ICE is disabled. uBlock and some other privacy addons might cause that. Go to about:config and check whether any of the media.peerconnection.* settings have been modified from their defaults (are displayed in boldface).
  Solution: disable the addons and return these settings to their default. If it works after that, you might re-enable the addons and whitelist your server.
• Some people don’t have headsets. Their microphones pick up ambient noise, overdrive, feed back, etc. and make audio a pain to listen to.
  Solution: get a USB headset. I have a Plantronics Blackwire C320 (mainly because it is compatible with my desk phone), which is a few years old and no longer sold, but you can buy its successor, the Plantronics Blackwire 3220. It’s cheap (around 30 Euros) and good enough for someone like me who only needs it for an hour or two per day. Of course, they are sold out everywhere, so be prepared to wait for multiple weeks to get yours delivered. Until then, use your smartphone headset as it’s still better than your computer’s built-in microphone, or dial into the conference via telephone.

Week 6 Update (2020-04-23)

We have had a few minor complaints about audio quality, mainly in direct comparison to Webex and Zoom. These seem to do better echo cancellation, apply some kind of magic audio processing that makes built-in microphones not sound as terrible, and have (better) packet loss concealment. Still, considering its price, privacy, and ease of use, I prefer BBB.

There is a more significant audio issue in BBB (#7007) though where you get occasional drops and crackles for no apparent reason. I set use-dtx=0, jitterbuffer=60, and energy-level=50 as suggested there and it gets a bit better, but there is still room for improvement. Hopefully that will be resolved by the BBB developers soon.

Today I updated to the latest version of BigBlueButton. It is as simple as running bbb-install again and only takes a minute or two.