I am not including comments in the SLOC of the diff, otherwise it ruins the story >:)
Also this is inspired by the OpenBSD tale.
One day, a young sysadmin stumbles across a debate within the XMPP community on Full Disk Encrypted servers. The concern within the XMPP community is XMPP hosting providers have to store the messages for a period of time to allow for syncing of history between devices (for more info see XEP-0313. Even with End to End Encryption (E2EE), a lot of the community feels strongly on the security of the data at rest on the server.
This young sysadmin, being obsessed with BSD and not Linux (unlike the majority of the XMPP community), decided to design a full disk encrypted server stack as a proof of concept.
Lets start with some very basic theory on FDE. The idea is you have some
encryption key, which is used to encrypt any data before it is written to
disk, so the physical data on disk is unreadable. The most common algorithm
used is AES-XTS with 256 or 512 bit keys.
Yes I know I left out data integrity verification, I am trying to keep this light.
Now lets address the methods typically used:
Both passphrase and keydisk need you to attend the boot, you manually need to type in the passphrase, or plug in the keydisk for the operating system to boot. With laptops, this works well, you always have your laptop on you when you want to use it.
But when it comes to servers, we often use our servers remotely, we are not there to attend the boot, and without a workaround, security patches become difficult to apply, as once the system is rebooted to load them, you would need to be there in person.
I am not going to cover enterprise hardware, which have management software embedded on the hardware, see: IPMI
In the case of TPM, it is loved within enterprise as it makes full disk encryption seemless. Unfortunately it relies on the hardware firmware being trustworthy, and in many cases, it is not. However ignoring this concern entirely, we need to think about how TPM works.
TPM stores the key on the hardware, therefore if we assume physical security is compromised, the adversary already has access to the hardware, and the storage, so nothing stops them decrypting the data.
Many companies use a passphrase paired with TPM to negate this however we go full circle, and we now have the requirement of attended boot again.
In the case of keydisk, we must be physically there to plug in the device, and if we leave it plugged in, then again if we assume an adversary has physical access, they take the keydisk and the storage disks and they make a runner.
In the end, we always come back to requiring a passphrase of some sort in the end.
All hope is not lost, this problem has been solved within enterprise for decades, not explicitly for full disk encryption, but for colocation. Customers which colocate their servers in a datacentre, need some way to remotely manage the device in case they loose ssh access, or need to reinstall the host. There is two main options here.
IPKVM literally standing for "keyboard, video and mouse" provide exactly what their name suggests, video output, and keyboard and mouse input to a device. Within datacentres these are rack mounted, and then connected up to servers within the rack, and the IPKVM then provides video output over the network allowing a customer to manage the server as if they physically plugged in monitor, keyboard and mouse.
This is the preferred option within the industry, because it is the most flexible. It works for both terminal based servers, and graphical servers (especially in the case of supporting Windows Server).
The downside is the hardware is expensive, and even the DIY solutions can run you up a hefty amount of upfront investment. The other downside is it requires graphical output on a server, and on a lot of desktop hardware, there is no integrated graphics card, and running a graphics card 24/7 can add an extra 10-20Wh to the enegy bill, and at scale, this can be costly.
Terminal servers are the tried and tested method of accessing a host remotely. Many embedded devices have a COM port, usually an RJ45 port, which can be connected to a server to aggregate serial connections, and allow devices to access them over the internet. RS232 is still provided by many servers, and some desktop hardware.
Some hosting providers still give serial access, but due it being terminal only, can be limiting for customers.
Another downside is a lot of consumer hardware does not provide serial output for uefi/bios configuration, meanwhile IPKVM typical works with all hardware which have a display output.
Many Linux distros do not support serial by default as well, making it more difficult to setup on certain distros.
So the young sysadmin first attempt involved two TTL USB adapters with the ends cut off, stripped and soldered so that rx on one went to the tx of the other, and vice versa (crossover is required for serial communications), this then effectively made a USB to USB serial adapter:
USB Port (Terminal Server) <--> UART <--> UART <--> USB Port (Server)
On FreeBSD, TTL UART adapters show up as ttyUx (notice the capital U), as
COM ports show up as ttyux. You can then configure
getty
to configure the interface, for this the following was added to /etc/ttys:
ttyU0 "/usr/libexec/getty 3wire.115200" vt100 on secure
This configures ttyU0 with 3wire as handshaking often doesn't work on
these hacky setups, at a baudrate of 115200, as a terminal vt100, on
so its enabled, and secure so you can login as root via serial. You can
set this to insecure to disable root login on this tty.
The young sysadmin found this worked well, however once he strayed into the bootloader, realised that USB based serial adapters were not going to cut it. He would find that the motherboard for the proof of concept server had a JCOM port (a header to attach a RS 232 pci bracket to). He bought an adapter, and an rs 232 to USB adapter.
However when plugging it all in, there was no interface showing up. Being
misled by someone on IRC, he assumed that it was down to crossover issues,
so bought a crossover adapter, same problem persisted, until someone in
##electronics pointed out that a USB adapter should always show up as an
interface no matter if there is a proper serial connection or not.
The young sysadmin banging his head against the wall finally realised the brand new cable was DOA (Dead On Arrival) and bought a replacement, and after reading the manual for the JCOM adapter, found that a crossover adapter was not even needed.
Next came setting up getty access again, as now its a "COM port" it follows
the ttyux naming, so /etc/ttys contained the following:
ttyu0 "/usr/libexec/getty 3wire.115200" vt100 on secure
Next, outputting loader to serial, by adding the following to
/boot/loader.conf:
console="comconsole,efi"
boot_serial="YES"
boot_multicons="YES"
comconsole_speed="115200"
The young sysadmin found when rebooting that the geli prompt was being outputted via serial, and thought over and over again he must have messed something up until realising when geli was decrypted that stage 2, the FreeBSD splash screen which allows you to configure the loader's boot behaviour for FreeBSD, did work correctly over serial, and efi console.
After much moaning, a wise developer by the username kevans got involved
in troubleshooting the issue.
The issue explained by kevans was that /boot was encrypted, so no
configuration at stage 1 (decryption) can be seen. This is when the
loader.env configuration file was introduced to the young sysadmin, with
its role of configuring the first stage of the loader.
After looking through the source code, the young sysadmin saw where kevans
got the config option of -hD from, and where the file was being loaded
from, so ran the following:
echo -n "-hD" > /boot/efi/efi/freebsd/loader.env
Then rebooted.
Upon rebooting, the young sysadmin realised that he still couldn't get the
efi decryption prompt via serial. kevans then suggested the issue is
likely down to the placement of the devinit() function call.
What was discovered was that devinit() was being called too early, but
that due to it handling both the init of the comconsole device and geli, it
was not currently possible to parse the loader configuration from
loader.env and init after.
Instead, a one line diff was created, move the devinit() function call and
pass the config via the efi entry.
To the young sysadmin's shock, it worked, and months of suffering had
finally come to the end. kevans however, discussed dividing devinit()
into a early and late init to solve the problem entirely without the
workaround.
This young sysadmin was me, and speaking in the third person was to give the blogpost a storytelling vibe!
Firstly the default efi entry made is insufficient, and this needs to be replaced:
efibootmgr -a -c -l /boot/efi/efi/freebsd/loader.efi -L FreeBSD-Serial -e console=comconsole,efi
This passes the console variable to the loader, allowing you to skip
loader.env which is currently not possible to use.
Next apply the diff on phabricator to
/usr/src, as this patch is not currently in the latest release.
Then compile the loader:
cd /usr/src/stand && make
Then install the modified loader:
mv /boot/efi/efi/freebsd/loader.efi /boot/efi/efi/freebsd/loader.efi.bak && cp /usr/obj/usr/src/amd64.amd64/stand/efi/loader_lua/loader_lua.efi /boot/efi/efi/freebsd/loader.efi
Reboot and you will see the geli prompt via serial, success!!!
And that concludes the tale of the one line diff, a tale of shear determination, of rebooting your system and typing in the same FDE password over, and over, and over again, edging yourself that little bit closer to insanity.
Thank you to kevans for sticking with the process, and writing the one
line diff to have a workaround for the problem. Hopefully loader.env one
day will fully function and allow the efibootmgr step to be skipped.
Thank you all for reading.