Ubuntu Server 12.04.3 / 12.10 / 13.04 / 13.10 with encrypted md mirrored rootfs and remote ssh unlock

My original plan was to boot from ZFS, but combining it with encryption and especially with mirrored disks turned out to be too troublesome. Yes, I did get it working and booting, but resulting configuration required manual fixing each time new kernel was installed. And don't even think about doing upgrade to next Ubuntu version.



Therefore I decided to stay with old and create md mirrored /boot (ext4, unencrypted) and md mirrored / (ext4, dmcrypt). This machine will be hosting KVM virtual machines and ZFS as storage for them. So it doesn't really need root on ZFS.


Boot from Ubuntu Server 12.04.3 install media

If there's any existing partitions delete them all AND REBOOT.

Proceed as usual until disk partitioning prompts. BTW, say NO to encrypt home with eCryptfs question earlier.

Select Manual partitioning.

Create new partitions on both SSDs:
- Size 500MB, Primary, Beginning of disk, Use as: Physical volume for RAID, Bootable flag: ON
- Size xxGB, Primary, Beginning of disk, Use as: Physical volume for RAID, Bootable flag: OFF

Configure software RAID
- Create MD device
- RAID1, Active devices 2, Spare devices 0, pick both 500MB partitions
- Create MD device
- RAID10, Active devices 2, Spare devices 0, pick both xxGB partitions
- Notice we're using RAID10 for root even with only two underlying devs. This will double your read performance compared to RAID1 (maybe). Yes, I'm serious. Trust me, I know what I'm doing.

Configure encrypted volumes
- Create encrypted volumes
- xxGB mirror should be visible as /dev/md1, select it
- Default for crypto are ok

Select md1_crypt. Format as ext4, mount point /, label root
Select md0 500MB and format as ext with mount point /boot, label boot

Finnish partitioning and write changes to disk

Yes, we want to boot even with degraded RAID
No, we don't need swap
Yes, write changes to disk

Rest is usual Ubuntu Server install - Except when/if you get asked for location of grub boot sector. Pick your first disk, typically /dev/sda. Installer will suggets /dev/md, but no such device exist. We'll install boot loader to other side of mirror later.

Boot, should prompt for password during initramfs.

Login, sudo to root, fix second disk boot sector
grub-install /dev/sdb (or whatever it is)
Reboot

Pretty standard and straightforward so far. Rest of steps are for enabling remote unlock of encrypted root. Mix of own attempts combined with several partial solutions found from The Interwebs.

# Apply updates and install both ssh servers we need for remote unlock over ssh
apt-get update
apt-get -y dist-upgrade
apt-get -y install openssh-server dropbear
 
# As usual, opensource quality bites us again. Remote unlocking in
# Debian and Ubuntu has been broken for years. Usual fix people do is
# hack and slash dropbear initramfs scripts. Not only those instructions
# are incomplete they also break if you don't blacklist dropbear from
# updates. Also upgrading to newer release of Ubuntu breaks them. No good.
 
# To workaround these problems we create new script named dropbear.fixup
# rather than touch any existing scripts. I've tested this and it does
# indeed survive updating 12.04.3 -> 12.10 -> 13.04 and even to 13.10 beta.
# Most of this script comes from someones blog. Thanks!
 
# There's interesting "feature" in 13.10 initramfs. It doesn't load
# drivers for USB keyboard at all and therefore only way to unlock such
# system is via SSH - and even that only with hack below implemented
# prior installing or upgrading to 13.10.
 
# Create script to run whenever initramfs is created and apply fixups
cat >/usr/share/initramfs-tools/hooks/dropbear.fixup <<'__EOF__'
#!/bin/sh
PREREQ="dropbear"
prereqs() {
    echo "$PREREQ"
}
case "$1" in
    prereqs)
        prereqs
        exit 0
    ;;
esac
  
. "${CONFDIR}/initramfs.conf"
. /usr/share/initramfs-tools/hook-functions
  
if [ "${DROPBEAR}" != "n" ] && [ -r "/etc/crypttab" ] ; then
 
# Create missing nsswitch.conf
echo "passwd: files" > "${DESTDIR}/etc/nsswitch.conf"
 
# Create missing shadow
grep "^root:" /etc/shadow > "${DESTDIR}/etc/shadow"
 
# Copy missing libs
cp /lib/x86_64-linux-gnu/libnss_* "${DESTDIR}/lib/"
 
# Create unlock script
cat > "${DESTDIR}/bin/unlock" << _EOF_
#!/bin/sh
if PATH=/lib/unlock:/bin:/sbin /scripts/local-top/cryptroot; then
    kill \`ps | grep cryptroot | grep -v "grep" | awk '{print \$1}'\`
    exit 0
fi
exit 1
_EOF_
chmod 755 "${DESTDIR}/bin/unlock"
 
# No idea :)
mkdir -p "${DESTDIR}/lib/unlock"
cat > "${DESTDIR}/lib/unlock/plymouth" << _EOF_
#!/bin/sh
[ "\$1" == "--ping" ] && exit 1
/bin/plymouth "\$@"
_EOF_
chmod 755 "${DESTDIR}/lib/unlock/plymouth"
 
# Update motd
echo To unlock root-partition run "unlock" >> ${DESTDIR}/etc/motd
 
fi
__EOF__
 
# Make it executable
chmod a+x /usr/share/initramfs-tools/hooks/dropbear.fixup

# Second script to handle converting SSH keys.
# You might NOT want to use this as now your SSH keys are stored inside
# plaintext initramfs instead of only encypted volume.
cat >/usr/share/initramfs-tools/hooks/dropbear.fixup2 <<'__EOF__'
#!/bin/sh
PREREQ="dropbear"
prereqs() {
    echo "$PREREQ"
}
case "$1" in
    prereqs)
        prereqs
        exit 0
    ;;
esac
   
. "${CONFDIR}/initramfs.conf"
. /usr/share/initramfs-tools/hook-functions
   
if [ "${DROPBEAR}" != "n" ] && [ -r "/etc/crypttab" ] ; then
  
# Convert SSH keys
/usr/lib/dropbear/dropbearconvert openssh dropbear /etc/ssh/ssh_host_dsa_key ${DESTDIR}/etc/dropbear/dropbear_dss_host_key
/usr/lib/dropbear/dropbearconvert openssh dropbear /etc/ssh/ssh_host_rsa_key ${DESTDIR}/etc/dropbear/dropbear_rsa_host_key

fi
__EOF__

# Make it executable
chmod a+x /usr/share/initramfs-tools/hooks/dropbear.fixup2

# Shutdown network after exiting initramfs
cat >/usr/share/initramfs-tools/scripts/init-bottom/dropbear.fixup3 <<'__EOF__'
#!/bin/sh
PREREQ=""
prereqs() {
        echo "$PREREQ"
}
case "$1" in
        prereqs)
                prereqs
                exit 0
        ;;
esac
. /scripts/functions

# Shutdown initramfs network before passing control to regular Ubuntu scripts
# Without this network config from initramfs is used forever plus causes extra
# few minutes of delay plus errors on bootup.
ifconfig eth0 0.0.0.0 down
__EOF__

# Make it executable
chmod a+x /usr/share/initramfs-tools/scripts/init-bottom/dropbear.fixup3

# Silence error messages from stock scripts looking wrongly named libs
touch /lib/libnss_

# Prevent root logins over openssh
sed -i.bak /etc/ssh/sshd_config \
    -e's|PermitRootLogin yes|PermitRootLogin no|g'
 
# Set root password
passwd root
 
# Enable static IP as there's no DHCP servers around
sed -i.bak /etc/initramfs-tools/initramfs.conf \
    -e's|^DEVICE=|DEVICE=eth0\nIP=192.168.249.111::192.168.248.1:255.255.248.0:ubuntu:eth0|g'

# Update initramfs and grub
update-initramfs -c -k all
update-grub
 
# DONE!