Ubuntu on the Raspberry Pi (Part I): Installing Ubuntu 20.04 with Btrfs on root

I have been using Btrfs as the main (and only) filesystem on my Laptop for some time now. Since I like using it, I decided to use it also on my raspberry pi running Ubuntu Server 20.04 LTS. Here is the guide on how I did it.

This is the first of my series on Ubuntu on the raspberry pi. You could also use this guide for other distros, probably with some minor changes. The next two articles will talk about the use of cloud-init for customizing your system; and how to recover using btrfs snapshots.

Why?

Btrfs has a couple of things that I really like. First of all, compression is great and it helps me save space and reduces the I/O to my disks. This is great for flash memory, such as SSDs, USB Sticks and SD Cards. It helps to reduce the wear and tear at the cost of some CPU. However with Zstandar compression (zstd), the penalty is negible.

Snapshots are also an awesome feature of Btrfs. I can create a snapshot before an update and rollback if my system breaks. I also take a snapshot of my container data before upgrading my containers. This has saved my bacon a couple of times, so it’s a nice thing to have.

Some people would say that btrfs is not stable enough because of bad past experiences. However, I’m not using raid, so I cannot judge about it. In my experience and my use case, Btrfs is stable and has been in the mainline Linux Kernel for a long time.

Finally, the main reason I am using it is, because I can and I have some fun doing it.

If you don’t like Btrfs and you would rather use zfs, I think that’s a great idea and you should check this guide from the OpenZFS Documentation

Let’s get to Business

Get the image and install it on an SD Card

Download the official 20.04 image from the Ubuntu website: https://ubuntu.com/download/raspberry-pi.

You can’t boot Ubuntu 20.04 from the USB ¹, it doesn’t matter if you got the latest firmware. So we will be booting from an SD card, but we’ll have the root partition on a USB Stick (or even better an USB SSD)². So let’s decompress the image:

# xzcat ubuntu-20.04.1-preinstalled-server-arm64+raspi.img.xz > /dev/sdb

Preparing the USB Stick with Btrfs

Now let’s prepare our USB Stick for creating our Btrfs root partition. I will be using the command line (because it’s the same for all Linux distros), but you can use a GUI tool such as gparted or partition manager.

# fdisk /dev/sdc

Command (m for help): o

Created a new DOS disklabel with disk identifier 0xXXXXXXXX.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p):↵

Using default response p.
Partition number (1-4, default 1): ↵
First sector (2048-61747199, default 2048): ↵
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-61747199, default 61747199): ↵

Created a new partition 1 of type 'Linux' and of size 29,5 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

# mkfs.btrfs -L btrfs-root /dev/sdc1

The first command opens a partitioning tool called fdisk; it creates a new DOS partition table and a new partition using all the space on the disk.

The second command will format our partition to Btrfs and assign the label “btrfs-root” to it. The label will be useful afterwards.

Mounting our partitions

Now let’s mount the partitions. We will be mounting our Btrfs partition with the compression flag, so all files are compressed when writing them to the disk.

First we create some mount points:

# mkdir /mnt/system-boot
# mkdir /mnt/old-root
# mkdir /mnt/new-root

Then we mount the partitions from the SD Card and the USB Stick:

# mount /dev/disk/by-label/system-boot /mnt/system-boot/
# mount /dev/disk/by-label/writable /mnt/old-root/
# mount /dev/disk/by-label/btrfs-root -o defaults,noatime,compress=zstd:1,autodefrag /mnt/new-root/

For understanding these mount options, we can check the explanations from the wiki of Btrfs³:

Creating the subvolumes

We will create subvolumes for different applications. This will allow us to avoid creating snapshots of unnecesary data (such as cache) and/or avoid the loss of data when doing a rollback. This was inspired by the openSUSE Wiki, Didier Roche blogpost about ZSys datasets layout and the OpenZFS Documentation:

Creating different subvolumes, instead of creating them all inside the root subvolume, allows us to delete it if we need it. If you need an example, when this can be needed, wait for the third part of this series, when we talk about Disaster Recovery.

So now that we know what subvolumes we want, let’s create them:

# btrfs subvolume create /mnt/new-root/\@
# btrfs subvolume create /mnt/new-root/\@home
# btrfs subvolume create /mnt/new-root/\@root
# btrfs subvolume create /mnt/new-root/\@swap # Only if you want to add a Swapfile
# btrfs subvolume create /mnt/new-root/\@usr_local
# btrfs subvolume create /mnt/new-root/\@tmp
# btrfs subvolume create /mnt/new-root/\@var_tmp
# btrfs subvolume create /mnt/new-root/\@var_log
# btrfs subvolume create /mnt/new-root/\@var_mail
# btrfs subvolume create /mnt/new-root/\@var_spool
# btrfs subvolume create /mnt/new-root/\@var_snap
# btrfs subvolume create /mnt/new-root/\@snapshots

There are some other directories that should have their own subvolume to avoid data loss. Here are some examples for important ones, you shold create the ones you need:

You will probably also want to disable compression and Copy-on-Write for some of these directories (specially with databases). Check the section about the Swapfile for more information on how to do it.

You will also need to create subvolumes for all the other directories that you need. For example, the docker ones:

# btrfs subvolume create /mnt/new-root/\@docker
# btrfs subvolume create /mnt/new-root/\@var_lib_docker

After we created the subvolumes we need to remount them in the proper locations. For doing this we will need to create mount points inside the root. We will specify the compression and autodefrag only on the first mounted subvolume¹¹:

# umount /mnt/new-root
# mount /dev/disk/by-label/btrfs-root -o defaults,noatime,compress=zstd:1,autodefrag,subvol=\@ /mnt/new-root
# # In the next step we create the mount points for our subvolumes
# mkdir /mnt/new-root/{home,root,swap,usr,usr/local,tmp,var,var/tmp,var/log,/var/snap,/var/mail,/var/spool,.snapshots}
# mkdir /mnt/new-root/{var/lib{,/docker},opt/{,/docker}} # This are my optional ones
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@home /mnt/new-root/home/
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@root /mnt/new-root/root/
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@swap /mnt/new-root/swap/
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@usr_local /mnt/new-root/usr/local/
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@tmp /mnt/new-root/tmp
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@var_tmp /mnt/new-root/var/tmp
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@var_log /mnt/new-root/var/log/
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@var_mail /mnt/new-root/var/mail/
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@var_spool  /mnt/new-root/var/spool/
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@var_snap /mnt/new-root/var/snap/
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@snapshots /mnt/new-root/.snapshots/

I will also mount my optional ones, however in my case I won’t actually need them:

# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@var_lib_docker /mnt/new-root/var/lib/docker
# mount /dev/disk/by-label/btrfs-root -o noatime,subvol=\@docker /mnt/new-root/opt/docker/

And then we check that everything mounted properly:

# mount | grep /mnt
/dev/sdc1 on /mnt/system-boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
/dev/sdc2 on /mnt/old-root type ext4 (rw,relatime)
/dev/sdb1 on /mnt/new-root type btrfs (rw,noatime,compress=zstd:1,space_cache,autodefrag,subvolid=264,subvol=/@)
/dev/sdb1 on /mnt/new-root/home type btrfs (rw,noatime,compress=zstd:1,space_cache,autodefrag,subvolid=265,subvol=/@home)
/dev/sdb1 on /mnt/new-root/.snapshots type btrfs (rw,noatime,compress=zstd:1,space_cache,autodefrag,subvolid=266,subvol=/@snapshots)
[...]

The places are the ones, that we specified, and all the flags (compression, autodefrag, noatime, etc.) are those, that we wanted, so we can continue to the next step.

Creating a swapfile (OPTIONAL)

By default, Ubuntu server on the Raspberry Pi doesn’t use a swapfile or a swap partition. It’s too much I/O for SD Cards, and it shortens the lifespan of your SSD or Flash Drive. However, if you don’t have one, your system will most likely crash, if it encounters too much memory pressure and it’s unable to kill processes in time. I would rather err on the side of caution.

You may also ask “Why not use a swap partition instead of a file?”, and the reasons for me are simple:

• I haven’t seen a difference between a swapfile and a swap partition.
• I can easily remove the swapfile if I think that I don’t need it anymore.
• Swapfiles are supported on Btrfs since kernel 5.0 with some small caveats¹².
• It makes it easier to write this guide, because if you don’t want a swapfile you can ignore this step. If you want a swap partition you need to change the whole partitioning of the disk.

So, with that cleared, let’s create our swapfile:

# touch /mnt/new-root/swap/swapfile # Creates a new file called swapfile
# chattr +C /mnt/new-root/swap/swapfile # Sets No Copy-on-Write for the file
# btrfs property set /mnt/new-root/swap/swapfile compression none # Disables compression
# fallocate --length 2GiB /mnt/new-root/swap/swapfile # Expand the file to 2 GiB
# chmod 600 /mnt/new-root/swap/swapfile # Make sure that only root can read our swap
# mkswap /mnt/new-root/swap/swapfile # Format as swap

In the end, I will set the swappiness to 20, instead of the default 60. This will help to avoid too much writing onto the swap.

Now that we have what what we needed, we can move the files into our new filesystem.

Copying the files into the new root

After we mounted everything in place, we need to copy all the files from the SD card to our Btrfs filesystem. We cannot copy and paste from the GUI because it could introduce many problems. For doing this, we use our well known rsync:

# rsync --archive --verbose --hard-links --whole-file /mnt/old-root/ /mnt/new-root/

If you want the shorter version:

#  rsync -avHW /mnt/old-root/ /mnt/new-root/

If you want to understand, what each of these flags does, before blindly copying and pasting¹³, you should check explainshell.com.

Modifying the fstab

Since we changed the basic structure of our file system, we will need a new fstab, that explains where each of the partitions and subvolumes should be mounted. In my gitlab repo you can find a well commented example. You should just download the repo (or copy and paste the parts that you need in a new file), modify it, and then copy it into your new-root.

You will at least need the part titled Boot partition and the Root filesystem one. You could mix and match the files system that you added, and there is also an example for an external NTFS formated disk.¹⁴.

If you need some help writing the fstab, you can check the articles about it in How to geek and Red Hat Documentation, and of course the man page for fstab.

After your are satisfied with your new fstab we copy it to the new filesystem:

#  mv fstab.example /mnt/new-root/etc/fstab

Changing the cmdline arguments

The raspberry pi uses a file called cmdline to store the Kernel Command Line arguments. Since we changed the filesystem that our pi will boot on, we need to modify these arguments:

root=LABEL=writable rootfstype=ext4

And change them for these ones:

root=LABEL=btrfs-root rootfstype=btrfs rootflags=subvol=@

What we did is change the LABEL of the partition, specify the filesystem type (rootfstype) and we supply the name of the subvolume that is used for the root filesystem.

If you want you can just use the cmdline in my repo¹⁵:

# cp cmdline.txt /mnt/system-boot/

After this your set up is completed and you should be able to boot into your system. However, it may be a good idea to take a snapshot beforehand if you need to go back:

# btrfs subvolume snapshot -r /mnt/new-root/ /mnt/new-root/.snapshots/before-first-boot

The -r flag will mark the snapshot as read-only.

Unmounting and first boot

After all this steps, our Pi is ready and we will be able to boot into it.

First we need to unmount the SD Card and our USB disk:

# umount /mnt/new-root/{var/lib/docker,opt/docker} # This are my optional ones
# umount /mnt/new-root/{home,root,swap,usr/local,tmp,var/tmp,var/log,/var/snap,/var/mail,/var/spool,.snapshots,}
# umount /mnt/old-root /mnt/system-boot

After that you can just connect your USB-Disk (our USB-Stick) to your raspberry pi, put the SD card in the correct slot. Connect it to power and voilà: You have Ubuntu 20.04, with BTRFS on root, running on your raspberry pi 4.

If you want to configure your system a little bit more before the first boot, you should wait until next week, when I publish how to use cloud-init to customize your system.

Final Words

I hope you like this guide and come back for the next one. If you find it useful, I would like to hear from you. Kind words are always appreciated.