# Mkosi for Arch Boxes

Deliverables

vagrant blogpost
building in CI
testing mkosi build images! (in CI)
reproducible images
talk!

Questions

how do we make btrfs partition of 40GB for basic
how do I add a user with sudo? => https://github.com/DaanDeMeyer/mkosi/blob/main/mkosi/resources/mkosi.md#frequently-asked-questions-faq
how do I cache the package manager step
how do I create a qcow2 after the final step? => mkosi.postoutput
how do we make profiles?
- profile for basic
- profile for cloud-init
- profile for vagrant
- profile for vagrant-libvirt
do we need to remove /etc/machine-id?
get mkswapfile / btrfs swap working by default => create it on first boot?!
How do we apply sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT=\"rootflags=compress-force=zstd\"/' "${MOUNT}/etc/default/grub"
how do we apply, this is apparently what all cloud images do (cloud-init image) echo 'GRUB_TERMINAL="serial console"' >>"${MOUNT}/etc/default/grub" echo 'GRUB_SERIAL_COMMAND="serial --speed=115200"' >>"${MOUNT}/etc/default/grub"
systemd-firstboot queries with timezone, this is unwanted!!!

Resources

https://btrfs.readthedocs.io/en/latest/Swapfile.html
https://swsnr.de/archlinux-rescue-image-with-mkosi/
https://noise.getoto.net/2024/01/10/a-re-introduction-to-mkosi-a-tool-for-generating-os-images/

Creation

cd projects/arch-boxes/cloud-init-image
mkdir mkosi.{cache,output}
mkosi build

Reproducible

SourceDateEpoch=
Seed=
Pin mirrors to a fixed set. (OR introduce .BUILDINFO, harder imo)
FS allocation might be unreproducible

Some documentation: https://reproducible-builds.org/docs/system-images/ mkosi reproducible: https://github.com/edgelesssys/reproducible-mkosi

Problems

image too big for diffoscope
how to diff an EFI executable, not supported by diffoscope it seems https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/181
split up the problem in smaller steps
- is btrfs swapfile repro
- is mkfs.btrfs repro
- what about ext4

btrfs

mkfs.btrfs is not reproducible.

qemu-img create -f raw test1.img 2G
qemu-img create -f raw test2.img 2G

0414cac598ebfa08e8e9c6d2544aa414385b9985c5d67d7a8746aa64324c715fa96ff63351016d30dd2b89276252c121c71619f15496b5ca95785d0b25fe4dfd  test2.img
0414cac598ebfa08e8e9c6d2544aa414385b9985c5d67d7a8746aa64324c715fa96ff63351016d30dd2b89276252c121c71619f15496b5ca95785d0b25fe4dfd  test1.img

mkfs.btrfs -f -L arch -U 58c7d0b2-d224-4834-a16f-e036322e88f7 test1.img
mkfs.btrfs -f -L arch -U 58c7d0b2-d224-4834-a16f-e036322e88f7 test2.img

Building in container in CI

Can we do this by default?
Try out a gitlab action

CI testing

Can we boot tests in CI
Can we boot test in CI? Check if swap is enabled etc.? Systemd units run? systemctl --failed is green?
Use mkosi ssh?

How do we apply, this is apparently what all cloud images do (cloud-init image) echo 'GRUB_TERMINAL="serial console"' >>"${MOUNT}/etc/default/grub" echo 'GRUB_SERIAL_COMMAND="serial --speed=115200"' >>"${MOUNT}/etc/default/grub"

This is for hosting companies like hetzner if they don't offer a VNC view.

[root@archlinux ~]# grep serial /boot/grub/grub.cfg
serial --speed=115200
terminal_input serial console
terminal_output serial console
[root@archlinux ~]# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=d7d6a869-2230-43de-b3c4-23cb261ac1bd rw net.ifnames=0 console=ttyS0,115200 rootflags=compress-force=zstd console=tty0 console=ttyS0,115200

Current mkosi image

net.ifnames=0 console=tty0 console=ttyS0,115200

Vendor grub.cfg - meh
provide default grub.cfg which mkosi can append too
Somehow sed it in an mkosi script?

Editing /etc/default/grub in mkosi.finalize does not seem to work, which runs before grub

Mkosi issues

Should mkosi qemu read BiosBootLoader options and then use mkosi --qemu-firmware bios qemu

How to exit QEMU? Maybe add ctrl+] support like nspawn? => ctrl+a x in QEMU

modifications in mkosi qemu stay, seems somewhat unexpected

Vagrant images

What is vagrant?
What does it support?
What is its image format?
How do we build this?
How to make this in mkosi?

Intro

Vagrant was a popular way to setup a development virtual machine for projects, all you had to do is provide a Vagrantfile and vagrant up would setup a development VM and for example mount your code into the VM via sshfs.

The image you use in your Vagrantfile are called boxes in Vagrant's terminology and are hosted by the Vagrant project.

Vagrant also supports the concept of providers, these are the backends on which the box would run, the default being virtualbox, alternatives are hyperv, libvirt and more.

Format

Usually the vagrant boxes are made using packer, but other tools such as kiwi, osbuild (only libvirt provider) also support creating these boxes. The alternative tools usually only support a subset of the providers packer supports. In the Linux community virtualbox and libvirt are usually the most popular ones. At least this is what Arch Linux supports.

The base box format is described in a very short summary what needs to be in the image:

Package manager
SSH server enabled and running
A vagrant user with SSH pubkey and password-less sudo
Enough disk space to do interesting things
Provider specific tools (for example virtualbox guest tools)

The image format is provider dependant and is described for Virtualbox and libvirt.

Mkosi

Libvirt

local virtual_size
virtual_size="$(grep -o "^[0-9]*" <<<"${DISK_SIZE}")"
echo '{"format":"qcow2","provider":"libvirt","virtual_size":'"${virtual_size}"'}' >metadata.json
qemu-img convert -f raw -O qcow2 "${1}" box.img
rm "${1}"

tar -czf "${2}" Vagrantfile metadata.json box.img

Presentation

Reproduciblity

Arch Linux is working on making packages 100% reproducible, currently we are ~ 90% reproducible but so far we haven't looked at reproducing our produced images.

The most common reproducibility issues are timestamps, hostnames, and other versioned identifiers (kernel, etc.), locale and build paths can also be recorded in the resulting artificat twarting reproducibility.

What does this mean for making an reproducible images?

Our minimal container image is already reproducible
Installed packages need to reproducible
Kernel notably is not reproducible

The goal is to allow others to reproduce the arch image bit by bit so we have to reduce any variation for this we can:

Pin the package repository of the build image to a fixed date (Arch Linux has support for this using the Archive)
Build an image with the same package set and compare

Problems:

The created cloud image is big!
Diffoscope simply OOM's
We need to reduce the problem scope and tackle things one by one

Unreproducibility

The generated UKI is not reproducible, (a seperate build artifact)
Diffing images is hard, so we work on a directory first (supported output type)

mkosi -d arch -p systemd --format directory -o foo -m https://archive.archlinux.org/repos/2024/06/30/
mkosi -d arch -p systemd --format directory -o bar -m https://archive.archlinux.org/repos/2024/06/30/
sudo diffoscope foo bar --html-dir output
xdg-open output/index.html

Timestamps, timestamps everywhere

export SOURCE_DATE_EPOCH=1662046009
mkosi -d arch -p systemd --format directory --source-date-epoch $SOURCE_DATE_EPOCH -o foo -m https://archive.archlinux.org/repos/2024/06/30/
mkosi -d arch -p systemd --format directory --source-date-epoch $SOURCE_DATE_EPOCH -o bar -m https://archive.archlinux.org/repos/2024/06/30/

This is a lot better!

Look at what mkosi does

Now the leftover issues are:

var/cache/ldconfig/aux-cache - we can opt to remove this post creation (post install script)
var/lib/pacman/local/*

20	%INSTALLDATE%	20	%INSTALLDATE%
21	1722251133	21	1722251028

https://gitlab.archlinux.org/pacman/pacman/-/merge_requests/213

lib/libalpm/add.c

	/* make an install date (in UTC) */
	newpkg->installdate = time(NULL);

sed -i -e '/INSTALLDATE/{n;s/.*/0/}' "${BUILDROOT}"/var/lib/pacman/local/*/desc || true

Now with a cloud image:

mkosi -d arch -p systemd -p linux -p base -p grub -p openssh -p sudo -p reflector -p btrfs-progs -p udev --source-date-epoch 1662046009 --format directory -o bar -m https://archive.archlinux.org/repos/2024/06/30/

This OOM's. on foo/boot arch-6.9.7-arch1-1.efi:

https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/181

Directory containing the UKI

Also unreproducible: bar/efi/loader/random-seed comes from bootctl?

https://www.freedesktop.org/software/systemd/man/latest/bootctl.html#random-seed https://systemd.io/RANDOM_SEEDS/

UKI tools:

llvm-objdump
pe-inspect (virt-firmware)
ukify inspect

Different sector: .initrd

.initrd:
  size: 202577872 bytes
  sha256: 04a3e836f5120c1991dda74285293fe9f7badf2f40602eb4587a9a2b37e8cadf

.initrd:
  size: 202577856 bytes
  sha256: b6cc6d6b62c0bd0e4d108a550bb8b59693fa79184987990d708348b40a316cd9

different:

var/cache/ldconfig

dump2efs test1.img > test1.txt
dump2efs test2.img > test1.txt
diffoscope test1.txt test2.txt

fallocate -l0.1G test1.img
mkfs.ext4 -U c1b9d5a2-f162-11cf-9ece-0020afc76f16 -E hash_seed=a24031c1-fc68-453d-80fa-00ad057a5780  test1.img

mkosi.conf.d/30-debian-ubuntu/mkosi.conf.d/20-ext4-orphan-file.conf:Environment=SYSTEMD_REPART_MKFS_OPTIONS_EXT4="-O ^orphan_file"

Environment=SYSTEMD_REPART_MKFS_OPTIONS_EXT4="-E hash_seed=a24031c1-fc68-453d-80fa-00ad057a5780"

To Do mkosi this!

mkosi --debug -d arch -p systemd --seed 0e9a6fe0-68f6-408c-bbeb-136054d20445 --source-date-epoch 1662046009 -m https://archive.archlinux.org/repos/2024/06/30/ --force -o foo --env-file env  --remove-files var/cache/ldconfig/aux-cache --environment=SYSTEMD_LOG_LEVEL=debug

env

SYSTEMD_REPART_MKFS_OPTIONS_EXT4=-E hash_seed=a24031c1-fc68-453d-80fa-00ad057a5780 -U 6de632fe-7638-44c4-917c-ecf4170af3b4

mkosi --debug -d arch -p systemd -p udev  --seed 0e9a6fe0-68f6-408c-bbeb-136054d20445 --source-date-epoch 1662046009 -m https://archive.archlinux.org/repos/2024/06/30/ --force -o bar --env-file env --remove-files var/cache/ldconfig/aux-cache --environment=SYSTEMD_LOG_LEVEL=debug

But adding -p linux fails, this probably installs a boot loader which is the issue?

The issue is likely making it bootable

mkosi --debug -d arch -p systemd -p udev -p linux --seed 0e9a6fe0-68f6-408c-bbeb-136054d20445 --source-date-epoch 1662046009 -m https://archive.archlinux.org/repos/2024/06/30/ --bootable false --force -o bar --env-file env --remove-files var/cache/ldconfig/aux-cache --environment=SYSTEMD_LOG_LEVEL=debug

vfat reproducible issues:

cfdisk foo.raw
cfdisk bar.raw

Show the same meta-data

Mounting with losetup shows:

losetup --find --show -P bar.raw

Mounting and diffoscope shows difference in

sudo mount /dev/loop1p1 -o ro /tmp/foo
sudo mount /dev/loop2p1 -o ro /tmp/bar

EFI directory and mtimes, they are recent. Unsure if due to mounting +Modify: 2024-08-24 14:23:34.000000000 +0000

/mnt/disk/EFI

[jelle@natrium][~/projects/arch-mkosi-boxes/reproducible/mkosi.output]%mkfs.vfat -i e7b87c0d -n ESP -F 32 -S 4096 test1.img
mkfs.fat 4.2 (2021-01-31)
[jelle@natrium][~/projects/arch-mkosi-boxes/reproducible/mkosi.output]%mkfs.vfat -i e7b87c0d -n ESP -F 32 -S 4096 test2.img
mkfs.fat 4.2 (2021-01-31)
[jelle@natrium][~/projects/arch-mkosi-boxes/reproducible/mkosi.output]%md5sum test1.img test2.img
9d95cb024f4a79a76345de000568dd0e  test1.img
2ccc5c8845667d6afe943c6ce1c6bd40  test2.img

Needs: https://github.com/dosfstools/dosfstools/commit/8da7bc93315cb0c32ad868f17808468b81fa76ec

But after that we are still not reproducible....

But when mounting and diffing the FS it is. So must be some other meta-data

Useful tools:

sfdisk -J foo.raw

0022000E 56 43
00220016 56 43
0022002E 56 43
00220036 56 43
0022004E 56 43
00220056 56 43
0022100E 56 43
00221016 56 43
0022102E 56 43
00221036 56 43
0D07500E 56 43
0D075016 56 43
0D07502E 56 43
0D075036 56 43

So systemd-repart implements creating fat partitions by:

mkfs.vfat
mcopy (mtools) the stuff into the fat partition.

Executing mkfs command: mkfs.vfat -i e7b87c0d -n ESP -F 32 /var/tmp/repart-OWnjo5 -S 4096 Executing mkfs command: mkfs.vfat -i e7b87c0d -n ESP -F 32 /var/tmp/repart-TA8zUX -S 4096

dd if=bar.raw of=bar-boot.raw bs=512 skip=2048 count=1048576 status=progress

Reproducible if I copy out one file, so time to:

Find the list of files copied either debug systemd-repart build or grab from mkosi
Make a reproducer with mcopy / mkfs.vfat

systemd-repart takes CopyFiles, copies the source to a tmpdir var_tmp_dir and then passes this over to mcopy

[Partition]
Type=esp
Format=vfat
# CopyFiles=/boot:/
CopyFiles=/efi:/
SizeMinBytes={"1G" if bios else "512M"}
SizeMaxBytes={"1G" if bios else "512M"}

‣  Normalizing modification times of /boot
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/boot/loader
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/boot/EFI
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/boot/loader/entries.srel
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/boot/loader/entries
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/boot/EFI/Linux
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/boot/EFI/Linux/arch-6.9.7-arch1-1.efi
‣  Normalizing modification times of /efi
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/efi/EFI
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/efi/loader
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/efi/EFI/systemd
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/efi/EFI/BOOT
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/efi/loader/loader.conf
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/efi/EFI/systemd/systemd-bootia32.efi
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/efi/EFI/systemd/systemd-bootx64.efi
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/efi/EFI/BOOT/BOOTIA32.EFI
‣ Normalize mtime of /home/jelle/.cache/mkosi/mkosi-workspace-9gik1art/root/efi/EFI/BOOT/BOOTX64.EFI

Filesystems reproducible

vfat - with patch
ext4 - yes
xfs
btrfs

FAT

fallocate -l512M test1.img
fallocate -l512M test2.img

mkfs.vfat -i e7b87c0d -n ESP -F 32 -S 4096 test1.img
mkfs.vfat -i e7b87c0d -n ESP -F 32 -S 4096 test2.img

TZ=UTC mcopy -p -s -Q -m -i test1.img boot efi :: && echo 1
TZ=UTC mcopy -p -s -Q -m -i test2.img boot efi :: && echo 1

mcopy -s -p -Q -m -i /var/tmp/repart-kAScbM /var/tmp/.#repartafc0df669fc911a5/EFI /var/tmp/.#repartafc0df669fc911a5/loader ::

/var/tmp/.#repart096a80760fee7737/EFI
total 0
drwx------ 1 root root 46 Jan  1  1970 BOOT
drwx------ 1 root root 44 Jan  1  1970 Linux
drwx------ 1 root root 78 Jan  1  1970 systemd
/var/tmp/.#repart096a80760fee7737/loader
total 8.0K
drwx------ 1 root root  0 Jan  1  1970 entries
-rw------- 1 root root  6 Jan  1  1970 entries.srel
-rw-r----- 1 root root 30 Jan  1  1970 loader.conf

drwx------ 1 root root 32 Aug 29 19:59 EFI
drwx------ 1 root root 60 Aug 29 19:59 loader

total 0
drwx------ 1 root root 32 Aug 29 20:02 EFI
drwx------ 1 root root 60 Aug 29 20:02 loader

mcopy -i option not documented

There is an mtools mailing list at info-mtools@gnu.org. Please send all bug reports to this list. Note: You must be subscribed to the list in order to be able to send comments.

BTRFS

if (blk == MKFS_FS_TREE) {
	time_t now = time(NULL);

static int make_root_dir(struct btrfs_trans_handle *trans,
		struct btrfs_root *root)
{
	struct btrfs_key location;
	int ret;

	ret = btrfs_make_root_dir(trans, root->fs_info->tree_root,
			      BTRFS_ROOT_TREE_DIR_OBJECTID);

From: 3362

To: 3300

erofs

Apparently can be made reproducible.

F2FS

fallocate -l 5M test1.img
fallocate -l 5M test2.img

mkfs.f2fs -U 588114f7-e142-40a1-8b99-30db4519183e -f  -r test2.img

[jelle@natrium][~/Downloads/tmp]%~/Downloads/bindiff.sh test1.img test2.img
01000020 56 59
01000028 56 59
01000030 56 59

 1000000 41ed 0000 03e8 0000 03e8 0000 0002 0000
 1000010 1000 0000 0000 0000 0002 0000 0000 0000
-1000020 3b39 66ca 0000 0000 3b39 66ca 0000 0000
-1000030 3b39 66ca 0000 0000 0000 0000 0000 0000
+1000020 3b3b 66ca 0000 0000 3b3b 66ca 0000 0000
+1000030 3b3b 66ca 0000 0000 0000 0000 0000 0000
 1000040 0000 0000 0000 0000 0001 0000 0000 0000
 1000050 0000 0000 0000 0000 0000 0000 0000 0000

-01000020  39 3b ca 66 00 00 00 00  39 3b ca 66 00 00 00 00  |9;.f....9;.f....|
-01000030  39 3b ca 66 00 00 00 00  00 00 00 00 00 00 00 00  |9;.f............|
+01000020  3b 3b ca 66 00 00 00 00  3b 3b ca 66 00 00 00 00  |;;.f....;;.f....|
+01000030  3b 3b ca 66 00 00 00 00  00 00 00 00 00 00 00 00  |;;.f............|

-T 0

Can be made reproducible with:

mkfs.f2fs -T0 -r

 -r set checkpointing seed (srand()) to 0
 -T timestamps

archiso

32 bit EFI
32 bit EFI eltorito
64 bit EFI
64 bit EFI eltorito
BIOS boot
BIOS boot el torito

Notes