Reproducible Builds

Rebuilderd-website re-design

  • Configurable
  • Themes (Debian, Fedora)
  • Per package page
  • Recent builds page

Current Debian frontend and current Debian rebuilder.

Last 24h, 48h builds

  {
    "name": "zziplib-bin",
    "version": "0.13.72+dfsg.1-1.3",
    "status": "BAD",
    "distro": "debian",
    "suite": "main",
    "architecture": "amd64",
    "artifact_url": "http://deb.debian.org/debian/pool/main/z/zziplib/zziplib-bin_0.13.72+dfsg.1-1.3_amd64.deb",
    "build_id": 79239,
    "built_at": "2024-11-11T01:52:55.033859620",
    "has_diffoscope": false,
    "has_attestation": false
  },

  {
    "name": "zzuf",
    "version": "0.15-3",
    "status": "GOOD",
    "distro": "archlinux",
    "suite": "extra",
    "architecture": "x86_64",
    "artifact_url": "https://geo.mirror.pkgbuild.com/extra/os/x86_64/zzuf-0.15-3-x86_64.pkg.tar.zst",
    "build_id": 459906,
    "built_at": "2023-05-23T12:37:43.509033888",
    "has_diffoscope": false,
    "has_attestation": true
  }

Arch

Python issues

For pyc differences PYTHONHASHSEED can be set to a fixed value to try and circumvent the random hash initialisation getting embedded in pyc files

For test files being show in the diffoscope results as pyc files and not in the rebuild package the issue is probably that pyc files generated by running tests are installed errorsnly. Exporting PYTHONDONTWRITEBYTECODE=1 when running the tests.

sphinx issue

sphinx-build also installs a environment.pickle file which is not reproducible and not needed in a package. A fix is to override SPHINXOPTS or alternatively extend our reproducible makepkg hooks for this?

/usr/share/makepkg/reproducible/python.sh

[jelle@t14s][~/projects/reproducible-website]%pacman -F environment.pickle
extra/dleyna-docs 0.8.2-2
    usr/share/doc/dleyna/.doctrees/environment.pickle
extra/ghc-static 9.0.2-3
    usr/share/doc/ghc/html/haddock/.build-html/.doctrees/environment.pickle
    usr/share/doc/ghc/html/haddock/.doctrees/environment.pickle
extra/libcamera-docs 0.1.0-2
    usr/share/doc/libcamera/html/.doctrees/environment.pickle
extra/python-eventlet 0.38.0-1
    usr/share/doc/python-eventlet/html/.doctrees/environment.pickle
extra/python-generic 1.1.3-3
    usr/share/doc/python-generic/html/.doctrees/environment.pickle
extra/python-uproot-docs 5.5.1-3
    usr/share/doc/python-uproot/.doctrees/environment.pickle

Man page gzip timestamp issue

Fixing all the gzip timestamp issue packages is a lot of work and patching upstream everywhere is not really doable. An idea might be to detect gzip files which are non-reproducible and let a makepkg option like zipman or extend zipman to take care of this.

touch foo
gzip foo
file bar.gz | grep modified &>/dev/null  && gunzip -c bar.gz | gzip -9 -n -c > test.gz

Haskell packages

GHC is reproducible when building with -j1, but for Arch this is a very noticable slowdown in package building. There is an open GHC issue about reprodiciblity and recently a potential fix was merged into GHC.

Handling irreproducibility

Write a makepkg hook for add-determinism

  • how does fedora run it
  • how would we integrate it
  • test it a package with unreproducible gz

Fedora

The reproducing script requires:

  • python3-koji
  • python3-requests
  • the user running it must be in the mock group (not relevant as rebuilderd-worker runs as root)

It seems to need more as you still get the following error to fix it I just installed fedpkg

koji.ConfigurationError: no configuration for profile name: koji

rebuilderd deps:

  • sqlite-devel
  • sqlite3
  • libzstd-devel

Rebuilder work

  • get rebuilderd-worker going with koji_rebuild of zbyszek
  • support releases
  • cleanup commits
  • make comparison work with rpm's, as we can't do straight binary comparisons
  • switch to Fedora rawhide for rebuilding
  • submit a PR to symlink comparison.json and the build rpm into $REBUILDERD_OUTDIR
  • package rebuilderd in a copr
  • ansible setup for rebuilderd
  • koji cache cleanup? for rebuilderd-worker
  • investigate postgresql <-> rebuilderd

BUILDTIME in header

│  IPv6 (default) host.
│ -BUILDTIME: 1738781093
│ +BUILDTIME: 1721227464

https://github.com/rpm-software-management/mock/issues/692 - clamp timestamps https://github.com/rpm-software-management/rpm/pull/1532 - build info file

Setting up a Rebuilder

No rebuilderd package (yet) for Fedora, so setup is manual.

The signup secret is generated with pwgen -1s 32

/etc/rebuilderd.conf

[http]
bind_addr = "0.0.0.0:8484"

[worker]
# set the generated secret for our workers here
signup_secret = ""

[schedule]
# 1 week
retry_delay_base = 168

/etc/rebuilderd-sync.conf

[profile."fedora-Everything"]
distro = "fedora"
suite = "Everything"
architectures = ["x86_64"]
source = "https://ftp.halifax.rwth-aachen.de/fedora/linux/releases/41/"
pkgs = ["joe", "nano", "cockpit*", "3mux", "6tunnel", "cmatrix", "mythes*", "python3-b*", "vim*", "pcre*", "neovim*", "dconf", "sssd*", "osinfo*", "redhat*", "fedora*"]

/etc/rebuilderd-worker.conf

endpoint = "http://localhost:8484"
signup_secret = ""

[build]
timeout = 3600

[diffoscope]
enabled = true
max_bytes = 10485760 # 10 MiB

[backend."fedora"]
path = "/usr/libexec/rebuilderd/rebuilder-fedora.sh"

/usr/libexec/rebuilderd/rebuilder-fedora.sh

#!/bin/sh

set -xe

rpmfile="${1}"
# extract nvr
nvr=$(rpm -qp --queryformat '%{NAME}-%{VERSION}-%{RELEASE}' ${rpmfile})

koji_rebuild.py ${nvr}

Testing the sync job:

in rebuilderd/tools

cargo run -- pkgs sync --release 41 fedora Everything https://ftp.halifax.rwth-aachen.de/fedora/linux/releases --architecture x86_64 --print-json

Flatpak

https://fedoramagazine.org/an-introduction-to-fedora-flatpaks/ https://blogs.gnome.org/mclasen/2018/07/07/flatpak-making-contribution-easy/ https://ranfdev.com/blog/flatpak-builds-are-not-reproducible/ https://github.com/flatpak/flatpak-builder/issues/251 https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues/1320

  • diffoscope support?
  • CI on flathub repositories?
  • reproducing

Diffing a flatpak

For Cockpit, comparing the build dir output

flatpak-builder --disable-cache  --disable-rofiles-fuse --force-clean flatpak-build-dir1  org.cockpit_project.CockpitClient.yml
flatpak-builder --disable-cache  --disable-rofiles-fuse --force-clean flatpak-build-dir2  org.cockpit_project.CockpitClient.yml
diffoscope flatpak-build-dir1 flatpak-build-dir2

Comparing using two repos:

flatpak-builder --repo=repo1 --disable-cache  --disable-rofiles-fuse --force-clean flatpak-build-dir  org.cockpit_project.CockpitClient.yml

flatpak-builder --repo=repo2 --disable-cache  --disable-rofiles-fuse --force-clean flatpak-build-dir  org.cockpit_project.CockpitClient.yml

Get the refs from ostree:

ostree refs --repo=repo1

ostree show --repo=repo1 runtime/org.cockpit_project.CockpitClient.Debug/x86_64/devel
ostree show --repo=repo2 runtime/org.cockpit_project.CockpitClient.Debug/x86_64/devel

Confirm the ContentChecksum is the same.

live iso

Reproducible live iso

Issues

  • hugin - gzip timestamps
  • pcp - gzip timestamp
  • libkolabxml XML ordering https://git.kolab.org/T2642 https://bugzilla.opensuse.org/show_bug.cgi?id=1060506 try to set XERCES_DEBUG_SORT_GRAMMAR, but that needs to be in xerces-c which is kinda untested and dumb
  • musescore https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/musescore3.html
  • php phar timestamps
  • dosemu timestamps
  • echoping hostname
  • python-lxml-docs timestamp in "Generated On"
  • ant-doc javadoc adds timestamp to documentation. Generated by javadoc (14.0.2) on Sun Nov 15 16:33:44 UTC 2020
  • nethack build date
  • python-lxml-docs timestamp in generated docs
  • glhack - timestamp
  • i7z - gzip timestamp
  • v2ray-domain-list-community - geosite.dat not ordered
  • libcec - hostname/timestamp
  • hevea - ocaml build /tmp/$tmp path differs https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786913
  • mari0 - zip file
  • ibus - date
  • argyllcms - (date) - https://www.freelists.org/list/argyllcms send email about created date containing hours/minutes/second and SOURCE_DATE_EPOCH
  • deepin-wallpapers => most likely order issue with the wildcard in the makefile nope, most likely image-blur is not reproducible

Ideas

Package pacman in Debian

 -> sudo pbuilder create
 -> sudo cowbuilder create
 -> sudo gbp buildpackage --git-ignore-new --git-pbuilder -nc

Java JAR reproducibility

gradle maven

Rebuilderd

Rebuilderd doesn't clean up old builds, to remove all builds which are no longer references to a package:

delete from builds where id not in (select build_id from packages where build_id is not null);

Rebuilderd also stores logs for succeeded builds which isn't required.

Requeue'ing bad builds can be done as following:

rebuildctl pkgs requeue --suite core --status BAD

Improvements

  • add build date to output of rebuildctl pkgs ls --status BAD --suite core
  • add build date to the /log output
  • add build host to the /log output (so one can identify if a host has a bad build env)
  • add a cleanup thread that runs occasionally cleaning up old rebuild results.

Autoclassify script

Make an autoclassify script based on the diffoscope html output stored in rebuilderd. Maybe using the rebuilderd database for now => extract the diffoscope html and inspiration drawn from this script