Reproducible Builds
Fedora DevConf notes
- website, overview
- fedora-repro-build -> use new mock API
- diffoscope -> filter out stuff of the rpm header
- shared cache location for fedora-repro-build
- rebuilderd -> diffoscope color output
- rebuilderd -> DynamicUser can own /var/lib/rebuilderd, how to solve the config file issue?
- upload failed packages to s3 with an expiration -> in the log
- change rebuilderd-worker to diffoscope --html /dev/stdout.
https://gitlab.haskell.org/ghc/ghc/-/blob/master/utils/haddock/haddock-api/src/Haddock/Backends/Hyperlinker/Renderer.hs#L253
Rebuilderd-website re-design
- Configurable -> config.json file on load?
- Themes (Debian, Fedora)
- Per package page
- Recent builds page
- Fully CSS, no SASS
- Use cockpit's branding.css approach?
Current Debian frontend and current Debian rebuilder.
Last 24h, 48h builds
{
"name": "zziplib-bin",
"version": "0.13.72+dfsg.1-1.3",
"status": "BAD",
"distro": "debian",
"suite": "main",
"architecture": "amd64",
"artifact_url": "http://deb.debian.org/debian/pool/main/z/zziplib/zziplib-bin_0.13.72+dfsg.1-1.3_amd64.deb",
"build_id": 79239,
"built_at": "2024-11-11T01:52:55.033859620",
"has_diffoscope": false,
"has_attestation": false
},
{
"name": "zzuf",
"version": "0.15-3",
"status": "GOOD",
"distro": "archlinux",
"suite": "extra",
"architecture": "x86_64",
"artifact_url": "https://geo.mirror.pkgbuild.com/extra/os/x86_64/zzuf-0.15-3-x86_64.pkg.tar.zst",
"build_id": 459906,
"built_at": "2023-05-23T12:37:43.509033888",
"has_diffoscope": false,
"has_attestation": true
}
Arch
Python issues
For pyc differences PYTHONHASHSEED can be set to a fixed value to try and circumvent the random hash initialisation getting embedded in pyc files
For test files being show in the diffoscope results as pyc files and not in the rebuild package the issue is probably that pyc files generated by running tests are installed errorsnly. Exporting PYTHONDONTWRITEBYTECODE=1 when running the tests.
sphinx issue
sphinx-build also installs a environment.pickle file which is not reproducible and not needed in a package. A fix is to override SPHINXOPTS or alternatively extend our reproducible makepkg hooks for this?
/usr/share/makepkg/reproducible/python.sh
[jelle@t14s][~/projects/reproducible-website]%pacman -F environment.pickle
extra/dleyna-docs 0.8.2-2
usr/share/doc/dleyna/.doctrees/environment.pickle
extra/ghc-static 9.0.2-3
usr/share/doc/ghc/html/haddock/.build-html/.doctrees/environment.pickle
usr/share/doc/ghc/html/haddock/.doctrees/environment.pickle
extra/libcamera-docs 0.1.0-2
usr/share/doc/libcamera/html/.doctrees/environment.pickle
extra/python-eventlet 0.38.0-1
usr/share/doc/python-eventlet/html/.doctrees/environment.pickle
extra/python-generic 1.1.3-3
usr/share/doc/python-generic/html/.doctrees/environment.pickle
extra/python-uproot-docs 5.5.1-3
usr/share/doc/python-uproot/.doctrees/environment.pickle
Man page gzip timestamp issue
Fixing all the gzip timestamp issue packages is a lot of work and patching
upstream everywhere is not really doable. An idea might be to detect gzip files which are non-reproducible and let a makepkg option like zipman or extend zipman to take care of this.
touch foo
gzip foo
file bar.gz | grep modified &>/dev/null && gunzip -c bar.gz | gzip -9 -n -c > test.gz
Haskell packages
GHC is reproducible when building with -j1, but for Arch this is a very noticable slowdown in package building. There is an open GHC issue about reprodiciblity and recently a potential fix was merged into GHC.
Handling irreproducibility
Write a makepkg hook for add-determinism
- how does fedora run it
- how would we integrate it
- test it a package with unreproducible gz
Fedora
Slides from 2024 about the work done.
The reproducing script requires:
- python3-koji
- python3-requests
- koji
- mock?
- rpmbuild
- the user running it must be in the
mockgroup (not relevant as rebuilderd-worker runs as root)
It seems to need more as you still get the following error to fix it I just installed fedpkg
koji.ConfigurationError: no configuration for profile name: koji
rebuilderd deps:
- sqlite-devel
- sqlite3
- libzstd-devel
Rebuilder work
flowchart TB
subgraph coordinator
rebuilderd-sync.timer -- new packages -->rebuilderd.service
end
subgraph worker
rebuilderd-worker -- rebuild request / post result -->rebuilderd.service
rebuilderd-worker -- rebuild rpm -->koji-rebuild
end
- get rebuilderd-worker going with koji_rebuild of zbyszek
- support releases
- cleanup commits
- make comparison work with rpm's, as we can't do straight binary comparisons
- switch to Fedora rawhide for rebuilding
- submit a PR to symlink comparison.json and the build rpm into $REBUILDERD_OUTDIR
-
package rebuilderd in a copr
- rebase rebuilderd patches on the latest release
- config files owned by user without hardcoding uid %attr(mode, user, group) in %files?
- sysusers
- completions
- man pages
- failing decompression tests
- blogpost about setting up a rebuilderd with Fedora
- classify issues
- ansible setup for rebuilderd
- koji cache cleanup? for rebuilderd-worker
- investigate postgresql <-> rebuilderd
koji rebuild
koji-rebuild fixes
/var/lib/rebuilderd-worker/repro1/cache
- info -> DiskCache -> cache of koji build info
- build -> builds
- rpms -> cache of rpm's
/var/lib/rebuilderd-worker/repro1/cache/build/0xFFFF-0.10-13.fc42
- mock.cfg - can be dropped
- rebuild (output)
- 0xFFFF-0.10-13.fc42.src.rpm
- 0xFFFF-0.10-13.fc42.x86_64.rpm
- build.log
- comparison.json
- hw_info.log
- installed_pkgs.log
- root.log
- state.log
- repo (dependencies repository), symlinks to cache, can be dropped
classify issues
- dll related issues - /usr/lib64/ayatana-appindicator3-sharp-0.1/policy.0.0.ayatana-appindicator3-sharp.dll, ..5........ /usr/x86_64-w64-mingw32/sys-root/mingw/bin/libvirt-gobject-1.0-0.dll
- krb5 - ..5........ /usr/share/krb5-tests/x86_64/config.log
- lapack and kiwi packages, no useful rpmdiff output, something with PROVIDES
- /lib64 versus /lib - S.5........ /usr/share/pkgconfig/libkdtree++.pc
- R-libSBML-5.20.4-6.fc43.x86_64.rpm
- intel-sgx
- mock errors:
- libfido2-1.15.0-3.fc42: marking rebuild as failed: Mock result 60 => chroot is locked -> requeue
- 30 - Package manager emitted an error of some sort
..5........ /usr/lib64/R/library/libSBML/DESCRIPTION
..5........ /usr/lib64/R/library/libSBML/Meta/package.rds
.M......... /usr/share/pccsadmin/lib/intelsgx
..5........ /usr/lib64/libmpa_network.so.1.22.100.1
": ""}[root@fedora-repro-build ~]# cat cache/build/lapack-3.12.0-8.fc42/rebuild/comparison.json | jq
{
"lapack-3.12.0-8.fc42.src": "added PROVIDES lapack64 = 3.12.0-8.fc42\nadded PROVIDES blas64 = 3.12.0-8.fc42\nadded PROVIDES lapack64_ = 3.12.0-8.fc42\nadded PROVIDES blas64_ = 3.12.0-8.fc42\n",
}
removed PROVIDES debuginfo(build-id) = b9c96524e3ed998af43ec16a2f1da7faa801dae2
added PROVIDES debuginfo(build-id) = dccdf49f1ce636a3ea88ff2677750b2711e51ee9
removed /usr/lib/debug/.build-id/b9
removed /usr/lib/debug/.build-id/b9/c96524e3ed998af43ec16a2f1da7faa801dae2
removed /usr/lib/debug/.build-id/b9/c96524e3ed998af43ec16a2f1da7faa801dae2.debug
added /usr/lib/debug/.build-id/dc
added /usr/lib/debug/.build-id/dc/cdf49f1ce636a3ea88ff2677750b2711e51ee9
added /usr/lib/debug/.build-id/dc/cdf49f1ce636a3ea88ff2677750b2711e51ee9.debug
..5........ /usr/lib/debug/usr/lib64/libgnatcoll_zlib.so.25.0.0-25.0.0-3.fc42.x86_64.debug
libkdtree++-devel-0.7.0-42.fc42.noarch.rpm
├── content
│ ├── file list
│ │ @@ -8,8 +8,8 @@
│ │ drwxr-xr-x 1 0 0 0 2025-01-17 00:00:00.000000 ./usr/share/doc/libkdtree++-devel
│ │ -rw-r--r-- 1 0 0 137 2008-12-30 13:06:36.000000 ./usr/share/doc/libkdtree++-devel/AUTHORS
│ │ -rw-r--r-- 1 0 0 9232 2008-12-30 13:06:36.000000 ./usr/share/doc/libkdtree++-devel/COPYING
│ │ -rw-r--r-- 1 0 0 9842 2008-12-30 13:06:36.000000 ./usr/share/doc/libkdtree++-devel/ChangeLog
│ │ -rw-r--r-- 1 0 0 1367 2008-12-30 13:06:36.000000 ./usr/share/doc/libkdtree++-devel/NEWS
│ │ -rw-r--r-- 1 0 0 4520 2008-12-30 13:06:36.000000 ./usr/share/doc/libkdtree++-devel/README
│ │ -rw-r--r-- 1 0 0 248 2008-12-30 13:06:36.000000 ./usr/share/doc/libkdtree++-devel/TODO
│ │ --rw-r--r-- 1 0 0 216 2025-01-17 00:00:00.000000 ./usr/share/pkgconfig/libkdtree++.pc
│ │ +-rw-r--r-- 1 0 0 218 2025-01-17 00:00:00.000000 ./usr/share/pkgconfig/libkdtree++.pc
│ ├── ./usr/share/pkgconfig/libkdtree++.pc
│ │ @@ -1,10 +1,10 @@
│ │ prefix=/usr
│ │ exec_prefix=/usr
│ │ -libdir=/usr/lib
│ │ +libdir=/usr/lib64
│ │ includedir=/usr/include
│ │
│ │ Name: libkdtree++
│ │ Description: C++ template container implementation of kd-tree sorting.
│ │ Version: 0.7.0
│ │ Libs: -L${libdir}
│ │ Cflags: -I${includedir}
S.5........ /usr/share/doc/ghc/html/libraries/pandoc-lua-marshal-0.3.0/src/Text.Pandoc.Lua.Walk.html
rc - timestamp
│ │ │ - 0x000122b0 20312e37 2e342032 3032352d 30322d31 1.7.4 2025-02-1
│ │ │ + 0x000122b0 20312e37 2e342032 3032352d 30352d32 1.7.4 2025-05-2
- rpmdiff cache/rpms/rcssserver3d-0.7.6-7.fc42/rcssserver3d-devel-0.7.6-7.fc42.noarch.rpm cache/build/rcssserver3d-0.7.6-7.fc42/rebuild/rcssserver3d-devel-0.7.6-7.fc42.noarch.rpm S.5........ /usr/include/rcssserver3d/rcssserver3d_config.h
Qt apps, json files, seems to be all kernel(!)
S.5........ /usr/lib64/qt6/modules/RemoteObjects.json
or
S.5........ /usr/lib64/qt6/modules/SvgWidgets.json
or
S.5........ /usr/lib64/qt6/modules/NetworkAuth.json
│ ├── usr/lib/qt6/modules/Quick3DPhysics.json
│ │ ├── Pretty-printed
│ │ │ @@ -7,14 +7,14 @@
│ │ │ "name": "Linux",
│ │ │ "targets": [
│ │ │ {
│ │ │ "abi": "x86_64-little_endian-lp64",
│ │ │ "architecture": "x86_64"
│ │ │ }
│ │ │ ],
│ │ │ - "version": "6.13.6-arch1-1"
│ │ │ + "version": "6.13.7-arch1-1"
grub platform bug https://gitlab.archlinux.org/archlinux/packaging/packages/grub/-/issues/3
openctm, documentation irreproducible S.5........ /usr/share/doc/OpenCTM-doc/APIReference.pdf
rasdaemon
│ │ --rw-r-xr-x 1 0 0 1620 2023-01-26 01:59:41.000000 ./etc/sysconfig/rasdaemon
│ │ +-rw-r--r-- 1 0 0 1620 2023-01-26 01:59:41.000000 ./etc/sysconfig/rasdaemon
.M......... /usr/share/licenses/python3-python-fcl/fcl-LICENSE
METADATA file not ordered rasterio-1.4.3
gimagereader-gtk-3.4.2-7.fc43.x86_64.rpm -> appdata -> xml
https://fedora-reproducible.ovh/api/v0/builds/16194/diffoscope
Ignore source rpm?
- Grep through all logs and get the ones with rpmdiff output ie. "......"
- A lot of logs without useful diffoscope or rpmdiff output see lapack
ocaml issues
--rw-r--r-- root root /usr/share/man/man1/alt-ergo.1.gz
+-rw-rw-r-- root root /usr/share/man/man1/alt-ergo.1.gz
--rw-r--r-- root root /usr/share/doc/alt-ergo/examples/invalid/bitv.why
--rw-r--r-- root root /usr/share/doc/alt-ergo/examples/valid/bitv.why
+-rw-rw-r-- root root /usr/share/doc/alt-ergo/examples/invalid/bitv.why
+-rw-rw-r-- root root /usr/share/doc/alt-ergo/examples/valid/bitv.why
BUILDTIME in header
│ IPv6 (default) host.
│ -BUILDTIME: 1738781093
│ +BUILDTIME: 1721227464
https://github.com/rpm-software-management/mock/issues/692 - clamp timestamps https://github.com/rpm-software-management/rpm/pull/1532 - build info file
Setting up a Rebuilder
No rebuilderd package (yet) for Fedora, so setup is manual.
The signup secret is generated with pwgen -1s 32
/etc/rebuilderd.conf
[http]
bind_addr = "0.0.0.0:8484"
[worker]
# set the generated secret for our workers here
signup_secret = ""
[schedule]
# 1 week
retry_delay_base = 168
/etc/rebuilderd-sync.conf
[profile."fedora-rawhide"]
distro = "fedora"
suite = "Everything"
architectures = ["x86_64"]
source = "https://ftp.halifax.rwth-aachen.de/fedora/linux/releases/41/"
pkgs = ["joe", "nano", "cockpit*", "3mux", "6tunnel", "cmatrix", "mythes*", "python3-b*", "vim*", "pcre*", "neovim*", "dconf", "sssd*", "osinfo*", "redhat*", "fedora*"]
/etc/rebuilderd-worker.conf
endpoint = "http://localhost:8484"
signup_secret = ""
[build]
timeout = 3600
[diffoscope]
enabled = true
max_bytes = 10485760 # 10 MiB
[backend."fedora"]
path = "/usr/libexec/rebuilderd/rebuilder-fedora.sh"
/usr/libexec/rebuilderd/rebuilder-fedora.sh
#!/bin/sh
set -xe
rpmfile="${1}"
# extract nvr
nvr=$(rpm -qp --queryformat '%{NAME}-%{VERSION}-%{RELEASE}' ${rpmfile})
koji_rebuild.py ${nvr}
Testing the sync job:
in rebuilderd/tools
cargo run -- pkgs sync --release 41 fedora Everything https://ftp.halifax.rwth-aachen.de/fedora/linux/releases --architecture x86_64 --print-json
cargo run -- pkgs sync --release rawhide fedora Everything https://ftp.halifax.rwth-aachen.de/fedora/linux/development --architecture x86_64 --print-json
Flatpak
https://fedoramagazine.org/an-introduction-to-fedora-flatpaks/ https://blogs.gnome.org/mclasen/2018/07/07/flatpak-making-contribution-easy/ https://ranfdev.com/blog/flatpak-builds-are-not-reproducible/ https://github.com/flatpak/flatpak-builder/issues/251 https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues/1320
- diffoscope support?
- CI on flathub repositories?
- reproducing
Diffing a flatpak
For Cockpit, comparing the build dir output
flatpak-builder --disable-cache --disable-rofiles-fuse --force-clean flatpak-build-dir1 org.cockpit_project.CockpitClient.yml
flatpak-builder --disable-cache --disable-rofiles-fuse --force-clean flatpak-build-dir2 org.cockpit_project.CockpitClient.yml
diffoscope flatpak-build-dir1 flatpak-build-dir2
Comparing using two repos:
flatpak-builder --repo=repo1 --disable-cache --disable-rofiles-fuse --force-clean flatpak-build-dir org.cockpit_project.CockpitClient.yml
flatpak-builder --repo=repo2 --disable-cache --disable-rofiles-fuse --force-clean flatpak-build-dir org.cockpit_project.CockpitClient.yml
Get the refs from ostree:
ostree refs --repo=repo1
ostree show --repo=repo1 runtime/org.cockpit_project.CockpitClient.Debug/x86_64/devel
ostree show --repo=repo2 runtime/org.cockpit_project.CockpitClient.Debug/x86_64/devel
Confirm the ContentChecksum is the same.
live iso
Reproducible live iso
Issues
- hugin - gzip timestamps
- pcp - gzip timestamp
- libkolabxml XML ordering https://git.kolab.org/T2642 https://bugzilla.opensuse.org/show_bug.cgi?id=1060506 try to set XERCES_DEBUG_SORT_GRAMMAR, but that needs to be in xerces-c which is kinda untested and dumb
- musescore https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/musescore3.html
- php phar timestamps
- dosemu timestamps
- echoping hostname
- python-lxml-docs timestamp in "Generated On"
-
ant-doc javadoc adds timestamp to documentation.
Generated by javadoc (14.0.2) on Sun Nov 15 16:33:44 UTC 2020 - nethack build date
- python-lxml-docs timestamp in generated docs
- glhack - timestamp
- i7z - gzip timestamp
- v2ray-domain-list-community - geosite.dat not ordered
- libcec - hostname/timestamp
- hevea - ocaml build /tmp/$tmp path differs https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786913
- mari0 - zip file
- ibus - date
- argyllcms - (date) - https://www.freelists.org/list/argyllcms send email about created date containing hours/minutes/second and SOURCE_DATE_EPOCH
- deepin-wallpapers => most likely order issue with the wildcard in the makefile nope, most likely image-blur is not reproducible
Ideas
Package pacman in Debian
-> sudo pbuilder create
-> sudo cowbuilder create
-> sudo gbp buildpackage --git-ignore-new --git-pbuilder -nc
Java JAR reproducibility
Rebuilderd
Rebuilderd doesn't clean up old builds, to remove all builds which are no longer references to a package:
delete from builds where id not in (select build_id from packages where build_id is not null);
Rebuilderd also stores logs for succeeded builds which isn't required.
Requeue'ing bad builds can be done as following:
rebuildctl pkgs requeue --suite core --status BAD
Improvements
- add build date to output of
rebuildctl pkgs ls --status BAD --suite core - add build date to the /log output
- add build host to the /log output (so one can identify if a host has a bad build env)
- add a cleanup thread that runs occasionally cleaning up old rebuild results.
Autoclassify script
Make an autoclassify script based on the diffoscope html output stored in rebuilderd. Maybe using the rebuilderd database for now => extract the diffoscope html and inspiration drawn from this script