Reproducible Builds

  • Python issues due to tests?: https://reproducible.archlinux.org/api/v0/builds/342940/diffoscope
  • Java jar generation in libs

Java JAR

gradle maven

Fedora

https://github.com/rpm-software-management/mock/issues/692 - clamp timestamps https://github.com/rpm-software-management/rpm/pull/1532 - build info file

  • try to reproduce cockpit with mockbuild

https://github.com/fepitre/rpmreproduce

flatpak

https://fedoramagazine.org/an-introduction-to-fedora-flatpaks/ https://blogs.gnome.org/mclasen/2018/07/07/flatpak-making-contribution-easy/ https://ranfdev.com/blog/flatpak-builds-are-not-reproducible/ https://github.com/flatpak/flatpak-builder/issues/251 https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues/1320

  • diffoscope support?
  • CI on flathub repositories?
  • reproducing

Diffing a flatpak

For Cockpit, comparing the build dir output

flatpak-builder --disable-cache  --disable-rofiles-fuse --force-clean flatpak-build-dir1  org.cockpit_project.CockpitClient.yml
flatpak-builder --disable-cache  --disable-rofiles-fuse --force-clean flatpak-build-dir2  org.cockpit_project.CockpitClient.yml
diffoscope flatpak-build-dir1 flatpak-build-dir2

Comparing using two repos:

flatpak-builder --repo=repo1 --disable-cache  --disable-rofiles-fuse --force-clean flatpak-build-dir  org.cockpit_project.CockpitClient.yml

flatpak-builder --repo=repo2 --disable-cache  --disable-rofiles-fuse --force-clean flatpak-build-dir  org.cockpit_project.CockpitClient.yml

Get the refs from ostree:

ostree refs --repo=repo1

ostree show --repo=repo1 runtime/org.cockpit_project.CockpitClient.Debug/x86_64/devel
ostree show --repo=repo2 runtime/org.cockpit_project.CockpitClient.Debug/x86_64/devel

Confirm the ContentChecksum is the same.

live iso

Reproducible live iso

Issues

  • libopensmtpd - mandoc has a "$Mdocdate$" variable which does not respect SOURCE_DATE_EPOCH
  • hugin - gzip timestamps
  • pcp - gzip timestamp
  • libkolabxml XML ordering https://git.kolab.org/T2642 https://bugzilla.opensuse.org/show_bug.cgi?id=1060506 try to set XERCES_DEBUG_SORT_GRAMMAR, but that needs to be in xerces-c which is kinda untested and dumb
  • mm-common
  • musescore https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/musescore3.html
  • openpmix PMIX_CONFIGURE_HOST
  • perl-crypt-random-tesha2 don't advertise entropy
  • ssr records $USER and $date
  • libgtop records uname
  • openxr script is not reproducible.
  • php phar timestamps
  • namazu records $(hostname)
  • dosemu timestamps
  • echoping hostname
  • python-lxml-docs timestamp in "Generated On"
  • ant-doc javadoc adds timestamp to documentation. Generated by javadoc (14.0.2) on Sun Nov 15 16:33:44 UTC 2020
  • emelfm2 kernel + timestamp
  • libiio timestamp
  • gajim man pages (gzip) and pyc bytecode
  • fs-uae zip file not ordered? permission? zip issues?!
  • gutenprint uname/ timestamp recording
  • libmp4v2 timestamp
  • gdk-pixbuf2-docs order issue in generated documentation
  • ghostpcl timestamp
  • libgxps timestamp
  • netcdf & netcdf-fortran uname
  • nethack build date
  • python-lxml timestamp in generated docs
  • qastools gzip timestamp (https://gitlab.com/sebholt/qastools/)
  • qtikz sqlite database with datetime difference in TimeStampTable
  • rmlint - gzip timestamp and timestamp in rmlint
  • glhack - timestamp
  • glob2 - timestamp
  • docker - timestamp
  • radamsa - needs a rebuild
  • eq10q - needs a rebuild
  • harvid needs a rebuild due to size issues with an older makepkg version (fails to build)
  • colord binary seems to embed the profile data as a random hash?
  • tbb timestamp, build host and build kernel
  • ruby-colorize timestamp in gemspec
  • rebuild ruby-* packages which do not remove "$pkgdir/$_gemdir/gems/$_gemname-$pkgver/ext" as it contains non-reproducible files.
  • i7z - gzip timestamp
  • openmpi - records hostname
  • v2ray-domain-list-community - geosite.dat not ordered
  • unrealircd - timestamp in binary
  • libcec - hostname/timestamp
  • hevea - ocaml build /tmp/$tmp path differs https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786913
  • mari0 - zip file
  • arj - date https://reproducible.archlinux.org/api/v0/builds/118386/diffoscope
  • ibus - date
  • argyllcms - (date) - https://www.freelists.org/list/argyllcms send email about created date containing hours/minutes/second and SOURCE_DATE_EPOCH
  • dd_rescue - man page gz timestamp => mail maintainer https://sourceforge.net/p/ddrescue/tickets/
  • deepin-wallpapers => most likely order issue with the wildcard in the makefile nope, most likely image-blur is not reproducible

openexr reproducer

python specification/scripts/genxr.py -registry specification/registry/xr.xml -o /home/jelle/projects/OpenXR-SDK-Source/build/include/openxr/   openxr_reflection.h

Man page gzip timestamp issue

Fixing all the gzip timestamp issue packages is a lot of work and patching upstream everywhere is not really doable. An idea might be to detect gzip files which are non-reproducible and let a makepkg option like zipman or extend zipman to take care of this.

touch foo
gzip foo
file bar.gz | grep modified &>/dev/null  && gunzip -c bar.gz | gzip -9 -n -c > test.gz

Haskell packages

Try to build them without !strip and then compare the packages.

https://gitlab.haskell.org/ghc/ghc/-/wikis/deterministic-builds https://gitlab.haskell.org/ghc/ghc/-/issues/12935

Ideas

  • Year blog post
  • Documentation about reproducible builds in the packager wiki / packaging wiki

Package pacman in Debian

 -> sudo pbuilder create
 -> sudo cowbuilder create
 -> sudo gbp buildpackage --git-ignore-new --git-pbuilder -nc

rebuilderd-website

  • Improve loading performance
  • add make install target

Python issues

For pyc differences PYTHONHASHSEED can be set to a fixed value to try and circumvent the random hash initialisation getting embedded in pyc files

For test files being show in the diffoscope results as pyc files and not in the rebuild package the issue is probably that pyc files generated by running tests are installed errorsnly. Exporting PYTHONDONTWRITEBYTECODE=1 when running the tests.

Rebuilderd

Rebuilderd doesn't clean up old builds, to remove all builds which are no longer references to a package:

delete from builds where id not in (select build_id from packages where build_id is not null);

Rebuilderd also stores logs for succeeded builds which isn't required.

Requeue'ing bad builds can be done as following:

rebuildctl pkgs requeue --suite core --status BAD

Improvements

  • add build date to output of rebuildctl pkgs ls --status BAD --suite core
  • add build date to the /log output
  • add build host to the /log output (so one can identify if a host has a bad build env)
  • add a cleanup thread that runs occasionally cleaning up old rebuild results.

Autoclassify script

Make an autoclassify script based on the diffoscope html output stored in rebuilderd. Maybe using the rebuilderd database for now => extract the diffoscope html and inspiration drawn from this script

Twitter bot

Twitter bot for notifications about reproducible builds in IRC and allowing tweets from irc.