Here's what happened in the Reproducible Builds effort between Sunday May 28 and Saturday June 3 2017:

Past and upcoming events

  • On June 9th, Chris Lamb will present at the Hong Kong Open Source Conference 2017 on reproducible builds.

  • We restarted our IRC meetings, now with a monthly schedule where we meet on the first Thursday of the month at a varying time. Our next meeting will be on the first 6th of July at 17:00 UTC. At the June meeting we had the following agenda, and if you are interested there is an automated summary and full logs too.

    1. Introductions
    2. Feedback for the reproducible.json spec format - is that suitable for Guix and F-Droid as well? LEDE/OpenWrt? Coreboot?
    3. writing parser + user interface for reproducible.json
    4. tests.r-b.o/Debian once Stretch has been released
    5. Reproducible Builds Summit 2017
    6. Next meeting
    7. Any other business

Documentation updates

Toolchain development and fixes

  • Chris Lamb wrote a proof-of-concept implementation for #863622 ("apt: warn when installing packages that are not reproducible"). Try it, it's fun!

  • Russ worked on making the debian-policy package build reproducibly which is a good opportunity to shamelessly plug that #844431 ("debian-policy: Packages should be reproducible") is still a work in progress.

Patches and bugs filed

4 package reviews have been added, 6 have been updated and 25 have been removed in this week, adding to our knowledge about identified issues.

Weekly QA work

During our reproducibility testing, FTBFS bugs have been detected and reported by:

  • Adrian Bunk (2)
  • Chris Lamb (1)

diffoscope development

tests.reproducible-builds.org

Mattia Rizzolo:

  • packages-tests: go back to the shorter way of specifying JUnit jobs.
  • update_jdn: call jenkins-jobs instead of jenkins-job-builder, as that's how it is named in 1.6.1 version.
  • Upload jenkins-job-builder 1.6.1-1~bpo8+1 to jessie-backports.

Daniel Kahn Gillmor:

  • Contribute a few typo fixes.

Vagrant Cascadian:

  • Update README to reflect switch to PostgreSQL (some time ago).
  • Add three new boards for armhf tests:
    • ff64a-armhf-rb.debian.net: Firefly-rk3399, Rockchip six-core (Cortex-A72 x2 + Cortex-A53 x4), 2GB RAM, USB-sata (future plans for native sata.)
    • jtx1a-armhf-rb.debian.net: Jetson-tx1, quad-core (big.LITTLE Cortex-A53/A57), ~3.5GB RAM, native SATA ~500GB disk
    • odc2a-armhf-rb.debian.net: Odroic-C2, quad-core (Cortex-A53), ~2GB ram, 128GB eMMC
  • Resurrect rpi2c-armhf-rb.debian.net (Raspberry PI 2B, broadcom bcm2836 quad-core (cortex-A7), 1GB RAM) from the dead.

Holger Levsen:

  • Configure the rc.local service on all build nodes to only start when the network is actually up, as configuring half of them to run in the future requires networking to determine the real current date. See Running Services After the Network is up if you want to learn more about networking with systemd. Somehow this doesn't work yet on the Ubuntu 16.05 arm64 nodes; help welcome, either on #debian-qa or #debian-reproducible.
  • Add rpi2c back to the armhf network.
  • Also add ff64a, odc2a & jtx1a to the armhf network.
    • Add pbuilder/schroot-setup jobs for the new nodes.
    • Add 10 new armhf builder jobs.
    • Disable all build jobs on odc2a again as haveged segfaults on 4.12~rc2, which is needed for this board & setup.
  • Reproducible_cleanup_nodes.sh: adapt for new build service
  • Make performance.html show if build jobs are down due to remote node problems and make code to count enabled jobs more robust.
  • reproducible_build_service: Make it cope with disabled workers.
  • jenkins-master-wrapper: Fail loudly if remote node is not accessible.
  • Increase maximum scheduling queue sizes, to adjust for faster scheduler.
  • New script, init_node, to initialize new build nodes (derived from refactored existing script).
  • Update INSTALL documentation to reflect that we're testing arm64 now too.

Misc.

This week's edition was written by Chris Lamb, Bernhard M. Wiedemann and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Posted 2017-06-06 07:52:25 UTC Tags: reproducible builds

Here's what happened in the Reproducible Builds effort between Sunday June 4 and Saturday June 10 2017:

Past and upcoming events

On June 10th, Chris Lamb presented at the Hong Kong Open Source Conference 2017 on reproducible builds.

Patches and bugs filed

Reviews of unreproducible packages

7 package reviews have been added, 10 have been updated and 14 have been removed in this week, adding to our knowledge about identified issues.

Weekly QA work

During our reproducibility testing, FTBFS bugs have been detected and reported by:

  • Adrian Bunk (4)
  • Chris Lamb (1)
  • Christoph Biedl (1)
  • Niko Tyni (1)

Two FTBFS issues of LEDE (exposed in our setup) were found and were fixed:

diffoscope development

  • Chris Lamb: Some code style improvements

tests.reproducible-builds.org:

Alexander 'lynxis' Couzens made some changes for testing LEDE and OpenWrt:

  • Build tar before downloading everything: On system without tar --sort=name we need to compile tar before downloading everything
  • Set CONFIG_AUTOREMOVE to reduce required space
  • Create a workaround for signing keys: LEDE signs the release with a signing key, but generates the signing key if it's not present. To have a reproducible release we need to take care of signing keys.
  • openwrt_get_banner(): use staging_dir instead of build_dir because the former is persistent among the two builds.
  • Don't build all packages to improve development speed for now.
  • Only build one board instead of all boards. Reducing the build time improves developing speed. Once the image is reproducible we will enable more boards.
  • Disable node_cleanup_tmpdirs

Hans-Christoph Steiner, for testing F-Droid:

  • Do full git reset/clean like Jenkins does
  • hard code WORKSPACE dir names, as WORKSPACE cannot be generated from $0 as it's a temporary name.

Daniel Shahaf, for testing Debian:

  • Remote scheduler:
    • English fix to error message.
    • Allow multiple architectures in one invocation.
    • Refactor: Break out a helper function. Rename variable to disambiguate with scheduling_args.message.
  • Include timestamps in logs
  • Set timestamps to second resolution (was millisecond by default).

Holger 'h01ger' Levsen, for testing Debian:

  • Improvements to the breakages page:
    • List broken packages and diffoscope problems first, and t.r-b.o problems last.
    • Reword, drop 'caused by'.
  • Add niceness to our list of variations, running with niceness of 11 for the first build and niceness of 10 for the second one. Thanks to Vagrant for the idea.
  • Automatic scheduler:
    • Reschedule after 12h packages that failed with error 404
    • Run scheduler every 3h instead of every 6h
  • Add basic README about the infrastructure and merge Vagrants notes about his console host.

Misc.

This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Posted 2017-06-13 20:50:40 UTC Tags: reproducible builds

Here's what happened in the Reproducible Builds effort between Sunday June 11 and Saturday June 17 2017:

Upcoming events

Upstream patches and bugs filed

Reviews of unreproducible packages

1 package review has been added, 19 have been updated and 2 have been removed in this week, adding to our knowledge about identified issues.

Weekly QA work

During our reproducibility testing, FTBFS bugs have been detected and reported by:

  • Adrian Bunk (1)
  • Edmund Grimley Evans (1)

diffoscope development

tests.reproducible-builds.org

As you might have noticed, Debian stretch was released last week. Since then, Mattia and Holger renamed our testing suite to stretch and added a buster suite so that we keep our historic results for stretch visible and can continue our development work as usual. In this sense, happy hacking on buster; may it become the best Debian release ever and hopefully the first reproducible one!

  • Vagrant Cascadian:
  • Valerie Young: Add highlighting in navigation for the new nodes health pages.
  • Mattia Rizzolo:
    • Do not dump database ACL in the backups.
    • Deduplicate SSLCertificateFile directive into the common-directives-ssl macro
    • Apache: t.r-b.o: redirect /testing/ to /stretch/
    • db: s/testing/stretch/g
    • Start adding code to test buster...
  • Holger Levsen:
    • Update README.infrastructure to explain who has root access where.
    • reproducible_nodes_info.sh: correctly recognize zero builds per day.
    • Add build nodes health overview page, then split it in three: health overview, daily munin graphs and weekly munin graphs.
    • reproducible_worker.sh: improve handling of systemctl timeouts.
    • reproducible_build_service: sleep less and thus restart failed workers sooner.
    • Replace ftp.(de|uk|us).debian.org with deb.debian.org everywhere.
    • Performance page: also show local problems with _build_service.sh (which are autofixed after a maximum of 133.7 minutes).
    • Rename nodes_info job to html_nodes_info.
    • Add new node health check jobs, split off from maintenance jobs, run every 15 minutes.
      • Add two new checks: 1. for correct future (2019 is incorrect atm, and we sometimes got that). 2.) for writeable /tmp (sometimes happens on borked armhf nodes).
    • Add jobs for testing buster.
    • s/testing/stretch/g in all the code.
    • Finish the code to deal with buster.
    • Teach jessie and Ubuntu 16.04 how to debootstrap buster.

Axel Beckert is currently in the process of setting up eight LeMaker HiKey960 boards. These boards were sponsored by Hewlett Packard Enterprise and will be hosted by the SOSETH students association at ETH Zurich. Thanks to everyone involved here and also thanks to Martin Michlmayr and Steve Geary who initiated getting these boards to us.

Misc.

This week's edition was written by Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Posted 2017-06-21 16:27:41 UTC Tags: reproducible builds

Here's what happened in the Reproducible Builds effort between Sunday June 18 and Saturday June 24 2017:

Upcoming and Past events

Our next IRC meeting is scheduled for the 6th of July at 17:00 UTC with this agenda currently:

  1. Introductions
  2. Reproducible Builds Summit update
  3. NMU campaign for buster
  4. Press release: Debian is doing Reproducible Builds for Buster
  5. Reproducible Builds Branding & Logo
  6. should we become an SPI member
  7. Next meeting
  8. Any other business

On June 19th, Chris Lamb presented at LinuxCon China 2017 on Reproducible Builds.

On June 23rd, Vagrant Cascadian held a Reproducible Builds question and answer session at Open Source Bridge.

Reproducible work in other projects

LEDE: firmware-utils and mtd-utils/mkfs.jffs2 now honor SOURCE_DATE_EPOCH.

Toolchain development and fixes

There was discussion on #782654 about packaging bazel for Debian.

Dan Kegel wrote a patch to use ar determinitiscally for Homebrew, a package manager for MacOS.

Dan Kegel worked on using SOURCE_DATE_EPOCH and other reproduciblity fixes in fpm, a multi plattform package builder.

The Fedora Haskell team disabled parallel builds to achieve reproducible builds.

Bernhard M. Wiedemann submitted many patches upstream:

Packages fixed and bugs filed

Patches submitted upstream:

Other patches filed in Debian:

Reviews of unreproducible packages

573 package reviews have been added, 154 have been updated and 9 have been removed in this week, adding to our knowledge about identified issues.

1 issue type has been updated:

Weekly QA work

During our reproducibility testing, FTBFS bugs have been detected and reported by:

  • Adrian Bunk (98)

diffoscope development

Version 83 was uploaded to unstable by Chris Lamb. It also moved the previous changes from experimental (to where they were uploaded) to unstable. It included contributions from previous weeks.

You can read about these changes in our previous weeks' posts, or view the changelog directly (raw form).

We plan to maintain a backport of this and future versions in stretch-backports.

Ximin Luo also worked on better html-dir output for very very large diffs such as those for GCC. So far, this includes unreleased work on a PartialString data structure which will form a core part of a new and more intelligent recursive display algorithm.

strip-nondeterminism development

Versions 0.035-1 was uploaded to unstable from experimental by Chris Lamb. It included contributions from:

  • Bernhard M. Wiedemann
    • Add CPIO handler and test case.
  • Chris Lamb
    • Packaging improvements.

Later in the week Mattia Rizzolo uploaded 0.035-2 with some improvements to the autopkgtest and to the general packaging.

We currently don't plan to maintain a backport in stretch-backports like we did for jessie-backports. Please speak up if you think otherwise.

reproducible-website development

  • Chris Lamb:
    • Add OpenEmbedded to projects page after a discussion at LinuxCon China.
    • Update some metadata for existing talks.
    • Add 13 missing talks.

tests.reproducible-builds.org

  • Alexander 'lynxis' Couzens
    • LEDE: do a quick sha256sum before calling diffoscope. The LEDE build consists of 1000 packages, using diffoscope to detect whether two packages are identical takes 3 seconds in average, while calling sha256sum on those small packages takes less than a second, so this reduces the runtime from 3h to 2h (roughly). For Debian package builds this is neglectable, as each build takes several minutes anyway, thus adding 3 seconds to each build doesn't matter much.
    • LEDE/OpenWrt: move toolchain.html creation to remote node, as this is were the toolchain is build.
    • LEDE: remove debugging output for images.
    • LEDE: fixup HTML creation for toolchain, build path, downloaded software and GIT commit used.
  • Mattia 'mapreri' Rizzolo:
    • Debian: introduce Buster.
    • Debian: explain how to migrate from squid3 (in jessie) to squid (in stretch).
  • Holger 'h01ger' Levsen:
    • Debian:
      • Add jenkins jobs to create schroots and configure pbuilder for Buster.
      • Add Buster to README/about jenkins.d.n.
      • Teach jessie and ubuntu 16.04 systems how to debootstrap Buster.
      • Only update indexes and pkg_sets every 30min as the jobs almost run for 15 min now that we test four suites (compared to three before).
      • Create HTML dashboard, live status and dd-list pages less often.
      • (Almost) stop scheduling old packages in stretch, new versions will still be scheduled and tested as usual.
      • Increase scheduling limits, especially for untested, new and depwait packages.
      • Replace Stretch with Buster in the repository comparison page.
      • Only keep build_service logs for a day, not three.
      • Add check for hanging mounts to node health checks.
      • Add check for haveged to node health checks.
      • Disable ntp.service on hosts running in the future, needed on stretch.
      • Install amd64 kernels on all i386 systems. There is a performance issue with i386 kernels, for which a bug should be filed. Installing the amd64 kernel is a sufficient workaround, but it breaks our 32/64 bit kernel variation on i386.
    • LEDE, OpenWrt: Fix up links and split TODO list.
    • Upgrade i386 systems (used for Debian) and pb3+4-amd64 (used for coreboot, LEDE, OpenWrt, NetBSD, Fedora and Arch Linux tests) to Stretch
    • jenkins: use java 8 as required by jenkins >= 2.60.1

Misc.

This week's edition was written by Ximin Luo, Holger Levsen, Bernhard M. Wiedemann, Mattia Rizzolo, Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.