Reproducible Build

[timkenhan]

Added a couple environment variables that alters BUILD_TIME & BUILD_ID to have deterministic values.

BUILD_TIME: can be "pkg" for the current time of the package build (default) or "ebuild" for the ebuild file's timestamp.

BUILD_ID_TYPE: can be "int" for autoincrement (default) or "hash" for the hash of the environment.

Currently tested to work with sys-apps/baselayout package with the command:

PKGDIR="/tmp/pkg" FEATURES="-getbinpkg" BUILD_TIME="ebuild" BUILD_ID_TYPE="hash" bin/emerge --ignore-default-opts -B sys-apps/baselayout

Known issues:

• some warning on mtime data type (should be string, but data is int)
• while image.tar.zst is in the clear, the other part of the files still has present mtime
• some more complex compilation process may still insert non-deterministic values on its own

[eli-schwartz]

• need to quote ${D}
• Modifying the timestamps of installed files is conditionally problematic, since installed file contents can contain the timestamps of other installed file contents and require a match. In particular, this is a problem for python bytecode. If you touch the timestamp of *.py files, then all *.pyc files will be invalidated and the next time they are imported as root, the interpreter will regenerate and rewrite the .pyc files with new values.

[timkenhan]

@eli-schwartz thank you for the feedback

I'll change the ${D} accordingly.

As for the timestamp, any alternative suggestion to make it deterministic?

[zmedico]

It's not possible if we are to fulfill the "Preservation of file modification times" requirement of PMS:
https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-146001r1

There was an ignore-mtime option dropped from #991 due to the same requirement.

[eli-schwartz]

Making file metadata deterministic when PMS explicitly says it shall not be deterministic is a tough topic.

All I can say is that from a pure usability standpoint, you don't really know what software depends on the timestamp. Python bytecode may be only one example.

Setting $SOURCE_DATE_EPOCH is explicitly respected by python bytecode to use a slower and less efficient bytecode invalidation format. It's also the actual reproducible builds specification. It is likely any other software depending on timestamps, will respect that variable if it respects anything at all.

[timkenhan]

Is there a chance we can revise this specification?

Reproducibility has become more and more relevant these days, and it has became relevant to us Gentoo users especially since binary packages are offered officially.

Just like binary packages of other distros (e.g. Debian, even Arch is activelly spending effort on it), it would be nice to be able to verify the official build somehow, even if it means having to match the USE flags and other configs.

[timkenhan]

I've submitted this ticket for EAPI to allow mtime modification in future version.