Support for isolated builds

[praiskup]

• Don't misuse the package_state plugin for this purpose
• Standardize the output JSON format format fixed for now, this will take time
• add a feature documentation page
• manual pages
• add unit-tests && behave test
• assert the bootstrap image sha1

[praiskup]

One of the experiments might look like:

1. prepare `f41-bootstrap` image that has dnf5-plugins available
2. download RPMs (cache): mock --calculate-build-dependencies -r fedora-rawhide-x86_64 /tmp/tito/python-copr-1.132-1.git.0.787a685.fc40.src.rpm
3. copy cached RPMs aside: cp -r /var/cache/mock/fedora-rawhide-x86_64/dnf_cache/fedora-2d95c80a1fa0a67d/packages/ /tmp/rawhide
4. create repo: createrepo_c /tmp/rawhide
5. isolated build: mock -r fedora-isolated /tmp/tito/python-copr-1.132-1.git.0.787a685.fc40.src.rpm --config-opts local_directory=/tmp/rawhide --config-opts=bootstrap_image=localhost/f41-bootstrap:latest --config-opts=bootstrap_image_skip_pull=True

Alternative way to 5 might look like:

5. install build deps: mock -r fedora-isolated --installdeps /var/lib/mock/fedora-rawhide-x86_64/result/python-copr-1.132-1.git.0.787a685.fc41.buildreqs.nosrc.rpm  --config-opts local_directory=/tmp/rawhide --config-opts=bootstrap_image=localhost/f41-bootstrap:latest --config-opts=bootstrap_image_skip_pull=True
6. run the final build (no clean): mock -r fedora-isolated --no-clean /tmp/tito/python-copr-1.132-1.git.0.787a685.fc40.src.rpm --config-opts local_directory=/tmp/rawhide --config-opts=bootstrap_image=localhost/f41-bootstrap:latest --config-opts=bootstrap_image_skip_pull=True

[pep8speaks]

Hello @praiskup! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-09-10 15:23:51 UTC

[xsuchy]

This "skip pull" commit is weird. I do not understand it.

[praiskup]

This "skip pull" commit is weird. I do not understand it.

The purpose is to not "podman pull" the image (and fail) if the image is not pushed into any registry. I'm going to document it (or drop the commit), sorry for the confusion.

[praiskup]

Updated with a bit more convenient use-case, see below.

First we calculate the build-deps:

$ mock -r fedora-rawhide-x86_64 --calculate-build-dependencies /tmp/tito/python-copr-1.132-1.git.0.787a685.fc40.src.rpm

Then, using the JSON file in result directory, we download the RPMs and prepare
the local repository:

$ repo-from-json --json /var/lib/mock/fedora-rawhide-x86_64/result/mock-build-environment.json --output-repo /tmp/repo

Then, with the local-repo prepared, we can run the isolated build:

$ mock -r fedora-isolated --isolated-build-config /tmp/prepared/mock-build-environment.json /tmp/repo/ /tmp/tito/python-copr-1.132-1.git.0.787a685.fc40.src.rpm

[praiskup]

The last part is now simplified, one just needs to

$ mock --isolated-build-config /tmp/prepared/mock-build-environment.json /tmp/repo/ /tmp/tito/python-copr-1.132-1.git.0.787a685.fc40.src.rpm

Warning: this whole PoC only works with bootstrap_image_ready = True.

[praiskup]

The experimental use-case looks like this now (after the recent update):

# we use the srpm twice
srpm=/tmp/tito/python-copr-1.132-1.git.0.787a685.fc40.src.rpm

# record list of RPMs and bootstrap image we need to download
mock --calculate-build-dependencies -r fedora-rawhide-x86_64 "$srpm"

# backup the json file
cp -r /var/lib/mock/fedora-rawhide-x86_64/result/mock-build-environment.json /tmp/

# pre-fetch the files
mock-isolated-repo --json /tmp/mock-build-environment.json --output-repo /tmp/repo

# run the isolated build
mock --isolated-build-config /tmp/mock-build-environment.json /tmp/repo/ "$srpm"

[praiskup]

We can not do isolated builds with centos-stream for now, reported: https://issues.redhat.com/browse/CS-2506

[tkopecek]

About json versioning: Should we also store in the config part which mock version created it? I'm more inclined to "no" as mock probably doesn't have enough information about itself (was it e.g. modified in the distribution?) and leave it on other tooling. OTOH, it can be easilly lost which mock should be used for the rebuild - is json version enough to have reliable reproducibility? E.g. json version needn't to change in future twenty mock versions while some other substantial process would be different meanwhile.

[tkopecek]

Saving whole (parsed) mock config could be valuable? E.g. what if I run the --isolated-build phase on different machine with different site-defaults.cfg? Maybe it is also up to "process above" to store/provide correct mock configs?

[praiskup]

About json versioning: Should we also store in the config part which mock version created it? I'm more inclined to "no" as mock probably doesn't have enough information about itself (was it e.g. modified in the distribution?) and leave it on other tooling.

I agree here. When it comes to full reproducibility, it's by far not just about Mock version. We probably want to reproduce on the same kernel, and run Mock-in-container, from the very same container image, with the very same version of container tooling, etc. The "feature-isolated-build" feature page discusses this a bit (the buildroot environment maintained by mock != the environment we run mock in).

OTOH, it can be easilly lost which mock should be used for the rebuild

The NVR of Mock in use is dumped into logs, and if the situation comes - we should be able to find the right version by git-bisect. Though, ... if you prefer to dump the mock version there, I will not object too much! And I really expect a heavy development here, so we may add it later - this PR is just the starting point.

Saving whole (parsed) mock config could be valuable? E.g. what if I run the --isolated-build phase on different machine with different site-defaults.cfg? Maybe it is also up to "process above" to store/provide correct mock configs?

Yes, my vote goes for "process above". If we have a mock container image ID/sha256 like this one, we know both the Mock version and the set of configuration files.

If all of this implicates we should document this more, please say what and where! 👍

[tkopecek]

If all of this implicates we should document this more, please say what and where! 👍

I think we should just stress it a bit more in "Limitations" section. For some it could look like it is providing reproducible builds without taking in account state of host machine software/hardware stack. Just a one sentence that there is still a big influence of external factors.

[praiskup]

I borrowed your wording and updated the Limitation section. Thank you!