composepost: apply add-determinism pyc-zero-mtime

[lukewarmtemp]

Fixes: ostreedev/ostree#1469

Assuming that add-determinism is installed on the system, apply add-determinism to set the embedded mtime in all .pyc files to zero.

[cgwalters]

Thanks for this!

add-determinism is generally a build tool, and I think it'd be nicer to support a model where it's not shipped onto the end system.

However, right now we don't quite expose such a "multi stage" mechanism via treefile-based builds. If we did it'd be useful for a whole lot of other things too (for example, in theory one could drop dracut or even rpm out of the runtime requirements). This is what https://gitlab.com/fedora/bootc/base-images-experimental is aiming to address to a degree.

[lukewarmtemp]

From what we understand, the main concern is that add-determinism would only need to used when building the base image, meaning that it would be nice if we could remove it during runtime. The hope is that whenever a "multi stage" mechanism is implemented, add-determinism would only be used during build time. However, would this also address the case when packages have been layered on top of the base image? We also need to apply add-determinism whenever ostree container commit is run since layering a new python package would still have mismatched mtimes.

As an alternative to having add-determinism on the system itself, would depending on add-determinism by importing it as a rust crate and using an API be a good option? There's discussion on this PR: keszybz/add-determinism#27 (comment) about potentially implementing an API and hashing out the details of what the caller would look like.

[cgwalters]

However, would this also address the case when packages have been layered on top of the base image? We also need to apply add-determinism whenever ostree container commit is run since layering a new python package would still have mismatched mtimes.

Yes, this is a notable point.

As an alternative to having add-determinism on the system itself, would depending on add-determinism by importing it as a rust crate and using an API be a good option?

It's an option; the advantage is it keeps add-determinism as an implementation detail. It's still shipped code, but probably a good bit smaller (since we wouldn't have duplication of the common libraries).

[cgwalters]

This should probably be its own function like run_add_determinisim(rootfs);

[cgwalters]

Also this is not a great way to check for the existence of a binary. While the Rust stdlib doesn't have a single function for this I think it's trivial to do by combining

• let path = std::env::var_os("PATH")
• https://doc.rust-lang.org/std/env/fn.split_paths.html

[cgwalters]

This won't do what you expect or want. In an rpm-ostree build, the /usr is whatever happens to be the build environment, not the target.

In rpm-ostree we try hard to use fd-relative accesses generally via cap-std.

To do so with an external process, a typical pattern is to use e.g. cwd_dir and then pass . to the child process as the target.

[keszybz]

However, would this also address the case when packages have been layered on top of the base image? We also need to apply add-determinism whenever ostree container commit is run since layering a new python package would still have mismatched mtimes.

Yes, this is a notable point.

Yeah, so it really sounds like it needs to be on the image.

As an alternative to having add-determinism on the system itself, would depending on add-determinism by importing it as a rust crate and using an API be a good option?

It's an option; the advantage is it keeps add-determinism as an implementation detail. It's still shipped code, but probably a good bit smaller (since we wouldn't have duplication of the common libraries).

Right. Currently, the binary is 2.3MB. I think this is totally fine for build environments, but it's a bit heavy-weight for being installed on final images. Using it as a crate is one option. Another option would be to provide a stripped-down separate binary build…

[cgwalters]

Currently, the binary is 2.3MB. I think this is totally fine for build environments, but it's a bit heavy-weight for being installed on final images.

rpm-ostree already drags in a lot of stuff transitively, most notably the libdnf stack, etc. The use cases we have today tend to have at least 400+MB images, commonly GB or more when you add desktops in the mix, libvirt, etc.

Was thinking about this more and I have no problem just making rpm-ostree soft depend on a separate add-determinism binary. Maybe even hard depend by default with a build-time option to disable it.