Testing story of slices

[anuraaga]

I have been looking at this project and it seems very interesting, I won't enumerate but see many things to like compared to existing distroless techniques.

One thing I'm worried about is the guarantees towards compatibility. I am looking at python specifically, and while it's not there yet am glad to see what's probably upcoming support

canonical/chisel-releases#82

Looking at the PR I'm concerned though about the testing or robustness of the chisel. It is literally listing out individual .py files from the apks. Actually I found a similar PR where many comments were of the form "this file is missing".

I think two concrete questions that came to mind

• Is there an absolute guarantee that files aren't added to an apk on patch releases? FWIU, chisels are targeted at an Ubuntu release version and will operate on the latest patch version of a package which can change, and if a file is added, it seems like it would be left out

• Is there any plan for integration testing against real code, in this case Python? On the linked PR, there are no CI checks running from what I can tell. How is it ensured that a slice is compatible with the ecosystem it is targeting? Given robust compatibility tests, the previous point would be less of a big deal since failures due to new files would be detected

Knowing this story would really help deciding on using chisel. For context I am currently using Google distroless and don't like that updates are slow even in the face of CVEs. But I would need to stick with it if there is risk of mysterious build failures even when targeting a specific Ubuntu release line.

[cjdcordeiro]

@anuraaga

Thanks for the insights :)

Before addressing your comments, let me say you're on the right track :) Something we're currently working (and hope to have within a couple of months, progressively) is continuous integration for slice definitions. You'll already find the beginnings of this work here, where we explain how the slice governance will work and work we expect contributors to submit their slices. Our CI will follow those guidelines, and eventually, in the future, even include smoke tests for slices :)

Is there an absolute guarantee that files aren't added to an apk on patch releases? FWIU, chisels are targeted at an Ubuntu release version and will operate on the latest patch version of a package which can change, and if a file is added, it seems like it would be left out

A slice definitions file is tightly coupled to the debian package it slices. So yes, if there's a patch release upstream that modifies the deb's tree, chisel won't pick up on those. If the upstream is disruptive, our CI will catch it, but if the upstream change is adding new content to the deb, then the slice definitions file must also be updated. That's unavoidable I'm afraid, because we can't predict upstream changes. We can only act on them, after the fact.

Is there any plan for integration testing against real code, in this case Python? On the linked PR, there are no CI checks running from what I can tell. How is it ensured that a slice is compatible with the ecosystem it is targeting? Given robust compatibility tests, the previous point would be less of a big deal since failures due to new files would be detected

yep :) as mentioned above. We'll start with generic black-box testing and we hope to introduce user-defined smoke tests in the future (medium-term).