Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds reproducibility tests #818

Merged
merged 3 commits into from
Apr 11, 2023
Merged

Adds reproducibility tests #818

merged 3 commits into from
Apr 11, 2023

Conversation

ForestEckhardt
Copy link
Contributor

Resolves #811

Checklist

  • I have viewed, signed, and submitted the Contributor License Agreement.
  • I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
  • I have added an integration test, if necessary.
  • I have reviewed the styleguide for guidance on my code quality.
  • I'm happy with the commit history on this PR (I have rebased/squashed as needed).

@ForestEckhardt
Copy link
Contributor Author

ForestEckhardt commented Aug 16, 2022

Passenger container-diff diff passenger1 passenger2 --type=file output:

-----File-----

These entries have been added to passenger1: None

These entries have been deleted from passenger1: None

These entries have been changed between passenger1 and passenger2:
FILE                                                                                                                                      SIZE1        SIZE2
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/passenger-6.0.14/gem_make.out        1.5K         1.5K

@ForestEckhardt
Copy link
Contributor Author

ForestEckhardt commented Aug 16, 2022

Puma container-diff diff puma1 puma2 --type=file output:

-----File-----

These entries have been added to puma1: None

These entries have been deleted from puma1: None

These entries have been changed between puma1 and puma2:
FILE                                                                                                                                 SIZE1        SIZE2
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/nio4r-2.5.2/ext/nio4r/Makefile                                  8.1K         8.1K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/puma-4.3.5/ext/puma_http11/Makefile                             8K           8K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/nio4r-2.5.2/gem_make.out        3.4K         3.4K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/puma-4.3.5/gem_make.out         1.2K         1.2K

@ForestEckhardt
Copy link
Contributor Author

ForestEckhardt commented Aug 16, 2022

Rackup container-diff diff rackup1 rackup2 --type=file output:

-----File-----

These entries have been added to rackup1: None

These entries have been deleted from rackup1: None

These entries have been changed between rackup1 and rackup2: None

@ForestEckhardt
Copy link
Contributor Author

ForestEckhardt commented Aug 16, 2022

Rails container-diff diff rails1 rails2 --type=file output:

-----File-----

These entries have been added to rails1:
FILE                                                                                                                                        SIZE
/layers/paketo-buildpacks_rails-assets/assets/public-assets/.sprockets-manifest-b918d147ec7a1a00a4a47fd643261466.json                       818B
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/Bf                                                          130B
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/Bf/BfHVVJbIHY1bS5uEKkAv7_7ch__Olro8_PCek8ZS1ZQ.cache        130B
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/C0                                                          1.7K
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/C0/C0AFMYKhc8b6_Md-ozPa_nq44mjo_eYRzeL7wPQD8Ag.cache        1.7K
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/H1                                                          1.4K
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/H1/H16fZfkugyondzV_1XT3EGD8aWjwprxVEVd3fzgAxLE.cache        1.4K
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/aN                                                          136B
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/aN/aNm6_u0imFiirEgfMU1BVJZf3-E-InsOhAmkEOh33ps.cache        136B

These entries have been deleted from rails1:
FILE                                                                                                                                        SIZE
/layers/paketo-buildpacks_rails-assets/assets/public-assets/.sprockets-manifest-664fa7c6f8d6db219f28831774b4860c.json                       818B
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/2f                                                          136B
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/2f/2fyBBHiLuTbeeKv_bsZMpBBTjO9ErOojJ20qp7Z_n1Q.cache        136B
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/58                                                          1.4K
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/58/58_5vKs80bVbicpya6K_AfDVoPdCZ6tW73njJF3lCkM.cache        1.4K
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/QB                                                          130B
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/QB/QB5qKDRSSW2sO2X0oZGlUa9loklHVoiE8aMq4jgb1FI.cache        130B
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/r2                                                          1.7K
/layers/paketo-buildpacks_rails-assets/assets/tmp-cache-assets/sprockets/v4.0.0/r2/r2yJDF9WWxa-AEp7LuJGwrzAYF8_7hWfjGmgZXnKy90.cache        1.7K

These entries have been changed between rails1 and rails2:
FILE                                                                                                                                                   SIZE1        SIZE2
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/ffi-1.15.0/ext/ffi_c/libffi-x86_64-linux/config.log                               56.2K        56.2K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/ffi-1.15.0/gem_make.out                           21.3K        21.3K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/sassc-2.4.0/ext/Makefile                                                          9.4K         9.4K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/ffi-1.15.0/ext/ffi_c/Makefile                                                     8.4K         8.4K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/nio4r-2.5.7/ext/nio4r/Makefile                                                    8.1K         8.1K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/puma-4.3.7/ext/puma_http11/Makefile                                               8K           8K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/racc-1.5.2/ext/racc/cparse/Makefile                                               7.9K         7.9K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/websocket-driver-0.7.3/ext/websocket-driver/Makefile                              7.9K         7.9K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/nio4r-2.5.7/gem_make.out                          3.4K         3.4K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/sassc-2.4.0/gem_make.out                          3.2K         3.2K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/puma-4.3.7/gem_make.out                           975B         976B
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/websocket-driver-0.7.3/gem_make.out               887B         889B
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/racc-1.5.2/gem_make.out                           842B         842B
/layers/paketo-buildpacks_rails-assets/assets/public-assets/manifest-b4bf6e57a53c2bdb55b8998cc94cd00883793c1c37c5e5aea3ef6749b4f6d92b.js.gz            22B          22B
/layers/paketo-buildpacks_rails-assets/assets/public-assets/application-04024382391bb910584145d8113cf35ef376b55d125bb4516cebeb14ce788597.css.gz        20B          20B

NOTE: container-diff must have some form of caching and it was giving me bad readings for images with the same name.

@ForestEckhardt
Copy link
Contributor Author

Rake container-diff diff rake1 rake2 --type=file output:

------File-----

These entries have been added to rake1: None

These entries have been deleted from rake1: None

These entries have been changed between rake1 and rake2: None

@ForestEckhardt
Copy link
Contributor Author

Thin container-diff diff thin1 thin2 --type=file output:

-----File-----

These entries have been added to thin1: None

These entries have been deleted from thin1: None

These entries have been changed between thin1 and thin2:
FILE                                                                                                                                        SIZE1        SIZE2
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/eventmachine-1.2.7/ext/Makefile                                        8.5K         8.5K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/eventmachine-1.2.7/ext/fastfilereader/Makefile                         7.9K         7.9K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/thin-1.8.1/ext/thin_parser/Makefile                                    7.9K         7.9K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/eventmachine-1.2.7/gem_make.out        892B         891B
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/thin-1.8.1/gem_make.out                860B         861

@ForestEckhardt
Copy link
Contributor Author

Unicorn container-diff diff unicorn1 unicorn2 --type=file output:

-----File-----

These entries have been added to unicorn1: None

These entries have been deleted from unicorn1: None

These entries have been changed between unicorn1 and unicorn2:
FILE                                                                                                                                      SIZE1         SIZE2
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/raindrops-0.20.0/mkmf.log            129.7K        129.7K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/kgio-2.11.4/mkmf.log                 42.3K         42.3K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/raindrops-0.20.0/gem_make.out        10.2K         10.2K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/kgio-2.11.4/ext/kgio/Makefile                                        9K            9K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/unicorn-6.1.0/ext/unicorn_http/Makefile                              8.1K          8.1K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/raindrops-0.20.0/ext/raindrops/Makefile                              8.1K          8.1K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/kgio-2.11.4/gem_make.out             2.5K          2.5K
/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/extensions/x86_64-linux/2.7.0-static/unicorn-6.1.0/gem_make.out           1K            1K

@ForestEckhardt
Copy link
Contributor Author

I think that the deviation in runs for most of these happens somewhere in these lines of code https://github.com/ruby/ruby/blob/00f411c58ac6105f135ff4501ea5bd1d691fcc32/lib/mkmf.rb#L2000-L2014 but I am unclear on how it is happening.

@ForestEckhardt
Copy link
Contributor Author

I have not been able to reproduce the the behavior where a seemingly random string is selected in the repl on the final build container. My next idea is to modify the bundle-install buildpack to allow the build phase to be intercept and see if it can be reproduced in repl there.

@ryanmoran ryanmoran force-pushed the adds-reproducibility-tests branch from 96715c3 to e61ed71 Compare April 3, 2023 18:21
@sophiewigmore sophiewigmore marked this pull request as ready for review April 11, 2023 10:01
@sophiewigmore sophiewigmore requested a review from a team as a code owner April 11, 2023 10:01
@sophiewigmore
Copy link
Member

sophiewigmore commented Apr 11, 2023

@ryanmoran I'd love to get these tests merged in to main before I cut a release of the ruby language family. Does anything else need to change here?

@sophiewigmore sophiewigmore added the semver:patch A change requiring a patch version bump label Apr 11, 2023
@ryanmoran ryanmoran merged commit 8f063fa into main Apr 11, 2023
@ryanmoran ryanmoran deleted the adds-reproducibility-tests branch April 11, 2023 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver:patch A change requiring a patch version bump
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Assert that builds are reproducible
3 participants