Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: create consistent output across architectures #69

Closed
wants to merge 3 commits into from

Conversation

nlf
Copy link
Contributor

@nlf nlf commented Mar 12, 2021

thanks to architecture specific optimizations in the default deflate compression strategies, we end up with inconsistent output between arm64 and x86_64.

this change forces the compression strategy to one that appears to give more consistent results. they are larger, but since they're just temp files that feels like an acceptable trade off in the name of consistency.

for posterity's sake i'll mention here that Z_HUFFMAN_ONLY also provides a consistent end result, though i found better performance and compression with Z_RLE in some brief testing of a handful of npm related repositories.

note: i updated the CI matrix to match that of https://github.com/npm/cli

yes, the windows tests fail. however, i felt it was important to test the updated compression strategy for consistency in windows as well as linux and osx. i can fix the rest of the tests for windows before this lands, or we can land it as-is with a follow up PR to fix the tests for windows

edit: the node 10.1 test failures were a surprise, though appear to be unrelated to these changes

References

Closes npm/cli#2846

gzip: {
// forcing the compression strategy to Z_RLE
// yields a bit-identical, though considerably
// larger, end result in both arm64 and x86_64.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... reproducible builds are good, but I feel like sending considerably more data over the wire is not great.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it any less bad if you add

{
  windowBits: 15,
  memLevel: 9,
  level: 9,
  strategy: 3,
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused, is the dir fetcher here in pacote also what libnpmpack uses to create the tarball for upload?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

those settings make no difference at all.

also, confirmed that yes this is indeed the module that packs a tarball for publishing. so, yes, you're right, this is not what we want to do by default. i can tweak this so that this setting will only take place as part of fetching and packing a remote git repository.

@nlf
Copy link
Contributor Author

nlf commented Mar 16, 2021

closing to rework this and split it into two distinct pull requests

@nlf nlf closed this Mar 16, 2021
@lukekarrys lukekarrys deleted the nlf/change-deflate-strategy branch September 23, 2022 00:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants