Can't produce byte-to-byte equivalent tar archives on MacOS

[istvanballok]

I'd like to produce byte-to-byte equivalent tar archives if the packaged file's content is the same.

See https://www.gnu.org/software/tar/manual/html_chapter/tar_8.html#SEC147

Notice, that if you create an archive in POSIX format (see section GNU tar and
POSIX tar) and the environment variable POSIXLY_CORRECT is set, then the two
archives created using the same options on the same set of files will not be
byte-to-byte equivalent even with the above option. This is because the posix
default for extended header names includes the PID of the tar process, which is
different at each run. To produce byte-to-byte equivalent archives in this case,
either unset POSIXLY_CORRECT, or use the following option:

---pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0

The default behavior was changed in 1.33, to omit the PID of the tar process by default:
https://git.savannah.gnu.org/cgit/tar.git/commit/?id=ef0f882382f6faac51aa343193f8d740dff91512

I used the following test case:

# shellcheck disable=SC2089
OPTIONS="--sort=name --mtime=1970-01-01 --owner=0 --group=0 --numeric-owner --pax-option=globexthdr.name=/GlobalHead.%n,exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0"

# shellcheck disable=SC2090
alias xtar='gtar $OPTIONS'

set -x
xtar --version | head -1
set +x

echo test > test.1
echo test > test.2

xtar -czf - test.* | hexdump > out.1

sleep 1
touch test.1

xtar -czf - test.* | hexdump > out.2

diff out.*

I tested with the bottled artifacts of the gnu-tar Formula in versions 1.32, 1.33, 1.34 on Big Sur and they all produce a diff, i.e. although the packaged file's content is the same, the resulting tar file is not byte-to-byte equivalent. I experimented with several combinations of the options, but all of them produced a diff.

I also tested with an alpine container, with gnu tar 1.33, and there the configured pax header works as expected and I get byte-to-byte equivalent outputs:

docker run -it -v $(pwd):/mnt alpine:3.13.2 sh
apk add tar
cd /mnt
# run test above

So I conclude that it is somehow related to the bottled artifacts, but I couldn't figure it out. I didn't attempt to build the binaries from source.

This golang issue had the same motivation as I have for this issue: golang/go#12358

[jonchang]

This is likely due to the filesystem and not Homebrew. AFPS does not guarantee a sorted order when you glob files in your example.

[istvanballok]

The diff suggests though, that the difference is a single byte, so I think it is not related to the sort order.

[gromgit]

Your test case is invalid, because you're telling GNU tar to gzip the tarballs, and the gzip file format contains a 4-byte timestamp right where your diff indicates a numerical difference of 1. Since you sleep for 1 second between generating the tarballs, the diff is completely accounted for.

Remove the z flag to GNU tar in your test scripts, and verify for yourself that there's now no diff in the two tar outputs.

[istvanballok]

Kudos, @gromgit!

My intention was to compress the tar file, and do it at different times, to assert that the resulting .tgz file is really byte-to-byte equivalent, if the content is the same - so in this sense, the test case reflects my intention and is hence valid. :)

Your observation gave a decisive hint, though, and I found this article: https://stackoverflow.com/questions/32997526/how-to-create-a-tar-file-that-omits-timestamps-for-its-contents
but then I realized that if my testcase was working in a docker container, something else must be different on MacOS. And it was gzip. :)

$ gzip --version
Apple gzip 321.40.3

so, after brew install gzip (and hash -r in the same terminal, or in a new terminal session)

gzip --version
gzip 1.10
Copyright (C) 2018 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <https://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Written by Jean-loup Gailly.

and my test case passed. 🎉

A simplified version:

echo test > test.1
echo test > test.2

gtar --sort=name --mtime=1970-01-01 --owner=0 --group=0 --numeric-owner -czf - test.* | sha1sum

now consistently yields 3f6e850e2df635c9ed11e693f7c712887ac86852

[gromgit]

If you're planning to use this on any other Macs than your own, I'd advise using a compression scheme that doesn't embed timestamps to begin with, like bzip2 (j), which is also "standard equipment" in macOS, and generally does quite a bit better at compression than gzip.

[istvanballok]

When tar is executed with the z option, it filters the archive through gzip. On my system, the Apple version of gzip was installed and after installing the GNU version with brew install gzip, my test case worked as expected.

@gromgit mentioned the alternative approach to use bzip2 instead of gzip, with j.