zopen-build: Generate a zstd-compressed asset, use sha256 checksum, change package naming convention

[IgorTodorovskiIBM]

• Adds code to build a zstd compressed asset based on discussion in Use better compression algorithm for pax files #613 (comment)
• Changes metadata.json to specify an array of assets, now including pax.Z and .zst
• changes the package naming convention to ${PACKAGE_NAME}-${versionString}-${date}.zos.pax" so that we include the correct version string

[v1gnesh]

The 'date' part can be found from the file's creation/modification time, right?
Typing this out to see how it looks 😁

current:
zstd-1.5.5.20231124_101200.zos.pax.Z

linux/s390x:
zstd-1.5.5-4.fc39.s390x.rpm

zos/s390x:
zstd-1.5.5-4.zos24.s390x.pax.zst

Makes me wonder... do we need to pax before running zstd on it?
pax is our rough equivalent of rpm/deb... as a container of files.

The -4 here refers to 4th release of the package at this v.r.m.

[IgorTodorovskiIBM]

The 'date' part can be found from the file's creation/modification time, right? Typing this out to see how it looks 😁

current: zstd-1.5.5.20231124_101200.zos.pax.Z

linux/s390x: zstd-1.5.5-4.fc39.s390x.rpm

zos/s390x: zstd-1.5.5-4.zos24.s390x.pax.zst

Makes me wonder... do we need to pax before running zstd on it? pax is our rough equivalent of rpm/deb... as a container of files.

The -4 here refers to 4th release of the package at this v.r.m.

That makes sense, I was a little confused by what to put under "release" so that helps!

I think until we add support for maintaining attributes like file tags in zstd and others, we should continue to use pax as the container format.

[IgorTodorovskiIBM]

It should look like this now:

git-2.43-zos24.s390x.pax

Ideally, release would be N.zos24, and N would increment from the last published release. We can probably use zopen list to query the latest release.

[DevonianTeuchter]

By adding in the release, are we now publishing 3 versions (2.4/2.5/3.1) ?? Linux platforms add in the os release as they do publish different releases for those os's. Eg. Fedora 39 has python3-productmd-1.37-1.fc39_1.38-1.fc39.noarch.drpm and released similar for fc38. The normal expectation for a user would be that if the package has a z/OS 2.4/2.5/3.1, then it will only apply to that version, not that it applies to that and later versions.
Is s390x applicable to z/OS rather than Linux or is there a different naming convention (uname reports OS/390!!) - unless we are doing 24/31/64-bit versions then maybe those could be added.
tbh, your original ${PACKAGE_NAME}-${versionString}-${date}&time.zos.pax made sense to me :-)

[v1gnesh]

are we now publishing 3 versions (2.4/2.5/3.1) ??

Setting up for it, hopefully. And dropping 2.4... soon.
We will end up setting up a "build matrix" in CI to do various combos.

I'm not entirely sure if, similarly, we need to keep separate builds for Go programs built in 1.19, 1.20, 1.21, etc.

python3-productmd-1.37-1.fc39_1.38-1.fc39.noarch.drpm

In this case, it may be a deltarpm file, that contains just the stuff to go from 1.37 to 1.39 in fc39.

The normal expectation for a user would be that if the package has a z/OS 2.4/2.5/3.1, then it will only apply to that version, not that it applies to that and later versions.

Yup, a package for a particular OS.

GOOS-GOARCH is zos/s390x so s390x still applies for z/OS.
s390 being 31-bit arch and s390x being 64-bit.

24/31/64 - 😱 - this is indeed an additional term we may need to add, assuming these refer to ports that run in 31-bit.

[v1gnesh]

Last 2 sentences of previous comment... I wonder... does that mean 31-bit builds should be called zstd-1.5.5-4.zos24.s390.pax.zst.

[IgorTodorovskiIBM]

For C/C++ we're still targeting V2R4 (https://github.com/ZOSOpenTools/comp_clangport/blob/main/buildenv#L84).

are we now publishing 3 versions (2.4/2.5/3.1) ??

Setting up for it, hopefully. And dropping 2.4... soon. We will end up setting up a "build matrix" in CI to do various combos.

I'm not entirely sure if, similarly, we need to keep separate builds for Go programs built in 1.19, 1.20, 1.21, etc.

python3-productmd-1.37-1.fc39_1.38-1.fc39.noarch.drpm

In this case, it may be a deltarpm file, that contains just the stuff to go from 1.37 to 1.39 in fc39.

The normal expectation for a user would be that if the package has a z/OS 2.4/2.5/3.1, then it will only apply to that version, not that it applies to that and later versions.

Yup, a package for a particular OS.

GOOS-GOARCH is zos/s390x so s390x still applies for z/OS. s390 being 31-bit arch and s390x being 64-bit.

24/31/64 - 😱 - this is indeed an additional term we may need to add, assuming these refer to ports that run in 31-bit.

Right now we're targeting 2.4 in our C/C++ builds: https://github.com/ZOSOpenTools/comp_clangport/blob/main/buildenv#L84 . Newer OS levels for z/OS are backwards compatible. So for us, this means the min OS level is 2.4, but it should work for 2.5, 3.1 as well. Perhaps we can use just "zos" as the release for now until we decide to add builds for each version?

[DevonianTeuchter]

You're last comment indicated to use zos as the release - the code seems to still be using zos24:
osRelease="zos24" #TODO: change based on build target
paxFileName="${PACKAGE_NAME}-${versionString}-${osRelease}.s390x.pax"

Otherwise, LGTM

[MikeFultonDev]

LGTM

[v1gnesh]

Does anything need to go in before this, @IgorTodorovskiIBM?
Would be nice to see standard file names in the releases section :)

[IgorTodorovskiIBM]

Does anything need to go in before this, @IgorTodorovskiIBM? Would be nice to see standard file names in the releases section :)

Don't think so, I'll need to the sync the branch and do further testing to make sure I'm not breaking anything