Feature: support variable naming in pkg_deb

[zegl]

Description of the problem / feature request:

pkg_deb takes a version parameter for setting the version of the version of the generated deb-package.

It would be appreciated if this parameter would support "stamping", so that an external script (--workspace_status_command) can dynamically generate this version.

Similar to how the container_push rule supports stamping in it's tag parameter.

Feature requests: what underlying problem are you trying to solve with this feature?

Simplify creation of pkg_debs, if this was supported I wouldn't need to save the version to a file and use the version_file parameter.

What's the output of bazel info release?

release 0.22.0

[aiuto]

Whatever mechism designed must work for pkg_rpm() as well.
What we do internally at Google is allow $var substitutions in the version strings and then provide those variables in the builds that make distributable versions.

Doing this well probabl requires some design. Open questions:

• should we jus allow variable substitution from the environment?
• should it only be from workspace_status_command?
• What do people even want as paramaterizable version numbers? Is a date/time based substitution sufficient if you want it automatic? If not, why is it better to parametirize it than to update the BUILD file?

[ashi009]

I think there are already some mechanisms to stamp the packages yielded, say using a dynamically generated version_file which consumes workspace_status. However, doing so means we have to write such genrule everywhere. It would be nice to have it integrated in the pkg rules directly.

Whatever mechism designed must work for pkg_rpm() as well.
What we do internally at Google is allow $var substitutions in the version strings and then provide those variables in the builds that make distributable versions.

Doing this well probabl requires some design. Open questions:

• should we jus allow variable substitution from the environment?

Personally, I don't like it. Setting environment variables is not as obvious as a flag passed to bazel.

• should it only be from workspace_status_command?

Keep it consistent with other rules using variables from workspace_status_command would be great.

• What do people even want as paramaterizable version numbers? Is a date/time based substitution sufficient if you want it automatic? If not, why is it better to parametirize it than to update the BUILD file?

Updating BUILD file on every release sounds like toil. Also introducing changes to codebase during the release is problematic -- consider the case where a cherry-pick is needed for an old release branch, such change on the BUILD file may prevent the cherry-picked patch from being applied.

[aiuto]

What I could see is a shared attribute env_vars:string_list.

If you set that on any rule you get the dependency on stable_status and enable the lookup, but only for those variable listed. That nicely documents that the rule depends on something outside the BUILD file (perhaps a default can be specified there). I could also see this variable expansion extend into the template and spec files.

This needs a better design with some examples before implementing it. Any implementation should also cover all the rules, not just one package type.

[prashantv]

Indirectly related to this, when a debian package is created, it's currently named with the version,

rules_pkg/pkg/pkg.bzl

Line 381 in 098022c

out_deb = "%s_%s_%s.deb" % (package, version, architecture)

There are tools that assume that the file is named in this way, e.g., in dak:
https://github.com/Debian/dak/blob/master/daklib/regexes.py#L126-L128
https://github.com/Debian/dak/blob/master/dak/import.py#L116-L124

If the version isn't known till build time, then it can't be used as part of the output file. Ideally, there would be a way to output a tar file with the files named correctly based on version as well.

[aiuto]

More flexibility in output file naming is part of #193, #143, and #101. The solution to that proposed in #198 will also handle this stamp case.

[terinjokes]

#198 was merged, but it's not possible to use the functionality with pkg_deb

[aiuto]

Yes. File renaming from configuration in all rules is on my list. Getting date and time stamps in the name is still a difficult task. That is simply not available at analysis time. I have an idea about how to hack around that but exploring it is not a priority for me.

…

On Mon, Dec 7, 2020 at 6:55 PM Terin Stock ***@***.***> wrote: #198 <#198> was merged, but it's not possible to use the functionality with pkg_deb — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#53 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHAQLN2VLKNVAUNE323STVTQLANCNFSM4IBLVJIA> .

[terinjokes]

I ended up calling out to dpkg-name as part of my releases pipeline, so I was running into the normal issues with the version not being available at analysis time.

[aiuto]

Version should be easy. Timestamp is the hard part.

…

On Mon, Dec 7, 2020 at 11:28 PM Terin Stock ***@***.***> wrote: I ended up calling out to [dpkg-name[( https://manpages.debian.org/buster/dpkg-dev/dpkg-name.1.en.html) as part of my releases pipeline, so I was running into the normal issues with the version not being available at analysis time. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#53 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHCL4O6FYJUKLNZPD2TSTWTPRANCNFSM4IBLVJIA> .

[jheaff1]

I too am interested in this functionality, specifically being able to produce a tar file that contains version info provided by the workspace status. Eg: my-tarball-1.2.3.tar.gz

[aiuto]

I'm going to declare this fixed. It was not about "stamping" as the other bazel rules use it (time stamping of the artifacts). This was about injecting variable names into the output artifact name.
The examples in https://github.com/bazelbuild/rules_pkg/tree/main/examples/naming_package_files
show how you can inject a variable from the command line into the names.

The original request was to extract variables form the workspace status output. I'm not going to build that because it is impossible to do it well. You have to parse the file during the execution phase, while the techniques we already have let you get the variable values at analysis time. Having it at analysis time lets us pass the information up in providers so other rules can act on the output sensibly.

Part of this discussion talked about time stamping, which is a different need. #288 adds an implmentation of --stamp and the stamp attribute for pkg_tar. As soon as that is in I can add it for the others.

[alokpr]

If package version is available in workspace-status files, using pkg_deb is still too cumbersome unless I am missing something. I need to provide the same version string via two mechanisms:

1. version_file
2. package_file_name + PackageVariablesInfo

If it is possible for pkg_deb to get the version number from version_file when constructing the package_file_name, it would make the most common use-case very convenient.

[aiuto]

It is not possible to get elements of the file name from the version_file. That would require reading the file at analysis time. The way we will have to work this is to generate the version_file from PackageVariablesInfo, or from attributes in the rule instance.

[alokpr]

Thanks for the quick response!

Would it be possible to provide an example for how to use pkg_deb if version is available in workspace-status files? Most (if not all) use-cases do not need fully custom file names. We just need a properly named output file.

[aiuto]

Why would your version ever be available in the workspace status file? If you want to specify it externally to the BUILD files, define a string flag and set it from the command line. Then write a rule that takes the flag and writes the version file. Take a look at https://github.com/bazelbuild/rules_pkg/blob/41a00ee70e7ec5eafe77e9fc21e042fde39a5339/examples/naming_package_files/BUILD#L121 for an example of creating the flag. From that you could write a rule that also gets the flag value and the does ctx.actions.write to create the version file.

…

On Tue, Jul 13, 2021 at 12:57 PM Alok Priyadarshi ***@***.***> wrote: Thanks for the quick response! Would it be possible to provide an example for how to use pkg_deb if version is available in workspace-status files? Most (if not all) use-cases do not need fully custom file names. We just need a properly named output file. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#53 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHAQ7WIOOHPZNHMX2ODTXRV5XANCNFSM4IBLVJIA> .

[alokpr]

I think it is pretty common to use --workspace_status_command to generate version number based on git tags:

bazel build --workspace_status_command="echo STABLE_PKG_VERSION $(git describe)"

Generating a version file is trivial using a genrule. I guess you are suggesting a custom rule instead of the genrule that would do both - provide PackageVariablesInfo and generate version file?

[aiuto]

workspace_status is probably the worst way to pass build time variables like the package name. Put them on the command line as string flags. That makes them available during analysis instead of execution time.

[alokpr]

The nice thing about workspace_status is that it can be added to workspace bazelrc which works for both - developers and CI - without any additional setup. I am not sure if CLI flags can be made to work without a wrapper script around bazel.

If it is not possible to produce a properly-named deb file, I guess another option is to rename the file before uploading to the apt repository. I will look into adding this step to the upload script.

[aiuto]

But the bottom line is that there is a basic bazel limitation to using the workspace file to name an output file. So don't waste your time trying to make that work.

…

On Tue, Jul 13, 2021, 3:40 PM Alok Priyadarshi ***@***.***> wrote: The nice thing about workspace_status is that it can be added to workspace bazelrc which works for both - developers and CI - without any additional setup. I am not sure if CLI flags can be made to work without a wrapper script around bazel. If it is not possible to produce a properly-named deb file, I guess another option is to rename the file before uploading to the apt repository. I will look into adding this step to the upload script. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#53 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHHNR5MOZOOVRDVTXUDTXSJBTANCNFSM4IBLVJIA> .

[alokpr]

Understood - thanks for the pointers!