Provide capability to rename archives based on configuration values

[aiuto]

This is to address #101 #193 and maybe help with #143

The goal is to allow all packaging rules to precisely name their output files in a way that is consistent with the naming conventions of the scheme, even if that requires pieces of the file name which can not be known until build time, such as cpu type.

The basic method is:

• allow all rules to specify the name of the output archive with package_file_name, rather than using a default
• create a PackageVarialbesInfo provider which specifies a map of values to replacements (e.g. {cpu:x86_64, copt:debug})
• if apkg_ rule target adds a package_variables attribute (which must provide PackageVariablesInfo), the provided dictionary will be used for substitutions in package_file_name
• pkg_ rules will also return a PackageArtifactInfo provider which will map target label to the file name actually generated. This is need in some situations when associated rules must know the exact name of the final package.

If a target does not use package_file_name then the out attribute is created as before (package + "." + extension).
If it does, then out will be a symlink to the name specified by package_name.

This PR only deals with pkg_tar. The other package types, and substitution into the spec files will come in followup changes.

[aiuto]

cc: @davschne @konste @KaoruDev

[konste]

I'd like to clarify the logic behind the new attributes - package_file_name and naming_info.
When neither of the two is specified the name of the rule is still used as the name of the package - this is for backwards compatibility.
Now what if only package_file_name is specified? This means instead of the name of the rule we need to take the string specified by package_file_name as the base for the package name. In many cases it is going to be sufficient and people don't need to mess with the new provider at all.
When both package_file_name and naming_info are specified it means: 1. name of the package should be taken from package_file_name and 2. package_file_name and other string typed attributes of the rule should be interpreted as the templates subject to expansion from the dictionary provided by naming_info.

Important points:

1. package_file_name can be specified without naming_info.
2. naming_info should affect not only package name, but also other string attributes. package_dir is especially important.

Do you agree?

[aiuto]

On Sun, Jul 12, 2020 at 7:42 PM Konstantin Erman ***@***.***> wrote: I'd like to clarify the logic behind the new attributes - package_file_name and naming_info. When neither of the two is specified the name of the rule is still used as the name of the package - this is for backwards compatibility. Now what if only package_file_name is specified? This means instead of the name of the rule we need to take the string specified by package_file_name as the base for the package name. In many cases it is going to be sufficient and people don't need to mess with the new provider at all.

Yes. package_file_name should be sufficient for many use cases. You only need a rule to provide naming_info if you need dynamic naming based on configuration. This was your use case, and probably more cases as people build for more execution platforms.

When both package_file_name and naming_info are specified it means: 1. name of the package should be taken from package_file_name and 2. package_file_name and other string typed attributes of the rule should be interpreted as the templates subject to expansion from the dictionary provided by naming_info.

Yes. That is the idea.

Important points: 1. package_file_name can be specified without naming_info. 2. naming_info should affect not only package name, but also other string attributes. package_dir is especially important. Do you agree?

It should work wherever it is reasonable. For tar and zip files, it would only be the file name. For RPM, deb, and yet unsupported formats, we may need to be able to work that into the expansion of the control files. The first implementations may only get a few of those but over time we can make it uniform. —

…

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHEH4N6LGYFKXTNICE3R3JC7BANCNFSM4OV46B7Q> .

[konste]

For tar and zip files, it would only be the file name.

Wait, wait, what about package_dir attribute? Without it being dynamically configurable pkg_tar would still be unusable in our case. I thought you want to get rid of package_dir_file or at least deprecate using it and dynamically configurable package_tar would be a good replacement.

[aiuto]

cc/ @nacl My question is if package_dir is the right abstraction. Should we have the packaging rules to the package redirection or should pkg_filegroup be the place where that happens?

…

On Sun, Jul 12, 2020 at 10:41 PM Konstantin Erman ***@***.***> wrote: For tar and zip files, it would only be the file name. Wait, wait, what about package_dir attribute? Without it being dynamically configurable pkg_tar would still be unusable in our case. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHHVCULX2YRLD3BNLS3R3JX5JANCNFSM4OV46B7Q> .

[konste]

Oh, this is deeper than I was thinking about it. When I discovered package_dir I liked it because it is easy to understand and does exactly what one would expect it to do. I was able to make it do what I need from the first try. When I figured we need it dynamic I dug further and found package_dir_file. It was not an elegant solution, but it worked and unblocked us, so ok.

Now you are talking about matters like package redirection and pkg_filegroup which I simply not aware of.
It would be a pity if in order to make things right we would make them harder to learn and use.
Hope we can keep an easy way for those who need just that and more powerful API for those who need more.

[aiuto]

I want the fewest number of constructs that work similarly across all the rules. If we want to build a binary that goes in /usr/lib/foo, it would be better to say "prefix everything with /usr/lib/foo" in one place (or at least with one syntax) rather than in every use of that place.

[konste]

Unfortunately I don't quite understand your example. package_dir is about internal folder structure of the archive. I understand that there better be one way to specify that info across all packaging rules, but I am not aware of other possible ways besides package_dir.

[nacl]

cc/ @nacl My question is if package_dir is the right abstraction. Should we have the packaging rules to the package redirection or should pkg_filegroup be the place where that happens?

Sorry for the realllly long delays in responding to this.

In the purest possible sense, pkg_filegroups are intended to define structure, while the packaging rules themselves would be most interested in actually consuming pkg_filegroups and emitting various per-package metadata.

Something like a package_dir attribute is definitely convenient, though: it saves the package authors from potentially writing additional pkg_filegroup rules to define package prefixes. As pkg_filegroup currently stands, we would need something like it (since it's not nestable), but if we happened to have something like what's mocked in #212, packaging rule attributes like package_dir attribute won't be necessary.

The pkg_filegroup design, therefore, should be given some strong scrutiny to make sure we get all the use cases ironed out.

[nacl]

I like the ideas behind this, but I think the templating framework could be used by pkg_filegroups too.

[aiuto]

Yes. I tend to forget the 'Info' when speaking about providers. It feels redundant (and slightly wrong) unless adding "information" to the description makes sense.

…

On Fri, Aug 28, 2020 at 11:58 AM Andrew Psaltis ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In pkg/providers.bzl <#198 (comment)>: > +PackageNamingInfo = provider( + doc = """Metadata about a package artifact.""", + fields = { + "values": "Dict of name/value pairs", + }, +) Not a bad idea, although it wouldn't mesh all that well with the standard that providers are named Info. On further thought, would it make sense to use the builtin TemplateVariableInfo <https://docs.bazel.build/versions/master/skylark/lib/TemplateVariableInfo.html> provider instead? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHBBPDVYZKVSCSJTJFTSC7HZHANCNFSM4OV46B7Q> .

[aiuto]

I have something now that I think I want to proceed with.

It demonstrates 3 key new features:

• to have the file name created by a pkg_ rule be different from the label name
• to allow variable substitution into that file name, where the variables can dynamically come from the build configuration
• That downstream rules can, by default, consume the archive produced by a package using the label name even if the actual filename is different.

This PR does not

• address the package_dir and package_dir_file attributes. Those impact the paths created inside the archive. That is a wholly different set of features. That should be handled in pkg_filegroup.
• do anything expect pkg_tar. There is no point doing it everywhere before we nail down design.
• try to dynamically use anything except the target cpu. We can address other platform and toolchain provided values in followup CLs as needed.
• deal with variables in rpm .spec file expansion. That should be relatively trivial one the infrastructure here is in place.

[nacl]

I like where this is going.

I think this needs a little cleanup and clarification on some points before proceeding. Documentation and testing can come later if you'd like.

[aiuto]

Thanks for the review. Back to you now.

[nacl]

Changes look good.