Relax restrictions on environment variable names.

[timoreimann]

Fixes #2707

The POSIX standard restricts environment variable names to uppercase letters, digits, and the underscore character in shell contexts only. For generic application usage, it is stated that all other characters shall be tolerated. (Reference here, my prose reasoning here.)

This change relaxes the rules to some degree. Namely, we stop requiring environment variable names to be strict C_IDENTIFIERS and start permitting lowercase, dot, and dash characters.

Public container images using environment variable names beyond the shell-only context can benefit from this relaxation. Elasticsearch is one popular example.

[k8s-ci-robot]

Hi @timoreimann. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[timoreimann]

/cc @mtaufen

[resouer]

/ok-to-test

[liggitt]

cc @kubernetes/api-reviewers

[smarterclayton]

So this makes env vars potentially unbounded? Special characters? Equals characters? Unicode? The null terminator?

I don't think we can allow = as that would alter how values are interpreted.

I'm not sure I'd start with "no restrictions" as the next step. I'd probably prefer to have a set of concrete additions, each backed by a concrete use case.

[redbaron]

@timoreimann , thank you very much doing it, it was a long standing headache for us.

@smarterclayton it is not for kubernetes to decide what is allowed and what is not, it should just pass whatever user asked for to an application.

> env - '{}&&!!55=__=1' /bin/bash -c 'printenv'
PWD=/tmp
SHLVL=1
{}&&!!55=__=1
_=/usr/bin/printenv

[timoreimann]

@smarterclayton yes, the environment variable namespace would be unbound in order to be in line with the standards recommendation I linked to previously. For the sake of convenience, here's the relevant section (emphasis by me):

Environment variable names used by the utilities in the Shell and Utilities volume of IEEE Std 1003.1-2001 consist solely of uppercase letters, digits, and the '_' (underscore) from the characters defined in Portable Character Set and do not begin with a digit. Other characters may be permitted by an implementation; applications shall tolerate the presence of such names.

Reading the section again, I realize I might have misinterpreted the passage: It looks to me now that those other characters which we should tolerate are limited to what the Portable Character Set gives. This would at least rule out unicode characters and maybe more obscure special characters.

Although one could argue that allowing the equal character simply moves the responsibility for passing non-ambiguous values onto the user, I can see how that could become a source of confusion. Keeping it on the black list may be the smarter approach indeed.

Regarding how to approach looser restrictions: My rationale was to solve this matter "once and for all". A minimalistic relaxation could lead to further feature requests and discussions as people start to use other characters not supported by Kubernetes. For instance, I think it's not too unrealistic that slashes could be employed in the future as a separation / hierarchy-building means.

The other argument is that IMHO Kubernetes should try to follow official POSIX recommendations, or otherwise have good reasons not to (the equal character being one such).

I can see two alternatives to accepting the PR in its current form:

1. Only add the dot character to the set of permissible characters, which is what dot character in environment variable name yields "invalid value" #2707 is strictly about.
2. Pick a reasonable sub-set from the Portable Character. All visible characters could be one such sub-set.

WDYT?

[timoreimann]

/test all

[smarterclayton]

No matter what, each expansion needs to be discussed thoroughly. We can't add "all" because we would never be able to restrict it in future versions. But we can grow. I'd recommend changing this Pr to have a new "environment variable name" validation fund that copies CIdentifier. Then we can add dot, and for 1.8 expand to a wider set with set of scoped additions. If we split the changes out we'll also have time to discuss some of the more problematic parts in the next sig-apps meeting. Is there a list of all env name related issues opened against Kube that we can review togrther? On Jul 16, 2017, at 4:56 PM, Timo Reimann <notifications@github.com> wrote: @smarterclayton <https://github.com/smarterclayton> yes, the environment variable namespace would be unbound in order to be in line with the standards recommendation I linked to previously. For the sake of convenience, here's the relevant section (*emphasis* by me): Environment variable names used by the utilities in the Shell and Utilities volume of IEEE Std 1003.1-2001 consist solely of uppercase letters, digits, and the '_' (underscore) from the characters defined in Portable Character Set and do not begin with a digit. *Other characters may be permitted by an implementation; applications shall tolerate the presence of such names.* Reading the section again, I realize I might have misinterpreted the passage: It looks to me now that those *other characters* which we should tolerate are limited to what the Portable Character Set <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap06.html#tagtcjh_3> gives. This would at least rule out unicode characters and maybe more obscure special characters. Although one could argue that allowing the equal character simply moves the responsibility for passing non-ambiguous values onto the user, I realize how that could become a source of confusion. Keeping it on the black list may be the smarter approach indeed. Regarding how to approach looser restrictions: My rationale was to solve this matter "once and for all". A minimalistic relaxation could lead to further feature requests and discussions as people start to use other characters not supported by Kubernetes. For instance, I think it's not too unrealistic that slashes could be employed in the future as a separation / hierarchy-building means. The other argument is that IMHO Kubernetes should try to follow official POSIX recommendations, or otherwise have good reasons not to (the equal character being one such). I can see two alternatives to accepting the PR in its current form: 1. Only add the dot character to the set of permissible characters, which is what #2707 <#2707> is strictly about. 2. Pick a reasonable sub-set from the Portable Character. All visible characters could be one such sub-set. WDYT? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48986 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_pxXWHyMkQjbJrRSmscuVhJO77qPrks5sOnkEgaJpZM4OZJPO> .

[timoreimann]

Alright, I'll simplify the PR accordingly.

I scanned through the open issues containing "environment variable" and only found #16863 which is about supporting dashes. Tim Hockin seems to have approved the feature request already.

I can't really tell if there are legitimate yet rejected requests among the closed issues though since there are just too many of them, and Github search seems way too fuzzy (or my search-fu is weak).

[liggitt]

Do nodes still run validation on pods? If so, loosening validation means a version skewed node couldn't run a pod using an envvar name containing a dot or dash for a couple releases (we have to consider nodes up to two releases older than the API server)

[liggitt]

Dash and dot seem like the most reasonable expansions if we're able to work through the impact of loosening validation

[timoreimann]

@liggitt kubelet checks whether pod environmental variables comply with the C_IDENTIFIER naming scheme. Is that what you are referring to when talking about node version skew?

How would we go about phasing in the relaxation two releases from now? Would we just announce the upcoming deprecation on the next release and only make the code change in 1.10? Or can we update the code today already with some kind of version guard?

Does the version compatibility guarantee only refer to the validation on the node, or do we have to make sure it covers all API operations?

Finally, do I understand correctly that we won't be able to allow dots and dashes until those two compatibility releases have elapsed?

Sorry for the tons of questions -- doing a backwards compatible feature is a new thing for me in Kubernetes.

[redbaron]

We can't add "all" because we would never be able to restrict it in future versions. But we can grow.

@smarterclayton , this exact thinking "let's restrict, then we can grow" lead to this PR and discussion and substantial amount of pain for users, didn't it? And now you are suggesting to repeat this approach, what for? So far problem is exactly opposite, k8s restricted it so much, that not every app can be run in kube, so lets solve exactly it. Problem is not that "." is not allowed, problem is that k8s tries to censor env var names, while it is not in the business of doing so.

@timoreimann , paragraph you are quoting is relevant to "utilities in the Shell and Utilities volume of IEEE Std 1003.1-2001" only. Which is

Definitions for a standard source code-level interface to command interpretation services (a "shell") and common utility programs for application programs

So if kube restricted itself to be running those utilities only, it would be a valid restriction, but kube meant to run arbitrary applications, therefore it should allow any env vars to be passed to an app, as defined by the standard.

[timoreimann]

@smarterclayton @liggitt I have reverted the PR to the previous state and extended the environment variable name validation logic to just tolerate dots and dashes (while keeping everything else unchanged).

Please take a look again and let me know what else needs to be done. Thanks!

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: smarterclayton, timoreimann

Associated issue: 2707

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• pkg/api/OWNERS [smarterclayton]
• pkg/kubectl/OWNERS [smarterclayton]
• pkg/kubelet/OWNERS [smarterclayton]
• staging/src/k8s.io/apimachinery/pkg/OWNERS [smarterclayton]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[timoreimann]

/retest

[k8s-github-robot]

Automatic merge from submit-queue (batch tested with PRs 50208, 50259, 49702, 50267, 48986)

[liuzhen9327]

@timoreimann
hi，timoreimann
I still have a problem, can you help me?
#52381