[Bug]: Templating logic change in garden 0.13.46 breaks deploys

[alexkuretz]

Bug

Current behavior

Error from server (Invalid): error when creating "STDIN": Ingress.networking.k8s.io "${this.name}-0" is invalid: [metadata.name: Invalid value: "${this.name}-0": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.rules[0].host: Invalid value: "${providers.kubernetes.outputs.default-hostname}": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.rules[0].http.paths[0].backend.service.name: Invalid value: "${this.name}": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name',  or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')]
Error from server (Invalid): error when creating "STDIN": Ingress.networking.k8s.io "${this.name}-1" is invalid: [metadata.name: Invalid value: "${this.name}-1": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.rules[0].host: Invalid value: "${providers.kubernetes.outputs.default-hostname}": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.rules[0].http.paths[0].backend.service.name: Invalid value: "${this.name}": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name',  or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')]
Error from server (Invalid): error when creating "STDIN": Service "${this.name}" is invalid: [metadata.name: Invalid value: "${this.name}": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name',  or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?'), spec.ports[0].targetPort: Invalid value: "${var.port}": must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-), spec.selector: Invalid value: "${this.name}": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]
Error from server (BadRequest): error when creating "STDIN": Deployment in version "v1" cannot be handled as a Deployment: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'

Expected behavior

To work the way it has

Reproducible example

Not entirely sure, so here's some snippets.
project.garden.yml:

variables:
  service: task
  deploy-extra-env-vars: []

environments:
  - name: dev
    defaultNamespace: user-${local.username}
    variables:
      buildversion: user-${local.username}
      default-hostname: ${var.service}-user-${local.username}.dev.mycompany.com

manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ${this.name}-0
  annotations:
    ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/enable-cors: "true"
spec:
  ingressClassName: nginx
  rules:
    - host: ${providers.kubernetes.outputs.default-hostname}
      http:
        paths:
          - pathType: Prefix
            path: "/v1"
            backend:
              service:
                name: ${this.name}
                port:
                  number: 80

Workaround

Reverting to 0.13.45 works.

Suggested solution(s)

Additional context

Your environment

• OS: Ubuntu
• How I'm running Kubernetes: EKS

garden version
0.13.46

[stefreak]

Thank you @alexkuretz, we'll have a look and working on this with high priority

[stefreak]

@alexkuretz can you also share a snippet of the module or action code? Thank you

[stefreak]

@alexkuretz never mind, we understand the problem now.

Would you be able to verify that the change in #6713 solves the problem for you?

You can download garden for this PR here:

• windows and linux: https://app.circleci.com/pipelines/github/garden-io/garden/28003/workflows/0af57e96-57d4-4cb3-b619-87486288b18e/jobs/587310/artifacts
• macos: https://app.circleci.com/pipelines/github/garden-io/garden/28003/workflows/0af57e96-57d4-4cb3-b619-87486288b18e/jobs/587336/artifacts

Thank you so much for reporting this @alexkuretz .

[stefreak]

@alexkuretz now you can update to edge bonsai using garden self-update edge-bonsai. If possible, let us know if you are still experiencing issues with that version.

[alexkuretz]

This works, thank you. I'm reading through the discussion in the PR, are we doing something in an unorthodox way? What's the guidance on the "proper" way to be handling variables in the manifests? I see mention of patchManifests which we aren't currently using, we do make use of patchResources.

[alexkuretz]

Search in garden docs shows no mention of patchManifests?

[stefreak]

Hi @alexkuretz thank you, sorry that was a typo on my end, I mean patchResources!

You are not doing something unorthodox, but there are some issues with the current implementation of template string evaluation in kubernetes manifest files, for example there is no error if a user references a variable that does not exist, and the value of variables referenced in the kubernetes manifest is not correctly taken into account in version calculation.

Fixing this requires breaking changes, so we will only be able to fully resolve these issues in 0.14; in the mean time you can use inline manifests using manifests instead of file, or you can use the patchResources feature. More information will follow in the future.

[stefreak]

There was another regression in the template parser that could cause issues like this, so I'm reopening this

[alexkuretz]

Ok thanks for the explanation and quick action on the fix.

[alexkuretz]

One of our devs is trying the edge-bonsai release and encountering a different error that doesn't happen on 0.13.45

Command "/home/user/.garden/tools/mutagen/722d0c1941ba875e/mutagen sync terminate k8s--build-sync--dev--project-user-username--py-dev--itqj4us1" failed with code 1:

Error: unable to connect to daemon: client/daemon version mismatch (daemon restart recommended)

He tried cleaning namespace and re-running but same failure.

[vvagaytsev]

One of our devs is trying the edge-bonsai release and encountering a different error that doesn't happen on 0.13.45

Command "/home/user/.garden/tools/mutagen/722d0c1941ba875e/mutagen sync terminate k8s--build-sync--dev--project-user-username--py-dev--itqj4us1" failed with code 1:

Error: unable to connect to daemon: client/daemon version mismatch (daemon restart recommended)

He tried cleaning namespace and re-running but same failure.

@alexkuretz that happened because of the Mutagen version upgrade (Mutagen is the tool that we use for file syncing).

Did any warn message with hints appear in the console? If not, please let us know.

The issue could be fixed by manual Mutagen daemon restart by running the command:

garden util mutagen daemon stop

from the project root directory.

If the command above doesn't help, please try to kill all active Mutagen processes by running

kill -9 $(pgrep mutagen)

[th-ao]

Dev here. Restarting the daemon fixed it for me.

I didn't see any warn other than what Alex posted above i.e. daemon restart recommended.

This is the output I received:

ℹ build.py                  → Syncing build context to cluster...
✖ build.py-dev              → Failed (took 100.3 sec)
✖ build.py-dev              → Failed processing Build type=container name=py-dev (took 118.72 sec). This is what happened:

────────────────────────────────────────────────────────────────────────────────────────────────────────
Command "/home/user/.garden/tools/mutagen/722d0c1941ba875e/mutagen sync terminate k8s--build-sync--dev--project-user-username--py-dev--nxdjhyq0" failed with code 1:

Error: unable to connect to daemon: client/daemon version mismatch (daemon restart recommended)
────────────────────────────────────────────────────────────────────────────────────────────────────────
✖ build.py                  → Failed (took 99.8 sec)
✖ build.py                  → Failed processing Build type=container name=py (took 120.27 sec). This is what happened:

────────────────────────────────────────────────────────────────────────────────────────────────────────
Command "/home/user/.garden/tools/mutagen/722d0c1941ba875e/mutagen sync terminate k8s--build-sync--dev--project-user-username--py-dev--nxdjhyq0" failed with code 1:

Error: unable to connect to daemon: client/daemon version mismatch (daemon restart recommended)
────────────────────────────────────────────────────────────────────────────────────────────────────────

An error occurred while running step 2/4.

2 requested build action(s) failed!

[stefreak]

Thank you for letting us know @th-ao we'll also fix that

[stefreak]

@alexkuretz we just released 0.13.47 with these fixes, let us know if you encounter any further issues ❤️