Resolving Cloud Tokens During Synthesis

[iliapolo]

---

• Hit the like (👍) or dislike (👎) button to let us know what you think about this feature.
• Feel free to ask any clarifying questions or suggest enhancements.
• This is just a proposal and intentionally doesn’t focus on technical details but rather on customer experience. If the proposal is accepted we will publish a detailed RFC where you’ll get a chance to provide more feedback.

---

Kubernetes applications often leverage cloud infrastructure for their operation. For example, a CronJob may perform some periodic computation and persist the results in an S3 bucket. We want customers to be able to easily define such applications with the various CDKs. In case the bucket name is explicitly defined, i.e known as synth time, this is already possible. For example, with the AWS CDK:

import * as aws from 'aws-cdk-lib';
import * as k8s from 'cdk8s';
import * as kplus from 'cdk8s-plus-25';

const k8sApp = new k8s.App();
const awsApp = new aws.App();

const stack = new aws.Stack(awsApp, 'Stack');
const manifest = new k8s.Chart(k8sApp, 'Manifest');

const bucketName = 'my-bucket';

// define the bucket with a well known name
new aws.aws_s3.Bucket(stack, 'Bucket', {
  bucketName
});

// pass the bucket name to the CronJob container
new kplus.CronJob(manifest, 'CronJob', {
 containers: [{ 
   image: 'job',
   envVariables: {
     BUCKET_NAME: kplus.EnvValue.fromValue(bucketName),
   }
 }]
});

k8sApp.synth();
awsApp.synth();

This application will synthesize a CloudFormation template, and a Kubernetes manifest:

"Bucket83908E77": {
  "Type": "AWS::S3::Bucket",
  "Properties": {
  "BucketName": "my-bucket"
  },
  "UpdateReplacePolicy": "Retain",
  "DeletionPolicy": "Retain"
}

apiVersion: batch/v1
kind: CronJob
metadata:
  name: manifest-cronjob-c86481e8
spec:
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            cdk8s.io/metadata.addr: Manifest-CronJob-c89809bc
        spec:
          containers:
            - env:
                - name: BUCKET_NAME
                  value: my-bucket
              image: job

At this point, customers can deploy each of these independently, with whatever engine they like. Conceptually, running cdk deploy && kubectl apply will suffice.

However, AWS CDK best practices actually advise against explicitly naming resources since it introduces limitations on resource replacements. In this case, our application will look like this:

import * as aws from 'aws-cdk-lib';
import * as k8s from 'cdk8s';
import * as kplus from 'cdk8s-plus-25';

const k8sApp = new k8s.App();
const awsApp = new aws.App();

const stack = new aws.Stack(awsApp, 'Stack');
const manifest = new k8s.Chart(k8sApp, 'Manifest');

// define the bucket without specifying a name
const bucket = new aws.aws_s3.Bucket(stack, 'Bucket');

// pass the bucket name attribute to the CronJob container
new kplus.CronJob(manifest, 'CronJob', {
 containers: [{ 
   image: 'job',
   envVariables: {
     BUCKET_NAME: kplus.EnvValue.fromValue(bucket.bucketName),
   }
 }]
});

k8sApp.synth();
awsApp.synth();

The Kubernetes manifest will now be:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: manifest-cronjob-c86481e8
spec:
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            cdk8s.io/metadata.addr: Manifest-CronJob-c89809bc
        spec:
          containers:
            - env:
                - name: BUCKET_NAME
                  value: ${Token[TOKEN.603]}
              image: job

Now though, this manifest isn’t deployable, because it references an AWS CDK token, representing the bucketName attribute of the S3 Bucket resource. To bypass this problem, customers need to run a 3 phase deployment pipeline:

1. Deploy the cloud infrastructure using, for example, cdk deploy.
2. Fetch the necessary cloud resource attributes using, for example, with boto.
3. Pass attributes to cdk8s synth (via env variables or a file) to generate a manifest with concrete values.

Phase 2 is tricky to implement because it is inherently decoupled from the application, which means it doesn’t know which attributes need to be fetched, or which identifier to use to fetch them. Maintaining it requires constant manual coordination with the app, which is an error prone mechanism.

We propose to add capabilities into cdk8s that will enable it to interpret deploy time tokens, and then fetch their concrete values during cdk8s synthesis. We would like to support both AWS CDK and CDK For Terraform. These capabilities will allow customers a simple and straightforward workflow to deploy their entire application:

1. Deploy their cloud infrastructure (e.g cdk deploy)
2. Synthesize their cdk8s application (e.g cdk8s synth)
3. Deploy the manifests (e.g kubectl apply)

[heckler1]

Another potential way this or a very similar capability could be leveraged would be to use Docker image tags to fetch Docker image digests from an image repository (or local lock file), for deploying images into Kubernetes pinned to a specific digest shasum.