[Feature]: App Environment Variables should be available during staging

[tcdowney]

Blockers/Dependencies

• Related: [Feature]: App Developer can see Service Credential Bindings projected onto running workloads via VCAP_SERVICES #462

Background

Cloud Foundry allows developers to configure environment variables for apps and exposes them during both staging and running contexts. Additionally CF also includes any bound services under the CF_SERVICES env var and has its own set of additional system env vars. The scope of this story is not to backfill or set any system env vars that are missing, but to provide the same env vars that we set for running apps right now to the kpack Image.

Some of the running env vars (such as PORT) don't make sense for staging so omit those. Cross check with this table to see what existing env vars in CF for VMs are present during running vs staging.

As an App Developer
I want my environment variables to be present during staging
So that any buildpacks that require environment variables have access to them

---

Acceptance Criteria

User-specified env vars

GIVEN I have set environment variables on my app
WHEN I run cf push or cf restage for my app
THEN I see, on the kpack Image and build Pod, environment variables that correspond with what I've set

---

VCAP_SERVICES env vars

Note: We did this partially in #463 via volume mounts, but did not configure VCAP_SERVICES.

GIVEN I have bound several user-provided service instances to my app
WHEN I run cf push or cf restage for my app
THEN I see, on the kpack Image and build Pod, an environment variable called VCAP_SERVICES with an encoded JSON string containing those credentials. See #462 for the structure of this.

---

Dev Notes

• We've done this before in cf-for-k8s Cloud Controller code, but we explicitly omitted VCAP_SERVICES (see this discussion). I think this may have been too future looking since although many new Cloud Native Buildpacks use K8s ServiceBinding spec volume mounts, any shimmed legacy buildpacks or custom buildpacks are likely still looking for VCAP_SERVICES.

Resources

• CF Docs: https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html
• cf-k8s-controllers code for creating the kpack Image: https://github.com/cloudfoundry/cf-k8s-controllers/blob/734e60bdffd7b977bff5ee3132fd9314bf7f712a/controllers/controllers/workloads/cfbuild_controller.go#L248
• cf-k8s-controllers code for env vars for running apps: https://github.com/cloudfoundry/cf-k8s-controllers/blob/734e60bdffd7b977bff5ee3132fd9314bf7f712a/controllers/controllers/workloads/cfprocess_controller.go#L303
• Cloud Controller code for Diego env vars(this codepath is messy, might be easier to just experiment on CF for VMs or look at the tests): https://github.com/cloudfoundry/cloud_controller_ng/blob/e1abc405612b682513728df3e13377dd9b3d7361/lib/cloud_controller/diego/task_environment.rb#L21

[tcdowney]

So this story was making me think a bit more about how we're essentially putting secret data (service bindings often contain connection strings and passwords) into plaintext by converting them into VCAP_SERVICES and putting them on the Eirini LRP and (in this story) the kpack Image.

I doubt this is the right place to do this, but one thought I had was what if we had something continuously reconciling all the CFServiceBindings for an app and building those up into a single "vcap-services-" secret that contains the JSON-ified version of those. We could pass that directly to the kpack Image since it uses core EnvVar types: https://github.com/pivotal/kpack/blob/main/pkg/apis/build/v1alpha2/image_types.go#L67

Then I think with some enhancements to the Eirini LRP (make it use the k8s EnvVar type instead of this string map) you could wire that through on eirini-controller as well for running apps.

[tcdowney]

I created an explicit issue for what I was talking about above ^^
#796

It's a lower priority than some of our other stuff, but I welcome feedback on it if anyone wants to take a look.

[danail-branekov]

Then I think with some enhancements to the Eirini LRP (make it use the k8s EnvVar type instead of this string map) you could wire that through on eirini-controller as well for running apps.

Follow up issues created to address this:
#924
#925