diff --git a/enhancements/builds/volume-mounted-resources.md b/enhancements/builds/volume-mounted-resources.md new file mode 100644 index 00000000000..08bed5b0b30 --- /dev/null +++ b/enhancements/builds/volume-mounted-resources.md @@ -0,0 +1,297 @@ +--- +title: volume-mounted-resources +authors: + - @bparees + - @adambkaplan +reviewers: + - @derekwaynecarr + - @smarterclayton +approvers: + - @bparees +creation-date: 2020-01-08 +last-updated: 2021-04-15 +status: implementable +--- + +# Volume Mounted Resources + +## Release Signoff Checklist + +- [x] Enhancement is `implementable` +- [x] Design details are appropriately documented from clear requirements +- [x] Test plan is defined +- [x] Graduation criteria for dev preview, tech preview, GA +- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/) + +## Open Questions [optional] + +1. Can we make this the default or even only behavior for builds? + No, need to make it opt-in to avoid potentially breaking existing buildconfigs. +2. What happens if a user overrides the default volume mounts used by Builds today? + We override the default with the sources chosen in the BuildConfig volumes array. + +## Summary + +Today builds support getting source input from configmaps and secrets, hereafter referred to as "resources". +When users utilize this feature, the resource is volume-mounted into the build pod and placed in the "build context" within the build's execution environment, alongside other build sources like git source code. +The next steps depend on whether it is an s2i or dockerfile build. + +For s2i builds, the generated Dockerfile contains commands to `ADD` the content at a path specified by the user, the assemble script is invoked, and then the injected content is zeroed out prior to committing the image via a `RUN rm` command added to the Dockerfile. + +For dockerfile builds, the user is instructed to add appropriate `ADD` and `RUN rm` commands to their dockerfile to inject the content that is available in the build's working directory (along with their application source, where applicable). + +There are a few undesirable aspects to this: + +1. In the dockerfile case, the content can still be found in lower layers of the image unless a layer squashing option is selected. +2. Requires extra work by the user in the Dockerfile, so each Dockerfile must be customized + +This enhancement proposes to introduce an option to use buildah's capability to mount a volume at build time. +The content mounted into the build pod would be then mounted into the container processing the Dockerfile, making that content available within the container so Dockerfile commands could reference it. +No explicit `ADD` would be required, and since mounted content is not committed to the resulting image, no `RUN rm` equivalent is required to clean up the injected content. + +To avoid security and lifecycle concerns, the following volume types will be supported initially: + +1. Secrets +2. ConfigMaps +3. `csi` - this is to enable use of [Projected Resource CSI driver](/enhancements/cluster-scope-secret-volumes/csi-driver-host-injections.md) volumes. + + +## Motivation + +### Goals + +* Simplify how users consume secret + configmap content in builds +* Increase the security of protected content being injected to images +* Simplify use cases that require consuming credentials during the build, but need to ensure those credentials do not end up in the output image. +* Eventually extend this api to allow the mounting of traditional volumes (such as those backed by persistent storage) + +### Non-Goals + +* This enhancement should not result in a change of behavior for users of the existing secret/configmap injection api. +* Provide immediate support for persistent volume claims in builds. + This is a long term goal that will be addressed in a future enhancement proposal. + + +## Proposal + +### User Stories [optional] + +The enabled use cases are essentially identical to what can be done with the configmap/secret input api in builds today, but with a better user experience and security as discussed above +It does not enable a new use case that is not already possible today, except that layer squashing will not be required. + +Future extensions to this enhancement could enable the use case of providing build input content from a persistent volume and allowing the build to store/cache content for future builds on such a volume. +Those will be discussed in the future enhancement at that time. + +### Implementation Details/Notes/Constraints [optional] + +We will need to introduce a new mechanism in the build api which allows the user to indicate that they want to inject "volume" content into the build. +Initially the only allowed volume types will be ConfigMaps, Secrets, and ephemeral CSI volumes. +The api will otherwise be similar to the existing secret/configmap injection api in which users identify the configmap/secret and the target path for injection. + +This will be done by adding our own `Volume[]` field to the Source and Docker strategy structs. +The `Volume[]` field will allow defining volumes to be mounted to the build pod in the same way that a normal pod allows for this. +However, the types of volumes that can be defined will be restricted to those explictly supported by Builds. +Similarly a `VolumeMount[]` field will be added, but without the MountPropagation and SubPath fields. +MountPropagation and SubPath can be considered for support in the future. + +These fields will be wired, via the build controller, directly to the build pod that is constructed. +In addition all mounts into the pod will be done at a path of our choosing, not the VolumeMount path specified, to ensure the user cannot overwrite critical function inside the build pod and use it as an escalation pathway. +Example - all volumes are mounted under `/var/run/openshift.io/volumes` in the build pod containers. + +The logic that invokes buildah will then pass the mounted directories as transient volume mount arguments. +The mount path provided to buildah will be determined from the VolumeMount specification. +Arguments such as "read only" will also be inferred. + +Builds today have default mounts that are provided via other means: + +1. Entitlement data from the node, mounted at `/run/secrets/etc-pki-entitlement`, `/run/secrets/redhat.repo`, and `run/secrets/rhsm` inside the build pod. +2. The cluster trust bundle - this is optionally mounted at `/etc/pki/ca-trust` via the BuildConfig spec's `mountTrustedCA` option. + +If a build volume's mount path conflicts with any of the paths specified above, the build volume mount will override the default. + +Proposed api/structs: +Note: DockerBuildStrategy will be updated in the same way. + +```go +// SourceBuildStrategy defines input parameters specific to an Source build. +type SourceBuildStrategy struct { + // From is reference to an DockerImage, ImageStream, ImageStreamTag, or ImageStreamImage from which + // the docker image should be pulled + From kapi.ObjectReference + + // PullSecret is the name of a Secret that would be used for setting up + // the authentication for pulling the Docker images from the private Docker + // registries + PullSecret *kapi.LocalObjectReference + + // Env contains additional environment variables you want to pass into a builder container. + Env []kapi.EnvVar + + // Scripts is the location of Source scripts + Scripts string + + // Incremental flag forces the Source build to do incremental builds if true. + Incremental *bool + + // ForcePull describes if the builder should pull the images from registry prior to building. + ForcePull bool + + // Volumes is a list of volumes that can be mounted by the build + // Only a subset of Kubernetes Volume sources are supported by builds. + // More info: https://kubernetes.io/docs/concepts/storage/volumes + Volumes []BuildVolume + + // VolumeMounts are volumes to mount into the build + VolumeMounts []VolumeMount +} +``` + +```go +// BuildVolume describes a volume that is made available to build pods, such that it can be mounted into the build execution environment. +// Only a subset of Kubernetes Volume sources are supported by builds. +type BuildVolume struct { + + // Volume's name. + // Must be a DNS_LABEL and unique within the pod. + // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + Name string `json:"name" protobuf:"bytes,1,opt,name=name"` + + // BuildVolumeSource represents the location and type of the mounted volume. + BuildVolumeSource `json:",inline" protobuf:"bytes,2,opt,name=volumeSource"` +} + + +// Represents the source of a volume to mount. +// Only one of its members may be specified. +// This is a subset of the core Kubernetes VolumeSource. +type BuildVolumeSource struct { + // Secret represents a secret that should populate this volume. + // More info: https://kubernetes.io/docs/concepts/storage/volumes#secret + // +optional + Secret *kapi.SecretVolumeSource `json:"secret,omitempty" protobuf:"bytes,1,opt,name=secret"` + + // ConfigMap represents a configMap that should populate this volume + // +optional + ConfigMap *kapi.ConfigMapVolumeSource `json:"configMap,omitempty" protobuf:"bytes,2,opt,name=configMap"` + + // CSI (Container Storage Interface) represents ephemeral storage that is handled by certain external CSI drivers (Beta feature). + // +optional + CSI *kapi.CSIVolumeSource `json:"csi,omitempty" protobuf:"bytes,3,opt,name=csi"` +} +``` + +```go +// VolumeMount describes a mounting of a Volume within the build execution environment. +type VolumeMount struct { + // This must match the Name of a Volume. + Name string + // Mounted read-only if true, read-write otherwise (false or unspecified). + // Defaults to false. + // +optional + ReadOnly bool + // Path within the build environment at which the volume should be mounted. Must + // not contain ':'. + MountPath string +} +``` + +Example usage (use of the existing secret/configmap injection api is included for comparison, it is not changing) + +```yaml +apiVersion: v1 +items: +- apiVersion: build.openshift.io/v1 + kind: BuildConfig + metadata: + name: mybuild + namespace: p1 + spec: + strategy: + sourceStrategy: + from: + kind: ImageStreamTag + name: nodejs:10-SCL + namespace: openshift + volumes: + - name: secret + secret: + secretName: somesecret + - name: config + configMap: + name: someconfigmap + items: + - key: somekey + path: volume/path/value.txt + - name: content-access + csi: + driver: csi-driver-projected-resource.openshift.io + volumeAttributes: + share: etc-pki-entitlement + volumeMounts: + - name: config + mountPath: /tmp/config + - name: secret + mountPath: /tmp/secret + - name: content-access + mountPath: /etc/pki/entitlement + type: Source + source: + secrets: + - secret: + name: myOtherSecret + destinationDir: /tmp/othersecret + configMaps: + - configMap: + name: myOtherConfigMap + destinationDir: /tmp/otherconfig +``` + +### Risks and Mitigations + +Since the build pod is privileged, we need to ensure that users cannot abuse this api to trick the build controller into creating build pods that can exploit those privileges. +This means ensuring that any volume mount specifications the user provides, which are translated into volumemounts in the build pod, cannot be used to alter the build logic. +To this end, we should explicitly control where the volumes are mounted within the build pod.(We can mount them anywhere the user specifies within the buildah container). + +We also need to ensure that the user can't use this api to inject/mount content that they could not normally mount into a pod they created themselves. +For this reason we must explicitly disallow `HostPath` volume types, for example. +We will mitigate this by only supporting a subset of VolumeSources that can be mounted into builds. +As additional types are requested, we will need to determine it is safe to add them. +Gating will also ensure that we address any volume lifecycle concerns (for example - binding of PersistentVolumeClaims). + +## Design Details + +### Test Plan + +This feature will need new e2e tests that leverage the new api option. +We have existing tests for configmap+secret injection would should be able to be copied+adapted to testing this feature relatively easily. +Tests for the CSI volume may have to be done independently of the core OpenShift test suite at outset. + + +### Graduation Criteria + +This should be introduced directly as a GA feature when it is implemented. + +### Upgrade / Downgrade Strategy + +N/A + +### Version Skew Strategy + +N/A + +## Implementation History + +Major milestones in the life cycle of a proposal should be tracked in `Implementation +History`. + +## Drawbacks + +Additional user complexity in choosing when to enable this behavior. +It is also unfortunate we can't default it because it will be a better choice for most users. + +## Alternatives + +Updating the existing injection apis to have a "asVolume" field was considered as it would be a simpler implementation (more code reuse) but it was rejected as there is a long term goal to allow builds to mount traditional volumes as well. +The existing injection api can't easily be extended to support such a thing, so the design proposed in this enhancement is a better stepping stone to that goal. +This also puts us on a better path to support using volumes for caching build content between build invocations which has been a longtime goal of the build api. +