Improve Container Images Security and Sustainability

[tkapin]

Context and Motivation

Docker is a key component of our infrastructure that allow us to use well defined environment for building and testing the product. We need to have clarity about what images are used by the product teams, what components do these images contain and have tight control over the images lifecycle. This is necessary to ensure that all Docker images used in the product development lifecycle are secure, patched and don't contain software components with known security vulnerabilities.

Business Objectives

The business objectives of this epic fall essentially in two categories.

• Secure Supply Chain
• Engineering Efficiency and Sustainability

Secure Supply Chain

Mariner as the underlying distro for our Linux builds

The Mariner team produces "golden images" of Mariner Linux, guaranteed to be completely secure and well maintained environment. Building on Mariner would let us leverage this secure environment. If it is not possible to build only on Mariner, for example to support MUSL-specific distros (Alpine), we should aim to minimize the number of distros used for building the product to an absolute minimum.

• Mariner is used as the underlying distro for the product builds

From the security perspective, use of the 1ES environments and standard dotnet tasks for building is considered to be secure as well.

Improved Container Image Lifecycle

Currently, our container definitions are stable, but rarely updated, with some of the definitions dating back to 2020. We don't have means to ensure that all container images we use contain the latest OS patches and CVE fixes. One of the main points of this proposal is to ensure that the containers would be changing rapidly, accepting servicing updates form the OS on a regular basis.

• Our container images are re-built regularly and they contain the latest underlying OS patches and CVE fixes
• There is a process and tools implemented for auditing our infrastructure to identify images that are out of date
• Implemented process and tools to delete old container images (older than 3-6 months) from MCR

Deletion of old images from MCR would ensure we are not using outdated images in our infrastructure and help us to improve our MCR spending.

Improved Container Inventory Governance

Following-up on the lifecycle of the images, we also need to be able to reason about content of the individual images to ensure we are using only supported and secure versions of our dependencies.

• Design and implement processes and tooling to keep versions of container image dependencies on supported versions
• Identify what components are needed for what scenarios (e.g., Azure CLI or Node are not needed in some scenarios). Ideally, we would use only our SDK images for repos where it would be possible.
• Ensure the components in our images are installed from standard sources (such as the distro repos) where possible
• Implement checksum recording and validation for components that are installed from external locations (except for the distro repositories)
• Ensure that updating major versions of components that might affect the build process (e.g., clang, cmake, ...) is done explicitly through PRs
• Design and implement new tagging policy of our containers so that it is possible to consume latest OS patches without changing the image tag

Standardized Container Images Consumption by Product Repos

Due to historical reasons and individual repo owner needs, there are teams that don't use the standard container images from the docker buildtools repo. Ideally, all teams should use standardized images so that we could ensure the images are secure.

• Reach out to product teams in identify all uses of non-standard (i.e., not defined in the buildtools prereqs repository) container images
• Work with the product teams to help them migrate to container images defined in the buildtools prereqs repo

Engineering Efficiency and Sustainability

There are also objectives that are rather related to improved efficiency and sustainability when working with container image definitions, rather than with security.

• It should be very easy to add or delete support for a distro version without risk of affecting other distros
• It is easy to share and reuse container image definitions through templating

Open Questions

• Should be docker images definition part of the arcade repo or standalone in the buildtools prereqs repository?
• How are the docker images going to be shared between the product releases?
• How are we handling major underlying distro (Mariner) updates during servicing when it brings new major versions of important dependencies (clang, cmake, ...)
• How much are the scripts to create rootfs changing with the product releases?

[mmitche]

Have conversation about ownership/labeling.

[tkapin]

@missymessa - although I was the original author of this epic, I don't think I should be an owner of this again as it's rather related to helix machines than to product construction and release infrastructure. We should summarize what has been done as part of this initiative and close this epic as the team responsible for implementation has been redeployed to another org. Do we have anyone to audit & provide summary on what has been accomplished within this epic? /cc @markwilkie

[richlander]

We still need to audit the Dockerfiles that we use. This is something I was intending to work on, but I'd be happy to get help.

[missymessa]

@tkapin all I did was unassign Chris from the epic. No idea why GitHub decided to spoof me and assign it to you.