Missing documentation/examples

[mjudeikis]

Would be nice to have few examples docs/examples how one could run all these pipelines outside VSTS and build their own images. For now this project looks like one-leg-opensource, where external consumer would have challenges to re-use it.

[alexeldeib]

admittedly this repo is heavily tied to AKS implementation today. it forked from aks-engine which is/was a much more community driven project. these days we'd point people to Cluster API Provider for Azure, which uses image-builder for reference images:

https://github.com/kubernetes-sigs/cluster-api-provider-azure
https://github.com/kubernetes-sigs/image-builder/tree/main/images/capi/packer/azure

we definitely can use better docs either way, however :)

[mjudeikis]

Any plans to unify image build process?

[alexeldeib]

not at the moment. there are some quirks, e.g. CAPI builds VM image per version while we have one mega VM image with as much crammed in as possible.

what's your goal exactly?

[mjudeikis]

Im just exploring what it would take to make bottlerocket running for AKS, so exploring all building blocks in the chain :)

[alexeldeib]

Hmm. So a few thoughts.

Do you really want it in AKS, or Azure? I see you mentioned both CoreOS and AKS-Engine in other issues. If you're fine self-managing, CAPI/CAPZ would allow you to use any images today.

You still have to do the legwork to ensure the OS is compatible. I've done this in the past for Talos Linux and I can't imagine it would be much harder for bottlerocket. bottlerocket-os/bottlerocket#1114 (comment) is very helpful in that vein.

From the Azure side, you'd want to ensure you fill the platform functionality requirements provided by the guest agent (for example, Talos doesn't use one, Ubuntu and Azure Linux and most distros use https://github.com/Azure/WALinuxAgent):
https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/agent-linux
https://learn.microsoft.com/en-us/azure/virtual-machines/linux/no-agent

For AKS, I'm not sure investing to support Bottlerocket as a managed OS makes a ton of sense, especially with the backdrop of existing work on AzLinux/Mariner https://github.com/microsoft/CBL-Mariner.

But even that can't really stop you, you can join whatever you want to the AKS control plane, it's just not supported: https://gist.github.com/alexeldeib/2a788d64d3229fe578f552e2239ce934

So...to me the obvious thing would be someone takes Sam's helpful comment from bottlerocket, tries to standup plain VMs in Azure, and if all goes well you could build CAPI reference bottlerocket images for Azure and use those just like the old AKS-Engine days.

[mjudeikis]

Ou, https://gist.github.com/alexeldeib/2a788d64d3229fe578f552e2239ce934 this is so nice write-up. I might be poking devil in the eye, but do you have something similar for windows nodes by any chance? :)

[alexeldeib]

unfortunately I do not :( I am not as familiar with windows, although something similar should be possible

[AbelHu]

Here is some doc for Windows: https://github.com/Azure/AgentBaker/tree/master/e2e/windows. cc @junjiezhang1997

[mjudeikis]

I can see you recently added need to MSI and windows_private_packages_url in your build.
does this means windows build are no more reproducible outside Microsoft infrastructure/CICD?

We would like to get windows self-managed hosts to join our AKS cluster using this.

@alexeldeib @junjiezhang1997 ?

[AbelHu]

I can see you recently added need to MSI and windows_private_packages_url in your build. does this means windows build are no more reproducible outside Microsoft infrastructure/CICD?

We would like to get windows self-managed hosts to join our AKS cluster using this.

@alexeldeib @junjiezhang1997 ?

@mjudeikis we need to test or build windows vhds for some urgent fixes. Those urls will be public finally.

[mjudeikis]

Is it possible not to block build of these urls and msi is not available? Just don't use them if not provided.

This way builds works without them too.