Support post-provisioning configuration hooks

[craiglpeters]

Describe the request
Provide a way to do post provisioning configuration on a node via a separate service prior to a node being marked ready.

Explain why AKS Engine needs it
Customers using aks-engine need a way to apply configurations on nodes via tools like Terraform, Ansible or DSC to do work like join the node to an asset management system, apply additional security settings, or configure virus scanners, prior to the node being available to the Kubernetes scheduler.

Describe the solution you'd like
A webhook mechanism that allows the external system to notify aks-engine that it is done doing its work.

Describe alternatives you've considered

Additional context

[PatrickLang]

One alternative we have today is:
aks-engine extensions

They are run within the CSE, so some operations like multiple reboots may cause a CSE failure, causing a deployment failure. They're also not re-entrant - you can't update a config setting in an extension and run it again on the same machine.

[PatrickLang]

I would also say an ideal solution should pull its configuration from a trusted location on Azure, instead of an unauthenticated source as aks-engine extensions do

[idanlevin]

As a customer that its entire infrastructure is built around K8s, you don't necessarily want to deal with another platform (e.g. chef, puppet, ansible) to provision your cluster. It'll be great to have the ability to run a bash script on the host, before it's ready.

This bash script can be used to:

1. Install security monitoring tools (Antivirus, Vulnerability scanners, EDR, configure RSyslog, etc.)
2. Change OS configuration (e.g. num of allowed TCP connections, automatic updates, etc.)

Worth mentioning that this bash script should run on new nodes as well, for example if cluster-autoscaler is enabled.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jackfrancis]

I think this net new functionality is out of scope for AKS Engine. The current boundary of AKS Engine permits other additional tooling (e.g., terraform, ansible) to perform configuration options.

Building custom images is probably the best practical current solution to customizing the OS layer in particular.