Update Cilium to v1.16.3

[xmudrii]

What this PR does / why we need it:

This PR updates Cilium to v1.16.3. This includes:

• Changing the Cilium architecture to adhere to the latest architecture with Envoy Proxy being a dedicated DaemonSet instead of being integrated into Cilium (users can revert back to the old architecture by overriding the Cilium addon, action required release note is added to the PR)
• Updating all components (Cilium, Hubble, CertGen) to their latest versions
• Updating Cilium configuration to be conformant with the latest Cilium version (the list of configuration changes is available here: Update Cilium to v1.16.3 #3415 (comment))
• Updating manifests to match the latest Helm chart
• Ensuring all files related to Hubble are stored in hubble.yaml
• Writing a migration for the hubble-generate-certs job

Additionally, this PR fixes a bug where we have been disabling CoreDNS instead of kube-proxy for clusters using kubeadm v1beta4 API. This bug only affects the main branch, so we don't need to cherry-pick this fix to the release branches.

Which issue(s) this PR fixes:
Closes #3347 🎉

What type of PR is this?
/kind feature

Special notes for your reviewer:

This PR has been tested manually in different scenarios and setups, we'll do more extensive testing in the testing phase.

Does this PR introduce a user-facing change? Then add your Release Note here:

[ACTION REQUIRED] Update Cilium to v1.16.3. This change might affect users that have nodes that are low on capacity (pods or resources wise). The Cilium architecture has been changed so that the Envoy Proxy is not integrated into Cilium, but is a dedicated component/DaemonSet. If you have nodes that are low on capacity, you might encounter issues when trying to start Envoy Proxy pods on those nodes. In this case, you'll need to override the Cilium addon to use the old architecture with Envoy Proxy integrated into Cilium.

Documentation:

NONE

/assign @kron4eg

[xmudrii]

Keys removed from the new config:

• enable-remote-node-identity
• skip-cnp-status-startup-clean
• sidecar-istio-proxy-image
• enable-bgp-control-plane
• proxy-prometheus-port

Keys that changed value in the new config:

• k8s-client-qps (5 -> 10)
• k8s-client-burst (10 -> 20)
• kube-proxy-replacement (strict / disabled -> true / false)
• external-envoy-proxy (false -> true)

Keys that don't exist in the old config, but exist in the new config:

• clustermesh-enable-endpoint-sync
• proxy-idle-timeout-seconds
• proxy-xff-num-trusted-hops-egress
• cluster-pool-ipv6-cidr
• bpf-events-policy-verdict-enabled
• enable-tcx
• k8s-require-ipv6-pod-cidr
• bpf-lb-sock-terminate-pod-connections
• datapath-mode
• envoy-keep-cap-netbindservice
• nodeport-addresses
• enable-runtime-device-detection
• cluster-pool-ipv6-mask-size
• proxy-xff-num-trusted-hops-ingress
• bpf-events-trace-enabled
• nat-map-stats-interval
• nat-map-stats-entries
• dnsproxy-socket-linger-timeout
• bpf-events-drop-enabled
• k8s-require-ipv4-pod-cidr
• envoy-base-id
• direct-routing-skip-unreachable
• enable-node-selector-labels
• clustermesh-enable-mcs-api

[kron4eg]

/lgtm
/approve

[kubermatic-bot]

LGTM label has been added.

Git tree hash: 1f4b862088cbae4a0868c61383903d6f0daccec6

[kubermatic-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kron4eg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kron4eg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment