Implement support for AWS ipv6 prefixes

[olemarkus]

Using this PR you can provision a cluster using AWS prefixes.

Going forward, I expect it to be possible to enable ipv6 prefixes as part of launch templates so kOps doesn't have to do it. Meanwhile I do believe this is k8s installer territory.

I hope that in the future, the IPAM controller can be a part of CCM, but meanwhile it runs fine as part of kops-controller.

[olemarkus]

This is the cluster settings I used.
I would expect this to use with any CNI though.

  kubeDNS:
    upstreamNameservers:
    - 2620:119:53::53
  kubeControllerManager:
    allocateNodeCIDR: false
    controllers:
      - "*"
      - "-nodeipam"
  kubeProxy:
    enabled: false
  networkCIDR: 172.20.0.0/16
  networking:
    cilium:
      version: v1.10.3
      disableMasquerade: true
      tunnel: disabled
  nonMasqueradeCIDR: fd00:10:96::/64

[olemarkus]

Since there were actually interest in getting this one merged, I made the code paths configurable.

The commits are still a bit messy, but feedback on the overall semantics would be appreciated.

/cc @justinsb

[olemarkus]

This is the cluster config I use for this PR now.
I have not tested NodePort, but it should probably work.

...
  kubeProxy:
    enabled: true
  networking:
    cilium:
      enableNodePort: false
  nonMasqueradeCIDR: fd00:10:96::/64
  podCIDRFromCloud: true
...

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[olemarkus]

@justinsb did you comment the nits? I only see the comment above. Since we need to enable a controller i kops-controller etc we need a flag that shows user's intent. Empty podCIDR implicitly meaning cloud IPAM sounds a bit magical to me.

[rifelpet]

/lgtm

[olemarkus]

/retest

[justinsb]

Ooops - I queued them up and forgot to hit send. I actually can't submit the comments now, so I guess I'll open a follow-up PR or two :-) It was mostly nits apart from the podCIDR == "" question, so this way probably works better anyway!