Support for an IPv6 prefix per VM in shared networks

[phsm]

Justification

Cloudstack is used a lot in hosting industry, meaning that each VM often belongs to a different tenant (VPS).

According to recommendations, an end-user assignment should not be less than /64.

Evidently, some big services apply rate-limits to their API calls per /64 subnet, and not per an individual IP address.
It makes sense as an end user can just use 2^64 IPv6 addresses from their provider-allocated subnet to overcome an individual IP address rate-limits.

What is the current state of IPv6 in Cloudstack

Cloudstack Advanced zone with security groups uses a /64 subnet per shared network.
Each instance is supposed to obtain an automatic EUI-64 IPv6 address via SLAAC.
Additional IPv6 addresses can be assigned to vNIC using addIpToNic API call. They're also get allowed in the VM ipset on a hypervisor.
However, only individual IP addresses can be added this way. Adding a prefix is not supported.

Proposal

What if it was allowed to pass an arbitrary IPv6 prefix to addIpToNic?
This way an IPv6 subnet would be added to the ipset on the hypervisor so the VM instance is actually allowed to send traffic from it.

However, there will be two things that a Cloudstack administrator must have solved outside of Cloudstack:

• Add a route on the VLAN router to the additional IPv6 prefix using the VM SLAAC IPv6 address as the nexthop. This can be done separately using route servers such as ExaBGP which would inject the route using BGP.
• A way to configure an IPv6 address from the assigned pool on the VM itself, be that using cloud-init or simply telling the user what to do.

Your thoughts on this? I know @wido is an IPv6 enthusiast, his input will be greatly appreciated.

[wido]

@phsm I do agree with you that routing subnets to a VM would be very welcome and useful, but as. you noticed it's not that easy.

At the moment @alexandremattioli is working on more IPv6 support for VPCs with dynamic routing functionality. This can be very easily expanded to individual VMs. That's not much different from allowing a VR to route a subnet, in the end they are both VMs on the hypervisor.

I wouldn't know what the exact state of the work is, but I think that any help here would be much appreciated!

[rohityadavcloud]

cc @weizhouapache

[weizhouapache]

@wido @phsm
we are working on static/dynamic routing for isolated networks and vpc. ACS allocates a /64 network for each network (isolated/vpc tier). It is not applicable for @phsm who uses share networks for public cloud (VPS)

For shared networks, there is a change in 4.19.1.0 (#9289) which allows operators to create a shared network with other cidr than /64
I think it is feasible, @phsm you can research it

• create a new guru extends DirectNetworkGuru
• override isSlaacV6Only (return false)
• override the method allocate or allocateDirectIp

[wido]

@wido @phsm we are working on static/dynamic routing for isolated networks and vpc. ACS allocates a /64 network for each network (isolated/vpc tier). It is not applicable for @phsm who uses share networks for public cloud (VPS)

For shared networks, there is a change in 4.19.1.0 (#9289) which allows operators to create a shared network with other cidr than /64 I think it is feasible, @phsm you can research it

* create a new guru extends DirectNetworkGuru

* override `isSlaacV6Only` (return false)

* override the method `allocate` or `allocateDirectIp`

I discussed with @alexandremattioli that this code for dynamic routing of a subnet to a VPC router can be expanded to a shared network in the future. It will require some work, but in the future it would be great if you can allocate and deallocate a subnet to a VM or even multiple VMs.

This would allow you to do Anycast within CloudStack or other fancy BGP things on your VMs.

The work being done now would lay the foundation for this, but I do agree, it's not in scope now.

[alexandremattioli]

Indeed, the current work allows for BGP only in Isolated networks and VPCs with basic ASN management in the Zone level.

We have a few enhancements in mind for future iterations:

• User selectable ASN and peer(s)
• Static routes in the ACS VR
• Support for shared networks.

[weizhouapache]

@alexandremattioli @wido

actually I am wondering what is the difference between

• an isolated network in Routed mode, which does not support lb/pf/dnat/snat
• and a shared network

[wido]

Indeed, the current work allows for BGP only in Isolated networks and VPCs with basic ASN management in the Zone level.
We have a few enhancements in mind for future iterations:

* User selectable ASN and peer(s)

* Static routes in the ACS VR

* Support for shared networks.

That was my point indeed. The work being done now opens the door for shared networks as it shares many similarities, however, it will not be in the first iteration.

Not much I think. Instead of having a VR which gets BGP autoconfigured you would allow the end-user to configure BGP inside their own VM. We would just need to provide the user:

• AS number to use
• BGP peering details
• Prefixes they are allowed to announce

For the VR this will now be done automatically, but we could provide these details to the end user.

[weizhouapache]

thanks @wido

I think all 3 will be included in the upcoming PR for dynamic routing which includes

• AS number management
• BGP peer management
• Ipv4 subnet management

Maybe we just need to allow users to get these information if they want to setup BGP inside the vms on shared networks.
(currently only root admin can manage the resources)

[wido]

thanks @wido

I think all 3 will be included in the upcoming PR for dynamic routing which includes

* AS number management

* BGP peer management

* Ipv4 subnet management

Maybe we just need to allow users to get these information if they want to setup BGP inside the vms on shared networks. (currently only root admin can manage the resources)

As this issue is about IPv6, I assume you mean IPv4/IPv6 subnet at your last bullet, correct?

[weizhouapache]

thanks @wido
I think all 3 will be included in the upcoming PR for dynamic routing which includes

* AS number management

* BGP peer management

* Ipv4 subnet management

Maybe we just need to allow users to get these information if they want to setup BGP inside the vms on shared networks. (currently only root admin can manage the resources)

As this issue is about IPv6, I assume you mean IPv4/IPv6 subnet at your last bullet, correct?

the ipv6 subnet management has been introduced in #5786