Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Retirement] Migrate from acs-mirror.azureedge.net to packages.aks.azure.com #4792

Open
rahulrai-in opened this issue Feb 12, 2025 · 9 comments
Assignees

Comments

@rahulrai-in
Copy link

rahulrai-in commented Feb 12, 2025

Due to the Edgio retirement, Azure Kubernetes Service (AKS) is updating its egress requirements. The FQDN acs-mirror.azureedge.net:443—currently used to download and install binaries for necessary dependencies (e.g., kubenet and Azure CNI)—will experience degraded performance starting March 31, 2025, and will be fully retired on May 31, 2025.

Required action

If you use a Layer 7 (FQDN) firewall to restrict AKS egress traffic, action may be required:

1. Azure Firewall with AzureKubernetesService FQDN Tag
If you rely on Azure Firewall Application rules with the AzureKubernetesService FQDN tag, no action is needed. The update will be applied automatically.

2. Other Firewalls or Egress Restrictions
If you restrict egress using another solution (not Azure Firewall as described above), you must add packages.aks.azure.com:443 to your allowed egress list by March 31, 2025. After this date, new AKS VHD images will require access to this hostname to successfully provision.

Updated Egress rules for AKS clusters are available in the Microsoft Learn documentation.

FAQs

Q. What are the destination IP addresses of the host packages.aks.azure.com?
You can view the current IP addresses at any time using the following command:

curl https://packages.aks.azure.com/acs-mirror/ips

Any changes to the current IP addresses will be communicated at least one week in advance through AKS Release Notes. However, we strongly recommend using a Layer 7 firewall rule that allows traffic to the domain instead of relying on specific IP addresses. This approach avoids the need to update your configuration whenever IP addresses change.

Q. What happens if I do not update my firewall to allow access to the new FQDN by May 31, 2025?

When you perform cluster or Nodepool upgrades, your AKS cluster nodes will use the new Virtual Hard Disk (VHD) image and download necessary binaries from the new FQDN. If you do not update your firewall to allow access to the new FQDN by the full deprecation date, May 31st, 2025, the nodes will fail to be provisioned, and your upgrade operation will fail. However, the existing nodes will continue to function until they are replaced by AKS Node Auto-Healing or a Nodepool upgrade.

Q. How long do I need to allow access to the old FQDN acs-mirror.azureedge.net:443?

New VHD images that reference the new FQDN-packages.aks.azure.com will be released in March 2025. After the new VHD images are applied to all the nodes of your AKS cluster, you can safely remove access to the old FQDN.

Q. What happens if my AKS cluster has the node auto-upgrade OS feature enabled, and it updates the cluster nodes to the new VHD that accesses the new FQDN before I add the new FQDN to my firewall rules?

We are implementing a fallback logic in the new VHDs that tries to access the old FQDN if it fails to access the new FQDN-packages.aks.azure.com. You might experience a small latency due to this fallback operation, but your node will still successfully provision.

Q. What happens if my cluster cannot upgrade to the new VHD image that accesses the new FQDN?

Please note that after May 31,2025, the acs-mirror.azureedge.net CDN endpoint will no longer function and therefore any operation that that causes the nodes to be replaced-such as the Node Auto-Healing-will fail to operate. If you are on a legacy Kubernetes cluster, we recommend following our extensive AKS Migration Guide to migrate to a new AKS cluster.

Help and Support

If you have questions, get answers from community experts in Microsoft Q&A. If you have a support plan and you need technical help, create a support request.

@rahulrai-in rahulrai-in self-assigned this Feb 12, 2025
@ms1111
Copy link

ms1111 commented Feb 22, 2025

Any changes to these IP addresses will be communicated at least one week in advance.

How will those changes be communicated?

@rahulrai-in
Copy link
Author

rahulrai-in commented Feb 24, 2025

Any changes to these IP addresses will be communicated at least one week in advance.

How will those changes be communicated?

Thanks for your comment @ms1111. Information regarding any changes to the IP addresses will be available in AKS release notes.

@TomkhaLoL
Copy link

TomkhaLoL commented Feb 24, 2025

  1. Azure Firewall with AzureKubernetesService FQDN Tag
    If you rely on Azure Firewall Application rules with the AzureKubernetesService FQDN tag, no action is needed. The update will be applied automatically.

Since today we're getting alerts that requests from our k8s cluster to packages.aks.azure.com are getting blocked, but we do have the FQDN Tag "AzureKubernetesServices" whitelisted in our Azure Firewall. Is this update of the FQDN tag still pending or do we maybe have to act here?

@JoeyC-Dev
Copy link

  1. Azure Firewall with AzureKubernetesService FQDN Tag
    If you rely on Azure Firewall Application rules with the AzureKubernetesService FQDN tag, no action is needed. The update will be applied automatically.

Since today we're getting alerts that requests from our k8s cluster to packages.aks.azure.com are getting blocked, but we do have the FQDN Tag "AzureKubernetesServices" whitelisted in our Azure Firewall. Is this update of the FQDN tag still pending or do we maybe have to act here?

I think this service tag is referring to "after change", not "before change".

If you check the IP result from two different domains, the deprecated one is not in the list.

@rlaveycal
Copy link

  1. Azure Firewall with AzureKubernetesService FQDN Tag
    If you rely on Azure Firewall Application rules with the AzureKubernetesService FQDN tag, no action is needed. The update will be applied automatically.

Since today we're getting alerts that requests from our k8s cluster to packages.aks.azure.com are getting blocked, but we do have the FQDN Tag "AzureKubernetesServices" whitelisted in our Azure Firewall. Is this update of the FQDN tag still pending or do we maybe have to act here?

I've been seeing that since last week and am trying to persuade Azure support that there is an issue!

@toredash
Copy link

1. Azure Firewall with AzureKubernetesService FQDN Tag If you rely on Azure Firewall Application rules with the AzureKubernetesService FQDN tag, no action is needed. The update will be applied automatically.

@rahulrai-in the ServiceTag is currently not working, is this pending, or should it be completed already?

I've raised a support ticket with Azure about this:

Support request ID	2502240050004619
Created on	Mon, Feb 24, 2025, 5:32:47 PM
Support plan	Unified Enterprise

@rahulrai-in
Copy link
Author

@toredash @rlaveycal @JoeyC-Dev @TomkhaLoL Thanks for the feedback.
The FQDN tag updates are yet to be deployed. Once deployed the change will be automatically applied to your Firewall if your Firewall rules allow access to the AzureKubernetesService tag.

@cfrost
Copy link

cfrost commented Feb 26, 2025

@rahulrai-in What is the timeline for the FQDN tag update ?

@rahulrai-in
Copy link
Author

@rahulrai-in What is the timeline for the FQDN tag update ?

Currently we are expecting to complete the roll out of the tag update in March'25.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants