rancher2 cluster delete order causing errors

[David-VTUK]

Hi,

Firstly - thanks a lot for this provider, awesome stuff.

I'm testing with provisioning an RKE cluster on EC2 instances via the use of the following objects:

rancher2.NewCloudCredential
rancher2.NewNodeTemplate
rancher2.NewCluster
rancher2.NewNodePool

in the snippet:

		// Create a Node Template
		nodetemplate, err := rancher2.NewNodeTemplate(ctx, "davidh-pulumi-ec2-medium", &rancher2.NodeTemplateArgs{
			CloudCredentialId: cloudcredential.ID(),
			Description:       pulumi.String("node template for ec2"),
			Name:              pulumi.String("davidh-pulumi-nodetemplate"),
			EngineInstallUrl:  pulumi.String("https://releases.rancher.com/install-docker/19.03.sh"),

			Amazonec2Config: &rancher2.NodeTemplateAmazonec2ConfigArgs{
				Ami:                pulumi.String("ami-0ff4c8fb495a5a50d"),
				IamInstanceProfile: pulumi.String("k8snodes"),
				InstanceType:       pulumi.String("t2.medium"),
				VpcId:              vpc.ID(),
				Tags:               pulumi.String("kubernetes.io/cluster/davidh-cluster:owned"),
				SecurityGroups:     pulumi.StringArray{sg.Name},
				Region:             pulumi.String("eu-west-2"),
			},
		})

		if err != nil {
			return err
		}

		cluster, err := rancher2.NewCluster(ctx, "davidh-pulumi-cluster", &rancher2.ClusterArgs{
			Description: pulumi.String("Cluster created by Pulumi"),
			Driver:      pulumi.String("rancherKubernetesEngine"),
			Name:        pulumi.String("davidh-pulumi-ec2"),
			RkeConfig: &rancher2.ClusterRkeConfigArgs{
				CloudProvider: &rancher2.ClusterRkeConfigCloudProviderArgs{
					Name: pulumi.String("aws"),
				},
			},
		})

		_, err = rancher2.NewNodePool(ctx, "davidh-pulumi-nodepool", &rancher2.NodePoolArgs{
			ClusterId:      cluster.ID(),
			ControlPlane:   pulumi.Bool(true),
			Etcd:           pulumi.Bool(true),
			HostnamePrefix: pulumi.String("davidh-pulumi-aio-"),
			Name:           pulumi.String("davidh-pulumi-pool"),
			Quantity:       pulumi.Int(3),
			Worker:         pulumi.Bool(true),
			NodeTemplateId: nodetemplate.ID(),
		})

		// End return
		return nil
	})

For provisioning a cluster it works flawlessly. For deleting, it fails:

     Type                            Name                      Status                  Info
     pulumi:pulumi:Stack             pulumi-rancher-aws-demo   **failed**              1 error
 -   ├─ rancher2:index:NodePool      davidh-pulumi-nodepool    deleted                 
 -   ├─ aws:ec2:DefaultRouteTable    default                   deleted                 
 -   └─ rancher2:index:NodeTemplate  davidh-pulumi-ec2-medium  **deleting failed**     1 error

Diagnostics:
  pulumi:pulumi:Stack (pulumi-rancher-aws-demo):
    error: update failed
 
  rancher2:index:NodeTemplate (davidh-pulumi-ec2-medium):
    error: deleting urn:pulumi:demo::pulumi-rancher-aws::rancher2:index/nodeTemplate:NodeTemplate::davidh-pulumi-ec2-medium: Error removing Node Template: Bad response statusCode [405]. Status [405 Method Not Allowed]. Body: [baseType=error, code=MethodNotAllow, message=Template is in use by a node.] from [https://demo-hosted.rancher.cloud/v3/nodeTemplates/cattle-global-nt:nt-tfjqb]

From what I can see, it tries to delete the NodeTemplate object before the related node pool has deleted, which happens before the entire cluster is deleted.

I think a cleaner way of doing this is to delete the cluster object entirely, which will also remove the node pools.

Pulumi version: v2.19.0
Rancher Version 2.5.5

[David-VTUK]

Been performing some further testing. I had hoped https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster_sync would help.

Implemented in the above example as:

_, err = rancher2.NewClusterSync(ctx, "davidh-clustersync", &rancher2.ClusterSyncArgs{
	ClusterId:   cluster.ID(),
	NodePoolIds: pulumi.StringArray{nodepool.ID()},
})

On creation, it's the last resource that's created and doesn't complete until the entire cluster is up (prior, Pulumi would simply state the cluster resource was created before all the nodes came up etc.

 +   pulumi:pulumi:Stack                pulumi-rancher-aws-demo   create     
 +   ├─ aws:ec2:Vpc                     david-pulumi-vpc          create     
 +   ├─ aws:ec2:InternetGateway         gw                        create     
 +   pulumi:pulumi:Stack                pulumi-rancher-aws-demo   create     
 +   ├─ aws:ec2:Subnet                  Subnet-0                  create     
 +   ├─ aws:ec2:Subnet                  Subnet-1                  create     
 +   ├─ aws:ec2:Subnet                  Subnet-2                  create     
 +   ├─ aws:ec2:DefaultRouteTable       default                   create     
 +   ├─ rancher2:index:Cluster          davidh-pulumi-cluster     create     
 +   ├─ rancher2:index:CloudCredential  davidh-pulumi-aws         create     
 +   ├─ rancher2:index:NodeTemplate     davidh-pulumi-ec2-medium  create     
 +   ├─ rancher2:index:NodePool         davidh-pulumi-nodepool    create     
 +   └─ rancher2:index:ClusterSync      davidh-clustersync        create   

Destroy operation:

Do you want to perform this destroy? yes
Destroying (demo)

View Live: https://app.pulumi.com/DH-Rancher/pulumi-rancher-aws/demo/updates/63

     Type                            Name                      Status                  Info
     pulumi:pulumi:Stack             pulumi-rancher-aws-demo   **failed**              1 error
 -   ├─ rancher2:index:ClusterSync   davidh-clustersync        deleted                 
 -   ├─ rancher2:index:NodePool      davidh-pulumi-nodepool    deleted                 
 -   ├─ aws:ec2:DefaultRouteTable    default                   deleted                 
 -   └─ rancher2:index:NodeTemplate  davidh-pulumi-ec2-medium  **deleting failed**     1 error
 
Diagnostics:
  pulumi:pulumi:Stack (pulumi-rancher-aws-demo):
    error: update failed
 
  rancher2:index:NodeTemplate (davidh-pulumi-ec2-medium):
    error: deleting urn:pulumi:demo::pulumi-rancher-aws::rancher2:index/nodeTemplate:NodeTemplate::davidh-pulumi-ec2-medium: Error removing Node Template: Bad response statusCode [405]. Status [405 Method Not Allowed]. Body: [message=Template is in use by a node., baseType=error, code=MethodNotAllow] from [https://rancherurl.com/v3/nodeTemplates/cattle-global-nt:nt-bn2fx]

[David-VTUK]

Workaround: Define an explicit dependency between a cluster and nodetemplate

		cluster, err := rancher2.NewCluster(ctx, "davidh-pulumi-cluster", &rancher2.ClusterArgs{
...
		}, pulumi.DependsOn([]pulumi.Resource{nodetemplate}))

[jleni]

Same issue here
all the workaround was adding a DependsOn. The code looks different as the API has changed a bit since David-VTUK suggested this.

    // Create cluster
    const cluster = new rancher2.Cluster(this.name, cluster_args, {
      provider: localProvider,
      dependsOn: Array.from(nodeTemplates.values()),
    });

[Frassle]

Given the NodePool is taking nodetemplate.ID() as the input value for NodeTemplateId it should be creating a dependency in the engine anyway, the DependsOn call should be superfluous. Which is suggestive of a tfbridge or core platform issue, I can't see why this would be specific to the rancher2 provider?