Removing a user from a team and schedule simultaneously frequently breaks because of ordering problems.

[g--]

When using Terraform to assign a person to both a pagerduty team and a schedule, Terraform can't remove the user in one run because of an ordering problem. Specifically: TF will first attempt to remove the user from the pagerduty team, which error because they are still in the schedule.. even when removing the user from the schedule was on the list of things to do in the run.

Attempted to control destroy behaviour with depends_on failed. I gave when I noticed that it doesn't help (https://www.reddit.com/r/Terraform/comments/cukfbn/control_destroy_order/)

Terraform Version

./terraform.sh -v
Terraform v0.12.18
+ provider.pagerduty v1.7.1

Affected Resource(s)

• pagerduty_team (indirectly)
• pagerduty_team_membership
• pagerduty_schedule
• pagerduty_escalation_policy

Note that pagerduty_team_membership and pagerduty_schedule have no direct link.. there's an indirect link through pagerduty_escalation_policy. Maybe changing the schedule first would be safer?

Terraform Configuration Files

(disclaimer, I haven't actually run this code. It's a stripped down version I have run though. If it's helpful, please reach out and I'll debug it.)

variable "people" {
  default = [ "P12345A", "P12345B", "P12345C" ]
}
resource "pagerduty_team" "test-team" {
  name = "test team"
}

resource "pagerduty_team_membership" "peoples" {
  for_each = toset(var.people)
  user_id = each.value
  team_id = pagerduty_team.test-team.id
}

resource "pagerduty_schedule" "schedule" {
  name = "example"
  time_zone = "Etc/UTC"
  layer {
    name = "layer"
    start                        = "2019-03-18T16:00:00Z"
    rotation_virtual_start       = "2019-03-18T16:00:00Z"
    rotation_turn_length_seconds = 604800
    users = var.people
  }
}
resource "pagerduty_escalation_policy" "escalation" {
  teams = [pagerduty_team.test-team.id]
  description = "managed by terraform"
  num_loops = 2
  teams = [pagerduty_team.test-team.id]
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id = pagerduty_schedule.schedule.id
    }
  }
}

Debug Output

(available on request)

Expected Behaviour

Team membership and schedule membership removed after TF is run.

Actual Behaviour

Schedule membership is removed, team membership is unchanged there's an error.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:
-1. populate the people list with real pagerduty user ids.
0. terraform init

1. terraform apply
2. remove one of the users from people
3. terraform apply

Typically (always?) I get:

Error: DELETE API call to https://api.pagerduty.com/teams/XXXXX/users/XXXXXX failed 400 Bad Request. Code: 2001, Errors: [User cannot be removed as they belong to an escalation policy on this team], Message: Invalid Input Provided

When run a second time, it succeeds.

Important Factoids

Not that I'm aware of.

References

not that I'm aware of. https://github.com/terraform-providers/terraform-provider-pagerduty/issues/66 looks similar, but not the same.

[stmcallister]

This is how the PagerDuty web UI behaves when you try to remove a user from a team that is already associated with an escalation policy. So, if you were were simply trying to remove the team member, this is expected behavior. However, if you're trying to remove the team members while also removing the team from the escalation policy, I'm thinking we should add some retry logic to the team_member deletion process, in case in tries to happen before the team deletion process. Thanks for bringing this up!