Terraform often fails with "http2: server sent GOAWAY and closed the connection" when using pagerduty_service_dependency

[NargiT]

Terraform Version

Terraform v1.3.7
on linux_amd64

• provider registry.terraform.io/hashicorp/null v3.2.1
• provider registry.terraform.io/pagerduty/pagerduty v2.9.2

Affected Resource(s)

Seems to affect only this resource

• pagerduty_service_dependency (around 2650)

Terraform Configuration Files

Sorry cannot share .tf files but there is nothing fancy for pagerduty_service_dependency, It's simply very high number.

resource "pagerduty_business_service" "foobar" {
  name = "foobar"
}

resource "pagerduty_service_dependency" "foobar_service_a" {
  dependency {
    dependent_service {
      id = pagerduty_business_service.foobar.id
      type = "business_service"
    }
   supporting_service {
      id = pagerduty_service.service_a.id
      type = "service"
   }
  }
}

Expected Behavior

What should have happened?

The provider should be able to apply or give a more human readable error.

Actual Behavior

What actually happened?

Receiving Error: http2: server sent GOAWAY and closed the connection; LastStreamID=199, ErrCode=NO_ERROR, debug=""

Steps to Reproduce

1. Create more than 2600 resources of pagerduty_service_dependency and everything else (business_service, services, schedules, escalation_policies and users)
2. terraform apply -no-color -auto-approve -parallelism=10 -lock-timeout=30s

Important Factoids

Terraform state is store in a local S3 storage.
We have a company proxy but so far it does not seems to come from it.

References

#476
#507
#537 does not seems to affect pagerduty_service_dependency

In #507 @stmcallister mention a roll out for other data sources, I expect this issue to be fix the same way.

[...]
After talking with PagerDuty engineering we concluded it would be best to remove the check for the 429 error code response and relax the resource.Retry to retry on all errors returned by the API. I implemented this on only the two data sources mentioned in Issue 476 to see if it remedies the issue. Will roll out to other data sources as the need arises.
[...]

[MaxRBosch]

@NargiT have you tried to make changes to get around this issue? I am running into the same issue and so far no luck yet. I also only get it when having only 20 service_dependencies

[NargiT]

@MaxRBosch not a viable solution, unfortunately. I have to manually re trigger either the full build and 🤞 or apply modification using terraform apply -target RESOURCE_ID.

EDIT: you can also reduce the -parallelism=5 it will increase the TTM but at least it could work. Worst case you put 1 and hope it will be enough.

[NargiT]

🆙 please

[jelmokri]

Hello, have exactly the same issue.

please help

[imjaroiswebdev]

Hey folks! a patch for this issue has just been released on v2.11.2 Terraform Provider version and is GA on Terraform Registry. This patch followed the same workaround successfully applied to other resources and data sources, so please don't hesitate on reopening this issue if you are still facing it✌🏽

[officel]

@imjaroiswebdev

Hi! I have the same problem with pagerduty_schedule.
But a little different, when I run it locally, the problem is not there.
When I run it with github actions, the problem occurs about once every 2 or 3 times.

Please let me know if I should reopen this ticket, raise a new issue, or do nothing and wait for a response.🙇‍♂️

got errors

Error: Get "https://api.pagerduty.com/schedules/PY97SS2": http2: server sent GOAWAY and closed the connection; LastStreamID=199, ErrCode=NO_ERROR, debug=""

  with pagerduty_schedule.hongkong_1st,
  on schedules.tf line 426, in resource "pagerduty_schedule" "hongkong_1st":
 426: resource "pagerduty_schedule" "hongkong_1st" {