Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic on pulumi refresh after using @pulumi/aws provider 6.22.0 #3429

Closed
davejhilton opened this issue Feb 14, 2024 · 3 comments
Closed

Panic on pulumi refresh after using @pulumi/aws provider 6.22.0 #3429

davejhilton opened this issue Feb 14, 2024 · 3 comments
Labels
kind/bug Some behavior is incorrect or out of spec resolution/duplicate This issue is a duplicate of another issue

Comments

@davejhilton
Copy link

What happened?

I'm getting a go panic every time running pulumi refresh on one of my stacks, using @pulumi/aws provider version 6.22.0

The output is this:

pulumi:pulumi:Stack (eng-applications-dev):
    panic: value is null
    goroutine 5265 [running]:
    github.com/hashicorp/go-cty/cty.Value.LengthInt({{{0x1151c18a8?, 0x14002da5ee0?}}, {0x0?, 0x0?}})
    	/home/runner/go/pkg/mod/github.com/hashicorp/go-cty@v1.4.1-0.20200723130312-85980079f637/cty/value_ops.go:989 +0x254
    github.com/hashicorp/terraform-provider-aws/internal/service/elbv2.flattenLbForwardActionOneOf({{{0x1151c18a8?, 0x14002da5f30?}}, {0x112448260?, 0x14003f355f0?}}, 0x110368f1a?, {{0x14001c0b0c0, 0x7}, 0x0, 0x0, 0x0, ...}, ...)
    	/home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/service/elbv2/listener.go:1036 +0x18c
    github.com/hashicorp/terraform-provider-aws/internal/service/elbv2.flattenLbForwardAction(0x11327e6e0?, {0x110368f1a, 0x6}, 0x5?, {{0x14001c0b0c0, 0x7}, 0x0, 0x0, 0x0, 0x14007e06330, ...}, ...)
    	/home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/service/elbv2/listener.go:1014 +0x188
    github.com/hashicorp/terraform-provider-aws/internal/service/elbv2.flattenLbListenerActions(0x112475480?, {0x110368f1a, 0x6}, {0x14008dcb770?, 0x1, 0x120?})
    	/home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/service/elbv2/listener.go:979 +0x240
    github.com/hashicorp/terraform-provider-aws/internal/service/elbv2.resourceListenerRuleRead({0x1151c1528?, 0x140064c6d50}, 0x140041dae00, {0x114fb13c0?, 0x14000266f00?})
    	/home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/service/elbv2/listener_rule.go:617 +0x72c
    github.com/hashicorp/terraform-provider-aws/internal/provider.New.(*wrappedResource).Read.interceptedHandler[...].func9(0x14a24d0d0?, {0x114fb13c0?, 0x14000266f00?})
    	/home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/provider/intercept.go:112 +0x1d4
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x1151c1608?, {0x1151c1608?, 0x11e6d0600?}, 0xd?, {0x114fb13c0?, 0x14000266f00?})
    	/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20240202163305-e2a20ae13ef9/helper/schema/resource.go:795 +0x64
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0x1400199c2a0, {0x1151c1608, 0x11e6d0600}, 0x140052d5790, {0x114fb13c0, 0x14000266f00})
    	/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20240202163305-e2a20ae13ef9/helper/schema/resource.go:1094 +0x430
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Refresh({0x1400019b2c0?, {0x14000d1acf0?, 0x11521b2e0?, 0x14001f62390?}}, {0x1400195bb30?, 0x104e78f60?}, {0x110408515, 0x15}, {0x1151c19c0?, 0x1400523c870}, ...)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.74.0/pkg/tfshim/sdk-v2/provider.go:151 +0x1a8
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.(*providerWithPlanResourceChangeDispatch).Refresh(0x14001f62390, {0x1151c1528, 0x14005af6f00}, {0x110408515, 0x15}, {0x1151c19c0, 0x1400523c870}, {0x11503fbe0, 0x140033cd2f0})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.74.0/pkg/tfshim/sdk-v2/provider2.go:710 +0xc8
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Read(0x14000d6f980, {0x1151c1528?, 0x14005af6ab0?}, 0x14004a34c00)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.74.0/pkg/tfbridge/provider.go:1131 +0x6f0
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Read.func1({0x11521fa20?, 0x14000d6f980?})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/muxer@v0.0.7/muxer.go:369 +0x3c
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.resourceMethod[...](0x14002cf0fc0?, 0x140035936c8, 0x140035936a8?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/muxer@v0.0.7/muxer.go:322 +0xc4
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Read(0x14003593708?, {0x1151c1528?, 0x14005af6ab0?}, 0x11283b320?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/muxer@v0.0.7/muxer.go:368 +0x5c
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Read_Handler.func1({0x1151c1528, 0x14005af6ab0}, {0x1147ce7a0?, 0x14004a34c00})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.105.0/proto/go/provider_grpc.pb.go:611 +0x74
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x1151c1528, 0x14005af65d0}, {0x1147ce7a0, 0x14004a34c00}, 0x1400222c540, 0x140053ee318)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x2e8
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Read_Handler({0x114dfe4e0?, 0x14002cf0fc0}, {0x1151c1528, 0x14005af65d0}, 0x14003ba8080, 0x140028318a0)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.105.0/proto/go/provider_grpc.pb.go:613 +0x12c
    google.golang.org/grpc.(*Server).processUnaryRPC(0x14002a00000, {0x1151c1528, 0x14005af6540}, {0x1151f3300, 0x140029a84e0}, 0x14004f80120, 0x14002d73830, 0x11e675a58, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1385 +0xb8c
    google.golang.org/grpc.(*Server).handleStream(0x14002a00000, {0x1151f3300, 0x140029a84e0}, 0x14004f80120)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1796 +0xc4c
    google.golang.org/grpc.(*Server).serveStreams.func2.1()
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1029 +0x8c
    created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 83
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1040 +0x150

This panic happens consistently every time I try to run pulumi refresh on this stack. I cannot reproduce the issue on other stacks that are using earlier versions of this provider.

Example

I'm not actually sure exactly how to reproduce this... other than:

  • update my project's version of the @pulumi/aws node module from 6.13.2 to 6.22.0
  • run pulumi up, which ONLY updates the provider version
  • run pulumi refresh

this consistently causes the go panic.

If I revert to the older node module version, and then run pulumi up (to once again update the provider in my stack's state, to use the older version)... then pulumi refresh... it no longer panics.

I can see that the error message mentions that the panic is happening inside of elbv2.flattenLbForwardActionOneOf(...) ... but I haven't touched anything related to ALB listeners in quite some time.

If more details are required about the ALB listener rules that we've got in place (or the URNs of specific resources, etc), I suppose I could try to include an (obfuscated) sample of what we're doing. Or, I can provide a more full sample, if it's through a less public channel than a github issue. Though to be honest, I have no idea which listener rules it would be breaking on, since we have several different listeners, each with different rules and configurations.

Output of pulumi about

Version      3.105.0
Go Version   go1.21.6
Go Compiler  gc

Plugins
NAME    VERSION
nodejs  unknown

Host
OS       darwin
Version  14.2.1
Arch     arm64

This project is written in nodejs: executable='/Users/daveh/n/bin/node' version='v18.14.2'

Current Stack: *****/eng-applications/dev

...

Dependencies:
NAME                              VERSION
@aws-sdk/client-secrets-manager   3.445.0
@pulumi/aws                       6.22.0
@pulumi/aws-native                0.84.0
@pulumi/command                   0.9.2
@pulumi/datadog                   4.23.0
@pulumi/github                    5.22.0
@pulumi/postgresql                3.10.0
@pulumi/pulumi                    3.92.0
@pulumi/random                    4.14.0
lodash                            4.17.21
pg                                8.11.3
@types/node                       20.9.0
@types/pg                         8.10.9
@typescript-eslint/eslint-plugin  6.10.0
@typescript-eslint/parser         6.10.0
eslint                            8.53.0
ts-node                           10.9.1
typescript                        5.2.2

Additional context

I haven't had time to do a version "bisect", to see which is the earliest version of @pulumi/aws that starts causing the panic. Right now all I know is it's somewhere between 6.13.2 (working) and 6.22.0 (broken). But I can try messing with that tomorrow if I get some time.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@davejhilton davejhilton added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Feb 14, 2024
@Frassle Frassle transferred this issue from pulumi/pulumi Feb 14, 2024
@davejhilton
Copy link
Author

after creating this issue, I just saw #3421, which looks like it's the same issue

@VenelinMartinov
Copy link
Contributor

VenelinMartinov commented Feb 15, 2024 •
edited
Loading

Hey @davejhilton, yeah this looks very much like #3421.

That issue is fixed in 6.22.1 - please update to that and LMK if you are still having the problem!

I'll close this issue as a duplicate for now.

@VenelinMartinov VenelinMartinov added resolution/duplicate This issue is a duplicate of another issue and removed needs-triage Needs attention from the triage team labels Feb 15, 2024
@davejhilton
Copy link
Author

looks like 6.22.1 fixes the issue. Thanks 👍

@jetersen jetersen mentioned this issue Aug 1, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Some behavior is incorrect or out of spec resolution/duplicate This issue is a duplicate of another issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@davejhilton @VenelinMartinov