Consider making Terraform CLI talk to the single provider server

[muvaf]

What problem are you facing?

Right now we have a single provider binary and all terraform invocations use that one but each of them spins up a new provider server to talk to. We haven't observed noticeable issues but it's likely that this will cause performance issues.

How could Terrajet help solve your problem?

@ulucinar discovered a way to make Terraform CLI talk to the existing provider server. He didn't feel great about it for the initial pass but we might need to reconsider it after testing the providers with 100+ custom resources using composition.

[muvaf]

We need more data about how much of a problem this is.

[ulucinar]

The shared gRPC server packages for provider-jet-azure are ready here:
ulucinar/provider-jet-azure-arm64:shared-grpc
ulucinar/provider-jet-azure-amd64:shared-grpc

We are still collecting performance data using these images.

[muvaf]

@ulucinar Could you share the branch you're working on as a draft PR if it's in a state where others can compile?

[turkenh]

@haarchri mentioned this PR could be related/helpful to/with this issue: flux-iac/tofu-controller#67

[ulucinar]

@ulucinar Could you share the branch you're working on as a draft PR if it's in a state where others can compile?

@muvaf, will do.

Now that we have a provider package implementing a shared gRPC server ready, we need to evaluate the performance of it via a series of experiments.

@turkenh, will check the issue you referenced above, thank you.

[muvaf]

With the results in #233 , I think we are good to go with implementation of this if we're all on the same page. @ulucinar @sergenyalcin thoughts on closing this and opening an implementation issue?

[sergenyalcin]

With the results in #233 , I think we are good to go with implementation of this if we're all on the same page. @ulucinar @sergenyalcin thoughts on closing this and opening an implementation issue?

@muvaf As you said that, using gRPC server-based implementation provides improvements in the context of performance metrics. So, in light of the findings in the test issues, it seems reasonable to close this issue and open a new one for implementation.