Weird error due to missing glibc compatibility

[joerg]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Running Atlantis in out environment with a custom provider gave us this weird error:
failed to instantiate provider "terraform.custom.cloud/custom/provider" to obtain schema: fork/exec .terraform/providers/terraform.custom.cloud/custom/provider/1.0.0/linux_amd64/terraform-provider-custom: no such file or directory

Of course running Terraform manually from our Laptops worked. We tried everything for more than a day starting from permissions in Azure down to files, filesystems, networking... everything. It turned out that the Docker container (we built a customized one up from the official image) was missing the gcompat library.
This is of course an error with our provider and how it was compiled but I thought it good to have this documented here and maybe even to include this gcompat by default since it does no harm but might help with some headaches.

Logs

failed to instantiate provider "terraform.custom.cloud/custom/provider" to obtain schema: fork/exec .terraform/providers/terraform.custom.cloud/custom/provider/1.0.0/linux_amd64/terraform-provider-custom: no such file or directory

Environment details

Atlantis 0.19.2 run from the official Atlantis image with some added tools on Azure Container instances.