darwin/arm64 build

[dansimau]

I didn't see an existing issue, so I thought I'd open an issue to track building an arm64 (Apple Silicon) binary for macOS.

After migrating to a new Mac, I have seen at least one issue using Terraform almost straight away:

module.certs_us-east-1["wildcard"].aws_acm_certificate_validation.validation: Still creating... [35m27s elapsed]
module.certs["wildcard"].aws_acm_certificate_validation.validation: Still creating... [34m57s elapsed]
module.certs_us-east-1["wildcard"].aws_acm_certificate_validation.validation: Still creating... [35m37s elapsed]
module.certs["wildcard"].aws_acm_certificate_validation.validation: Still creating... [35m7s elapsed]

assertion failed [abi_info.kind == AbiKind::TranslatedCode]: emulated forward to an arm pc that isn't in translated code. arm_pc=0x1053794d8 abi_kind=6 emulation_interval=[0x105464e60,0x105464e74) instruction_interval=[0x105464e4c, 0x105464e74) x86_rip=0x106148c
(ThreadContextRegisterState.cpp:677 move_to_instruction_boundary)


^C

(CTRL-C did not kill it; when I killed the provider processes they became zombies; I had to SIGKILL the main terraform process.)

Current Terraform Version

Terraform v0.14.2

References

• runtime: tracking bug for ARM-based macOS and GOOS/GOARCH values golang/go#38485 (will at least require Go build support for darwin/arm64)

[apparentlymart]

Hi @dansimau! Thanks for raising this.

There are some things that must happen before we could produce official Terraform releases on arm64 macOS, or before such releases would be practically useful:

• The Go toolchain we use must have production-quality support for that platform. Based on available information, it seems that Go 1.15 (our current version) has only early, experimental support, and that full support is expected in Go 1.16 which is not due until February.

• We will need to acquire hardware with the new CPU in order to verify that all of the platform features we depend on are available in Go's new port to the platform.

• At least the most commonly-used Terraform providers must have at least one release that supports this platform, so that Terraform will be able to install and run those plugins. (A Terraform executable without any provider plugins is not useful.)

  The provider development teams are also constrained by the availability of a suitable Go toolchain and hardware, so they will probably also only be able to do make experimental efforts until Go 1.16 is released, in which case there will be some lag after the Go 1.16 release before sufficient providers are available.

• Our policy for Go language upgrades is that we only accept patch-level upgrades in minor releases, and so the earliest we would adopt Go 1.16 is in the forthcoming v0.15 release. Given that the dependencies here include releases of several other pieces of software, it seems that we should not assume that everything lands in the correct order for macOS arm64 support to necessarily be included in v0.15, but it seems like that's the earliest release where we could support it.

Given that Apple's strategy for gradual ecosystem migration is emulation, I feel optimistic that problems in the current version of the emulation layer will be gradually addressed as the Apple team becomes aware of the problems, so hopefully future updates to the arm64 version of macOS will address the problems that are currently causing the assertion failure you saw. We'll monitor Go releases to see if there are any minor release updates to Go 1.15 that might be issued to improve compatibility with Apple's emulator when running x86_64 binaries, in which case we may be able to include them in v0.14.x minor releases.

[dansimau]

Makes sense.

After using it a bit more, I have concluded that Terraform is basically unusable under emulation on macOS arm64 (Big Sur 11.01).

Here are a sample of some more errors I have been getting:

$ terraform import module.auth.aws_security_group.presignup-lambda sg-XXX
module.auth.aws_security_group.presignup-lambda: Importing from ID "sg-XXX"...
module.auth.aws_security_group.presignup-lambda: Import prepared!
  Prepared aws_security_group for import
module.auth.aws_security_group.presignup-lambda: Refreshing state... [id=sg-XXX]

^C
^C^C^C
^C
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C

(Eventually I killed the provider processes).

$ terraform import module.auth.aws_iam_role.presignup-lambda cognito-presignup-lambda

Error: Could not load plugin


Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

Failed to retrieve schema from provider "registry.terraform.io/hashicorp/aws":
rpc error: code = Unavailable desc = connection error: desc = "transport:
authentication handshake failed: context deadline exceeded"

(Is the second time I have had to reinit).

$ terraform apply

Error: timeout while waiting for plugin to start


assertion failed [abi_info.kind == AbiKind::TranslatedCode]: emulated forward to an arm pc that isn't in translated code. arm_pc=0x104d204d8 abi_kind=6 emulation_interval=[0x104d77908,0x104d7791c) instruction_interval=[0x104d778f4, 0x104d7791c) x86_rip=0x4044049
(ThreadContextRegisterState.cpp:677 move_to_instruction_boundary)

At the moment it's roughly a coin toss as to whether a command will work or not.

[danieldreier]

@dansimau as it happens, I just got a an M1 Macbook, so I'm able to reproduce this. I did a state import of a security group, and it worked perfectly for me. However, I have my shell set to run under emulation by default, and when I use the default shell, I (on one run, haven't been able to reproduce since then) see the same behavior as you where it hangs waiting for the plugin to start.

Can you try running a shell under rosetta with a command like arch -x86_64 bash and then re-run these commands to see if the issue you're seeing still happens running in an emulated shell?

[dansimau]

@danieldreier I was already running an x86_64 bash (v5), because I was using /usr/local/bin/bash from Homebrew that was copied over after running Migration Assistant.

[dansimau]

Just FWIW, another symptom:

I randomly opened Activity Monitor and found this provider process using 100% CPU. Not even sure where it came from because I stopped running Terraform locally (unless the vscode language server runs it?)

Anyway, I will update to macOS 11.1 and see if there is any change (there is at least one report of Rosetta bug fixes in 11.1).

Side note, I had a lot of issues with other (non-Terraform) Go binaries compiled for darwin/x86_64 (e.g. my VSCode environment ground to a halt). I compiled Go HEAD for arm64, trashed my $GOHOME and reinstalled/recompiled everything. Now that environment is mostly working and stable (which is to be expected) with the notable exception of anything using cgo.

[tuannvm]

@dansimau

Anyway, I will update to macOS 11.1 and see if there is any change (there is at least one report of Rosetta bug fixes in 11.1).

I had same problem when kubectl & terraform x86 went crazy and ate up a lot of CPU. But upgrading to macOS 11.1 fixed that.

[dansimau]

Upgraded to 11.1 and it didn't change anything. The first time I ran Terraform it hung with the same symptoms (provider at ~100% CPU).

Here's a sample of the process:
https://gist.github.com/dansimau/b424531d9ad69acd1dcc08f2713341e0

[steveetm]

For v0.12 users, terraform plan -out was crashing me as well but since 11.1 it is fixed.

[billinghamj]

For anyone who just wants to get on with being able to do their job...

# on your Apple Silicon (/opt/homebrew) brew installation, _not_ your Intel (/var/local) brew installation
brew install --build-from-source terraform

Make sure your brew bin is in your PATH (e.g. echo 'eval $(/opt/homebrew/bin/brew shellenv)' >> $HOME/.zprofile).

Make sure to restart your terminal if you've just changed your PATH or installed terraform to /opt/homebrew.

git clone https://github.com/hashicorp/terraform-provider-aws.git
cd terraform-provider-aws
git checkout v3.22.0
cd tools
go get -d github.com/pavius/impi/cmd/impi
cd ..
make tools
make build

Then copy from $GOPATH/bin/terraform-provider-aws to:

~/.terraform.d/plugins/registry.terraform.io/hashicorp/aws/3.22.0/darwin_arm64/terraform-provider-aws_v3.22.0_x5

Then back in your workspace:

rm .terraform.lock.hcl
terraform init

It should pick up the arm64 build from your global plugins (https://www.terraform.io/docs/commands/cli-config.html#implied-local-mirror-directories)

The same basic process should work for all providers

The resultant .terraform.lock.hcl file should not be committed as-is, because it will break TF for other developers. Please see @apparentlymart's comment below for a better way to handle that

[apparentlymart]

The strategy of building the providers locally for unsupported architectures is certainly a possible workaround. Thanks for sharing the instructions, @billinghamj.

One thing I'd add here is that following your instructions exactly will cause the .terraform.lock.hcl file to only contain the checksum for your local darwin_arm64 build, which means that the configuration will therefore only be usable on that platform: if you have coworkers using other platforms then the installation will fail because the packages for their platform won't be recorded in the lock file.

If you are in an environment where you need to mix both supported and unsupported platforms then probably the best approach would be to review the git diff created by that final terraform init, see which checksum it recorded in the lock file, and then git checkout .terraform.lock.hcl before adding that new checksum in addition to the "official" checksums Terraform learned from the upstream registry. If you commit that result to the repository then that should leave you in a state where both the official packages and your unofficial build are both acceptable.

---

Relatedly, I'm not sure if the local build process described here serves as a reproducible build such that two different people running those steps on their own laptops would end up producing identical binaries. If not, you may run into similar trouble if you have multiple people in your environment that are using this new platform and creating their own builds. To mitigate that, you could produce one "official-within-my-company" build of the provider and share it with all users, thus causing everyone to agree on what the checksum ought to be.

[billinghamj]

Yeah, in my case I'm just probably not going to commit the modified lock file, and we likely will just share the binaries between each other too.

[xiazemin]

(pprof) top
assertion failed [abi_info.kind == AbiKind::TranslatedCode]: emulated forward to an arm pc that isn't in translated code. arm_pc=0x101ddf4d8 abi_kind=6 emulation_interval=[0x101e3cb64,0x101e3cb7c) instruction_interval=[0x101e3cb4c, 0x101e3cb7c) x86_rip=0x7fff20392ee4
(ThreadContextRegisterState.cpp:677 move_to_instruction_boundary)

I have the same problem on mac M1

[brainstorm]

Today on master (instead of the git checkout v3.22.0) the build breaks:

% make tools && make build
cd awsproviderlint && GO111MODULE=on go install .
cd tools && GO111MODULE=on go install github.com/bflad/tfproviderdocs
cd tools && GO111MODULE=on go install github.com/client9/misspell/cmd/misspell
cd tools && GO111MODULE=on go install github.com/golangci/golangci-lint/cmd/golangci-lint
cd tools && GO111MODULE=on go install github.com/katbyte/terrafmt
../../../go/pkg/mod/github.com/hashicorp/terraform-exec@v0.12.0/tfinstall/download.go:12:2: missing go.sum entry for module providing package github.com/hashicorp/go-getter (imported by github.com/hashicorp/terraform-exec/tfinstall); to add:
	go get github.com/hashicorp/terraform-exec/tfinstall@v0.12.0
make: *** [tools] Error 1

But even with the local build described by @billinghamj in our environment there are at least 2 other providers requiring attention for the Apple Silicon M1 ("github" and "external" providers in our case):

% terraform init .

Initializing the backend...


Initializing provider plugins...
- Finding hashicorp/github versions matching "~> 3.0.0"...
- Finding hashicorp/aws versions matching "~> 3.22.0"...
- Finding latest version of hashicorp/external...
- Installing hashicorp/aws v3.22.0...
- Installed hashicorp/aws v3.22.0 (unauthenticated)

Warning: Additional provider information from registry

The remote registry returned warnings for
registry.terraform.io/hashicorp/github:
- For users on Terraform 0.13 or greater, this provider has moved to
integrations/github. Please update your source in required_providers.


Error: Incompatible provider version

Provider registry.terraform.io/hashicorp/github v3.0.0 does not have a package
available for your current platform, darwin_arm64.

Provider releases are separate from Terraform CLI releases, so not all
providers are available for all platforms. Other versions of this provider may
have different platforms supported.


Error: Incompatible provider version

Provider registry.terraform.io/hashicorp/external v2.0.0 does not have a
package available for your current platform, darwin_arm64.

Provider releases are separate from Terraform CLI releases, so not all
providers are available for all platforms. Other versions of this provider may
have different platforms supported.

Inconvenient but totally doable: just compiled github & external manually and copied the binaries to the .terraform.d plugins dir as described for the aws provider and terraform plan works fine.

Obviously, for this to be practical, a concerted effort to test and CI as many plugins as possible for this to be usable in day-to-day devops tasks at this point in time, unfortunately :/

Does @hashicorp have some kind of (public) roadmap for this Apple M1 porting effort somewhere other than the [Tracking] issues opened by @skyzyx ?

[billinghamj]

I think it's really crazy that Hashicorp are refusing to touch this until Go 1.16 is released. The beta/RC is drastically more stable than running TF in Rosetta, which is what thousands of engineers are now being forced to do.

Brew has already been distributing TF built with 1.16 for M1 and it works perfectly. I really don't think there's any excuse or justification for the current stance :\

Every other tool and runtime we use has been working flawlessly for months at this point. Terraform is the sole outlier.

[skyzyx]

Since the providers all communicate with Terraform over gRPC, I anticipate (educated speculation) that Terraform core needs to go first (since everything is driven through that). As long as the other providers aren't broken on arm64 and can run in Rosetta, it seems plausible that they could continue to work via Rosetta since CPU instructions wouldn't directly affect gRPC communication between the provider and Core.

---

Separately, @brainstorm wrote:

Does @hashicorp have some kind of (public) roadmap for this Apple M1 porting effort somewhere other than the [Tracking] issues opened by @skyzyx?

I would love a centralized resource, but I don't think there's anything requiring Hashicorp to put it together. Any member of the community could put together a dashboard, and ask people to contribute the up-to-date information for the providers they care about. That's all that Hashicorp would be doing if they were to do it themselves, I suspect.

I started filing tickets for the providers I care about. There's nothing preventing others from doing the same and linking them together. I don't have the bandwidth at the moment to lead such an effort, but I'd happily participate since I'm watching those providers pretty closely anyway.

Inspiration:

• https://isapplesiliconready.com
• https://doesitarm.com

[billinghamj]

Go 1.16 is out now. Please could this be prioritised reasonably highly so providers can start accepting PRs etc to add arm support?

[skyzyx]

Go 1.16 is out now. Please could this be prioritised reasonably highly so providers can start accepting PRs etc to add arm support?

https://golang.org/doc/go1.16#darwin

[ameeno]

right, so I followed the instructions above, and built 2 providers. that got me past the terraform init stage, but when I tried to terraform apply (using remote backend ) it gives me these errors.

Error while installing hashicorp/google v3.4.0: the current package for
registry.terraform.io/hashicorp/google 3.4.0 doesn't match any of the
checksums previously recorded in the dependency lock file


Error: Failed to install provider

Error while installing hashicorp/google-beta v3.5.0: the current package for
registry.terraform.io/hashicorp/google-beta 3.5.0 doesn't match any of the
checksums previously recorded in the dependency lock file```

[yi2020]

@apparentlymart v0.15 just came out but seems to not be using Go 1.16 still?

[billinghamj]

@yi2020 appears they switched to Go 1.16 in this commit, which is included in the v0.15.0 tag - e3abe90

[yi2020]

@billinghamj the go.mod is still using 1.14 as the minimum though?

[apparentlymart]

The entry in go.mod specifies which version of the Go language Terraform is targeting, not which version of the Go toolchain it's built with. The .go-version file is what defines the toolchain version we use for building the release binaries.

The go directive in go.mod remains a bit hypothetical at this point, but the intent of it is to potentially allow later versions of the Go language to include minor breaking changes that individual modules can opt in to, while allowing other modules in the same program to remain on the old version of the language. But the Go team also tries very hard to avoid breaking changes that would need that, so it hasn't come up a lot so far and, as a result, we don't tend to pay much attention to that entry in our go.mod right now.

[yi2020]

Ah that makes sense. So I'll go ahead and build it now on my own. Do you expect you guys to add darwin_arm64 to the releases directory anytime soon?

[apparentlymart]

Due to the requirement that the macOS builds be notarized via HashiCorp's Apple developer credentials that our team doesn't have direct access to, we're relying on support from a centralized release engineering team to add this new platform to the release pipeline. Their schedule is not under our direct control, so I can't say with any certainty when we'll be able to get that included, but on the plus side it's a process step common to all HashiCorp products distributed in this way so I expect that the approach for one will be the approach for all, once it's designed, and there are similar requests pending from all of the other product teams too.

If all of the providers you intend to use have darwin_arm64 releases available by now though, indeed you can use your own build from the relevant tag in this repository to get a similar effect to what we'd release officially. (The provider teams have been able to get these new packages in place sooner because provider plugins are run only indirectly via Terraform and so they are not subject to the macOS notarization requirement.)

[billinghamj]

It's available on brew already too: https://github.com/Homebrew/homebrew-core/blob/master/Formula/terraform.rb

So don't really need to wait for Hashicorp to post arm64 builds if you don't want

[andig]

Imho we do need to wait (or compile) since this will require the notarized plugins which are not available yet.

[billinghamj]

@andig The plugins work fine with the brew version already - they have been working great for a couple of months now, and @apparentlymart seems to have indicated that the plugins do not need to be notarized.

[andig]

@billinghamj as far as I understand terraform will need to download providers that match the architecture. For that purpose it is not sufficient to have a brew arm64 binary:

❯ file $(which terraform)
/opt/homebrew/bin/terraform: Mach-O 64-bit executable arm64

❯ terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/google from the dependency lock file

Error: Incompatible provider version

Provider registry.terraform.io/hashicorp/google v3.61.0 does not have a
package available for your current platform, darwin_arm64.

Provider releases are separate from Terraform CLI releases, so not all
providers are available for all platforms. Other versions of this provider may
have different platforms supported.

If the respective provider is available with matching platform things are fine again:

❯ terraform init -upgrade

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/google...
- Installing hashicorp/google v3.64.0...
- Installed hashicorp/google v3.64.0 (signed by HashiCorp)

Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

❯ terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/google from the dependency lock file
- Using previously-installed hashicorp/google v3.64.0

Terraform has been successfully initialized!

[Tenzer]

I have hacked together a quick way to check which versions of a provider support macOS on arm64:

PROVIDER="hashicorp/google"
curl --silent https://registry.terraform.io/v1/providers/${PROVIDER}/versions | jq -r '.versions[] | select(.platforms[] | contains({os: "darwin", arch: "arm64"})) | .version'

the above would for instance output:

3.63.0
3.64.0

I haven't found any other obvious way of detecting which platforms each provider version are available as, so this could be handy for figuring out when all the providers you use are available for arm64.

[billinghamj]

@andig Yeah that's right. A lot of providers have already published arm64/darwin binaries, but indeed some have not yet. It looks like hashicorp/google does as of v3.63.0 though.

But either way, notarization does not appear to be required for plugins - just updating to Go 1.16

[anthonynsimon]

For those encountering this issue with hashicorp/template (I know it's deprecated but some providers depend on it internally):

│ Error: Incompatible provider version
│ 
│ Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.
│ 

Here's a workaround to run it on darwin/arm64:

• Clone the repo: https://github.com/hashicorp/terraform-provider-template
• Build from source (eg. go build).
• Move the resulting binary to the global terraform plugins path: mv <resulting-binary-path> ~/.terraform.d/plugins/registry.terraform.io/hashicorp/template/2.2.0/darwin_arm64/terraform-provider-template_v2.2.0_x5
• Give it exec permissions (eg. chmod +x <binary-path-above>).
• Then just do terraform init back in your repo.

Hope it helps.

[abhishek-galoda]

For those encountering this issue with hashicorp/template (I know it's deprecated but some providers depend on it internally):

│ Error: Incompatible provider version
│ 
│ Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.
│ 

Here's a workaround to run it on darwin/arm64:

• Clone the repo: https://github.com/hashicorp/terraform-provider-template
• Build from source (eg. go build).
• Move the resulting binary to the global terraform plugins path: mv <resulting-binary-path> ~/.terraform.d/plugins/registry.terraform.io/hashicorp/template/2.2.0/darwin_arm64/terraform-provider-template_v2.2.0_x5
• Give it exec permissions (eg. chmod +x <binary-path-above>).
• Then just do terraform init back in your repo.

Hope it helps.

Work or me only when file copied to ~/.terraform.d/plugins/registry.terraform.io/hashicorp/template/2.2.0/darwin_arm64/

[Junkern]

I am working in a lot of different Terraform projects and many of them have pinned providers version to old version. So either they are pre-darwn/arm64 or don't have a pre-compiled binary. After following the workarounds manually every time I decided to write a small CLI to handle all the details: https://github.com/kreuzwerker/m1-terraform-provider-helper

Hope this of help to anyone!

[apparentlymart]

Hi all,

Terraform v1.0.2 is the first release to include an official build for macOS on arm64.

We've not yet added this new platform to our supported platforms in the v1.0 compatibility promises. Although we have tested this build on our own systems and believe it to work correctly, we'd like to let this new version soak for a release or two before we commit to compatibility with it in the same sense as we have for the other supported platforms, just in case we learn from feedback that something isn't behaving in an ideal way and thus we may need to make adjustments to some details.

With that said, we don't have any specific plans to change or remove that release in the future, and so this is still an official release that we recommend for use, since our testing suggests that it is functioning correctly. We intend to add it as a fully-supported platform in the near future.

If you try it and run into strange behaviors that you didn't previously see with the macOS amd64 releases then please open a new GitHub issue about that and let us know.

The supported platforms for individual providers are decided by the teams who release those providers and so we (the Terraform Core team) can't promise that all providers you use will have matching support for darwin_arm64 (the internal name for the "macOS on arm64" platform). If you find that a provider you depend on doesn't yet have a darwin_arm64 release, or if you are not yet ready to upgrade to a newer version that supports that platform, we recommend registering a feature request in the repository for the relevant provider (which you can locate via its entry in Terraform Registry) and then continue to use the darwin_amd64 build of Terraform through Apple's official emulation layer until support is available in all of the providers you need.

Thanks for your patience while we worked through getting this implemented. Properly supporting a new platform is not a trivial matter but fortunately it's a relatively rare event! Again, please open a new bug report issue if you encounter problems with the new build, so that we can investigate each distinct problem separately.