plugin.(Repository).CreateCatalog: plugin.(HostCatalogSecret).encrypt: no attributes defined: parameter violation: error #100

[jeremy-rescale]

I'm trying to create a host catalog plugin for AWS. I was able to originally create a host plugin for AWS using terraform without issue, but as of today, when trying to apply terraform, the host catalog plugin started failing. I destroyed all of the resources, and re-recreated, but I can not get past the error creating the host catalog plugin.

I was able to replicate the issue I was seeing with the terraform provide using the cli.

boundary host-catalogs create plugin -token=env://BOUNDARY_TOKEN  -scope-id $PROJECT_ID   -plugin-name aws   -attr disable_credential_rotation=true   -attr region=us-gov-east-1   -secret access_key_id=env://BOUNDARY_ACCESS_KEY_ID   -secret secret_access_key=env://BOUNDARY_SECRET_ACCESS_KEY
Error from controller when performing create on plugin-type host catalog

Error information:
  Kind:                Internal
  Message:             host_catalogs.(Service).createPluginInRepo: unable to create host catalog: plugin.(Repository).CreateCatalog: in project: p_DFQKgxQmW2: db.DoTx: plugin.(Repository).CreateCatalog: plugin.(HostCatalogSecret).encrypt: no attributes
  defined: parameter violation: error #100
  Status:              500
  context:             Error from controller when performing create on plugin-type host catalog

Both my boundary controller and cli are using version 0.15.4.
There's a 500 if I attempt to create the dynamic host catalog via the UI as well.

To Reproduce
Steps to reproduce the behavior:

1. Follow the steps listed in the tutorial

Expected behavior
I expect to be able to create a host catalog plugin for AWS.

Additional context
This issue arose after:
After rotating IAM credentials successfully using terraform. We then attempting to add a new group and role. We applied that change, but saw the errors around the host catalog plugin. I then destroyed all of the terraform and then tried to recreate. I then attempting to create using the cli.

[moduli]

After rotating IAM credentials successfully using terraform

Could you clarify what exactly this means? How did you go about accomplishing this?

If you rotated the credentials, are you using the new credentials when trying to create a new host catalog in Boundary?

Separately, does your access key start with AKIA? We currently only support access keys that start with that prefix for dynamic host catalogs (i.e. long-term credentials).

[jeremy-rescale]

Yes, the access key starts with AKIA.
I created a new access key via terraform with

resource "aws_iam_access_key" "boundary_aws_host_plugin_blue" {
  user = aws_iam_user.boundary_aws_host_plugin.name
}

And then updated the host catalog plugin to reference this new key.
When testing via the boundary cli, I manually created a new access key via the IAM console.

[jeremy-rescale]

I recently upgraded my controller and cli to version 0.18 and am still running into the issue

$ boundary host-catalogs create plugin -token=env://BOUNDARY_TOKEN  -scope-id $PROJECT_ID   -plugin-name aws   -attr disable_credential_rotation=true   -attr region=us-gov-east-1   -secret access_key_id=env://BOUNDARY_ACCESS_KEY_ID   -secret secret_access_key=env://BOUNDARY_SECRET_ACCESS_KEY
Error from controller when performing create on plugin-type host catalog

Error information:
  Kind:                Internal
  Message:             host_catalogs.(Service).createPluginInRepo: unable to create host catalog: plugin.(Repository).CreateCatalog: in project: p_kntJRoS70s: db.DoTx:
  plugin.(Repository).CreateCatalog: unknown, unknown: error #0: rpc error: code = FailedPrecondition desc = aws describe instances failed: operation error EC2: DescribeInstances,
  https response error StatusCode: 401, RequestID: 66a86d87-df4e-4611-912d-e731489e55e2, api error AuthFailure: AWS was not able to validate the provided access credentials
  Status:              500
  context:             Error from controller when performing create on plugin-type host catalog

But, if I use the key directly

$ AWS_ACCESS_KEY_ID="${BOUNDARY_ACCESS_KEY_ID}" AWS_SECRET_ACCESS_KEY="${BOUNDARY_SECRET_ACCESS_KEY}" aws ec2 describe-instances
{
    "Reservations": [
        {
            "Groups": [],
            "Instances": [
                {
                    "AmiLaunchIndex": 2,
....
}

[jeremy-rescale]

I was able to resolve this by upgrading boundary to the latest version, 0.18.1. I was also able to get the role_arn working so we could get rid of the static keys.