Support multiple backends for SOPS

[sgibson91]

Similar in nature to #560 and #561 but involving sops

Summary

Related to needing to access resources stored in two-eye-two-see, but 2i2c.org account doesn't have access to the destination project (i.e. Pangeo/Columbia).

After we decided to use a different terraform backend so the state can be stored in the same project as the infrastructure, I am running into a similar issue with sops. I created a new GCP KMS in the Pangeo/Columbia project to handle the encryption of the deployer key in #489 and updated the .sops.yaml file to use it. So far so good.

https://github.com/2i2c-org/pilot-hubs/blob/94f4be1cf54f67c424440f2f74ba57d9eece14e1/.sops.yaml#L2-L3

The conflict arises from the auth0 key stored in config/secrets.yaml, for which the sops key is stored in two-eye-two-see and I am not logged into an account that can access that.

• Can two gcp_kms keys be adding to a single creation rule in .sops.yaml? (I seriously doubt it.)
• Also seems not best practice to have two keys for auth0 doing the same thing

I'm really not sure about how to go about tackling this one, tbh!

Additional context

I realised that the order of the creation rules in .sops.yaml matters. Such that, if a path matches more than one path_regex key, sops will only try to en/decrypt with the first match and not progress onto the second match if the first one fails.

Therefore:

# Example with two-eye-two-see key first
creation_rules:
  - path_regex: config/secrets.yaml$
    gcp-kms: blah/blah/two-eye-two-see/blah/blah
  - path_regex: config/secrets.yaml$
    gcp-kms: blah/blah/pangeo-integration-blah/blah/blah

will always fail for the Pangeo project; and

# Example with Pangeo key first
creation_rules:
  - path_regex: config/secrets.yaml$
    gcp-kms: blah/blah/pangeo-integration-blah/blah/blah
  - path_regex: config/secrets.yaml$
    gcp-kms: blah/blah/two-eye-two-see/blah/blah

will always fail for the two-eye-two-see project.

[yuvipanda]

An alternative is that we use the two-eye-two-see-org project for the key, and grant access to that key to both your 2i2c and columbia accounts. How does that sound?

[sgibson91]

That's definitely a solution! Still concerned we're getting more and more embedded into not-scalable practices though 😕

[sgibson91]

I've just added my Columbia UNI to the two-eye-two-see project with the KMS Encrypter/Decrypter role so that I can deploy a test hub as per #599