Secrets in pipelines

[hnamazianeu]

Problem description

This user story is intended to remove all sensitive data from all .env files and bring the ability of storing and retrieving secrets, private keys. API keys, etc in AWS, Azure, Google Secret Manager as well as Hashicorp Vault Key storage.
Each service must be provided by an exclusive credential which enables the service to be authorized by the Secret Manager in order to access its own data.
In case of supporting Hashicorp Vault, all configurations, scripts and docker compose files for both develop and product environments must be provided.
Interface classes per each Secret Manager infrastructure must be developed in the lower layer as the common libraries. All interfaces must be abstracted by a higher level class which facilitates interaction with any of Secret Manager options, which is selected by an environment variable through the .env files.
All secrets, keys and other sensitive data that have been loaded into the application from .env files, must be fetched from the selected Secret Manager by utilizing the SecretManager class.
A complete guide to setup, configuration and operation of the KMS infrastructures must be documented.

Requirements

Here are some best practices to store secrets and sensitive data in a secure and safe manner in production:

1. Centralizing Secret
Storing secrets in a centralized data storage will help to build governance, auditing, and security around who accesses those secrets, when they access those secrets, etc. By centralizing those into one place, you're able to much more easily manage the security around that.

2. Access Control List (ACL)
With secrets stored in a centralized location, the next step is to create Access control lists of human, machines, applications that who has access to what portion of data and secrets.

3. Dynamic Secrets Each human, machine, application must have its own temporary credentials to access secrets. That guarantees that at any time it is possible to revoke the permission of each entity to access secrets.

4. Encryption as a Service
EaaS is aimed to encrypt/decrypt data using secrets which are stored safely in storage without exposing the secret. Besides, it helps developers to easily use cryptographic operations without implementing them by themselves.

5. Auditing
Auditing is the ability to verify who has accessed what.

There are several technologies providing Secret Management Storage which they support above mentioned criteria categorized into Cloud and Local (on-premise) KMS technologies.
The most common local solution is Hashicorp Vault which offers a secret and key management engine that can be coupled with several storage technologies including Hashicorp Consul, SQL Databases, file storage, etc. Additionally Hashicorp Vault brings Cloud Agnostic solutions that regardless of the underlying cloud infrastructure, the configurations, source codes, etc will not change.
AWS, Azure and Google provide Cloud based KMS solutions that take off the burden of deployments and configurations. Interaction to Cloud KMS would be simply done by the use SDKs provided by Cloud providers. However, using Cloud based KMS will bring extra development especially in case of migration from one Cloud to the other one.

Definition of done

• Secrets, Keys, etc must be fetched from Secret Manager instead of .env files
• AWS, Google, Azure Secret Manager Service and Hashicorp Vault must be supported for secret storage
• KMS infrastructure must be selected by an Environment Variable in a global .env file
• Each service must be provided by a specific credentials
• KMS must be configured in order that each service will have access to its own data only
• Services’ credentials must be securely stored
• Hashicorp Vault configurations and deployment for development and production environment must be provided
• Setup, configuration and management of KMS must be well documented

Acceptance criteria

All functionalities of Guardian application are expected to be working the same by running any Secret Manager options.

More information provided here:
https://github.com/hashgraph/guardian/blob/develop/docs/secrets%20manager/guardian-vault.md

[voycey]

Secrets management certainly needs to be beefed up - this is a great requirements list!

As I mentioned here: #1647 (comment) Hashicorp Vault can act as a proxy to all of the cloud KMS systems meaning that a single integration could be done rather than 20 - then everyone has the choice of exactly where they want the secrets stored with minimal work