Add dev environment

[bmonkman]

Right now we don't have specific instructions about how to have an effective dev environment, which tools to use, etc.
If you have just one backend service it's likely easy enough but we need to consider how we make it easy for them to set up multiple services, dependencies like databases, etc.
Look into fully local solutions like minikube, k3s, and consider cloud-based or hybrid solutions like https://www.telepresence.io/
Emphasis should be on developer experience.

[sshi100]

Confirmed Telepresence working with a Demo to the team. We will move forward to the next steps to check if Telepresence can be combined with tooling to combine and provide better experience for developers. Here we would collect some real world experiences when they deal with local and cloud.

Feedback:

• how to support compiled (non-interpreted) language development
• how to support multiple developers for multiple services
• ...

[bmonkman]

Once we have put more thought into those items (especially the developer experience and multiple services) let's also do a small PoC with one or two of the Helix devs and telepresence to see if there are any cases we are not covering or rough edges with the experience.

[sshi100]

Target that initial PoC, good to conduct one round of "chatting" with few devs and make an AC for this ticket :)

[sshi100]

Per our last Thursday chat, deep Zero backend review and further practicical tests of Skaffold and Telepresence, on top of our current Zero value (build Staging/Production infra within a day), I am almost sure that we can do "Add Dev Experience" (not environment) into realistic environments (Staging/Production), to help Developers address some gaps from monolith app to micro-service development.

Gaps/Pains in common:

1. How to apply my trial of a single service changes rapidly and continually to a realistic env? Once comfirmed, I will go with gitops flow.
2. How can I help narrow down a Production issue that could not be reproduced anyway in local, QA and even Staging?

Both needs in conditions:
a. No impact on regular realistic env and traffic flow
b. No extra people collaboration cost

Key points in solution:

• hot build: reload, compile
• hot redirect: redirect a specific request to specific service / pod

[sshi100]

Some notes of the 2nd review:

• concept: Add Dev Environment = Add Dev Experience into realistic environments
• service/pod: to create personal namespace on Staging and run pods on it
• database: to create a new dev database in RDS, we want the data persistent; if not working, we move to use container DB with Stateful Set + EFS volume, AP: how to solve RDS master password exposure
• secrets: use all db-ops secrets for now
• domain: use ingress, external-dns annotation
• image: tagging rule, see if reusable for Staging usage
• traffic (later): with canonical cookie in app request, direct traffic to specific TPed/Skaffolded service point

[sshi100]

Quick summary on changes:

[X] users: detect new users under developer group (infra, TF)
[X] database: use pre-created dev database (per developer) in RDS, we want the data persistent; support multiple user-db creation in one step
[X] credentials: copied from pre-created AWS secretmanager secrets (backend, infra, TF)
[X] service account: copied from existing service account (backend)
[X] domain: use hostname for ingress, support CORS with localhost access (backend, kustomize)
[X] image: use pre-created 'last-deployed' tagged image by build/deploy pipeline (backend, circleci)
[X] service/pod: create personal namespace on Staging and run pods on it
[X] traffic: route cloud request to local (backend, telepresence)
[ ] traffic (selective routing): with canonical cookie in app request, direct traffic to specific TPed/Skaffolded service point

How to use it:
Create Dev Environment (for Admin)
Start Dev Environment (fro Developer)

[sshi100]

PR:
commitdev/zero-aws-eks-stack#164
commitdev/zero-backend-go#45