Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding go-cli #67

Merged
merged 139 commits into from
Jul 19, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
30da87c
adding go-cli
6za Jun 29, 2022
bd989f4
sign commit
6za Jun 29, 2022
c8db3e4
rename project
6za Jun 29, 2022
ee55614
adding init/create steps
6za Jun 29, 2022
171bd5d
adding globals note
6za Jun 29, 2022
9d7a5a7
removing the templates
6za Jun 30, 2022
de30a93
update globals
6za Jun 30, 2022
c0faaed
update security check
6za Jun 30, 2022
643070f
rebranding cli
6za Jun 30, 2022
6906657
Add check tools and update outputs
6za Jun 30, 2022
ebf480b
simplify os.shell calls
6za Jun 30, 2022
3a0d9bf
minor notes tweaks
6za Jul 1, 2022
ef7b21c
cleaning create steps
6za Jul 1, 2022
a03550f
address zip slip
6za Jul 1, 2022
0eecf34
address zip slip
6za Jul 1, 2022
4756113
cleaning some steps
6za Jul 1, 2022
410f0a3
reshuffle create steps
6za Jul 1, 2022
438fa36
Redirecting out to logs
6za Jul 4, 2022
97b5e70
add create dry-run
6za Jul 4, 2022
a98418a
Merge pull request #68 from kubefirst/go-cli-v0.0.2
6za Jul 4, 2022
d4d0076
add progress bars
6za Jul 4, 2022
8b2a84a
fix pr comments
6za Jul 4, 2022
bb64a1e
Merge pull request #70 from kubefirst/go-cli-v0.0.2
6za Jul 4, 2022
f0a63e4
reorg funcs
6za Jul 4, 2022
5c077bc
reorg funcs
6za Jul 4, 2022
e29a749
synching changes
6za Jul 5, 2022
f749ebb
synching changes
6za Jul 5, 2022
59774ac
synching changes
6za Jul 5, 2022
233a1a3
synching changes
6za Jul 5, 2022
de5b6b8
review create/init
6za Jul 5, 2022
92d9632
review imports
6za Jul 5, 2022
a0508c3
fix dry-run create
6za Jul 5, 2022
405f81d
Merge pull request #71 from kubefirst/go-cli-0.1.1
6za Jul 5, 2022
12e0d82
add some skip steps
6za Jul 5, 2022
145af0f
Adding skip steps
6za Jul 5, 2022
65bfc9d
feat: add docker compose to kubefirst-dev
Jul 6, 2022
5de2a04
Merge pull request #74 from kubefirst/add_docker_compose
Jul 6, 2022
a98e421
refactor: update gitlab folder structure, and clean up
Jul 6, 2022
961528b
Merge pull request #75 from kubefirst/update_gitlab_structure
Jul 6, 2022
0d8d641
refactor: update vault calls
Jul 6, 2022
18843fe
Add destroy buckets
6za Jul 6, 2022
eb292de
Merge conflict
6za Jul 6, 2022
ca6a4a3
added new lines
6za Jul 6, 2022
7bfcc0f
Merge pull request #72 from kubefirst/go-cli-v0.1.3
6za Jul 6, 2022
893b5d7
refactor: move setup functions to configs, add more descriptive docum…
Jul 6, 2022
00ef485
chore: add better log data
Jul 6, 2022
19d16d4
chore: add better log data
Jul 6, 2022
97d33ed
Merge pull request #76 from kubefirst/update_pkg_kubefirst
Jul 6, 2022
8b866ff
refactor: add initial single source of truth for config, re-structure
Jul 7, 2022
4928b34
refactor: move kubernetes client requests to internal
Jul 7, 2022
78e9b26
refactor: move kubernetes client requests to internal
Jul 7, 2022
17a77fc
refactor: move gitlab functions to gitlab internal
Jul 7, 2022
07d56c1
refactor: remove globals, add values to config
Jul 7, 2022
da76738
refactor: move git to internal
Jul 7, 2022
1145aee
refactor: remove unused commands, refactor argocd and helm calls
Jul 7, 2022
1576b1e
refactor: move ssh and gitlab functions
Jul 7, 2022
f5b2625
refactor: move vault and softserve to internal
Jul 7, 2022
0bfeb1e
chore: update logs to use logs.panic
Jul 7, 2022
0c8ed06
refactor: move command line flags to function callers
Jul 8, 2022
96d9b6d
refactor: update init progress bar
Jul 11, 2022
97ea13e
added cluster-name and gitops-version
pagottoo Jul 11, 2022
d756e18
blank space
pagottoo Jul 11, 2022
6ca1e55
Merge pull request #82 from kubefirst/feat/cluster-name-gitops-version
pagottoo Jul 11, 2022
f20eb40
chore: merge arcocdsync and clean commands
Jul 11, 2022
52e2a56
chore: merge in progress, code is breaking, now we can start validati…
Jul 11, 2022
96869c7
chore: merge in progress, code is breaking at init command
Jul 11, 2022
b533dd6
refactor: update progress bar
Jul 11, 2022
95fc203
chore: update step names
Jul 11, 2022
b414683
chore: update installation steps
Jul 11, 2022
88b2813
Merge pull request #84 from kubefirst/update_cmd_structure-manual-mer…
Jul 12, 2022
d2a5864
Merge pull request #78 from kubefirst/update_cmd_structure
Jul 12, 2022
97b0940
Rename repo
6za Jul 12, 2022
1ecda99
Merge pull request #86 from kubefirst/go-cli-rename
Jul 12, 2022
f98a324
docs: update readme with more general details
Jul 12, 2022
76d3db8
docs: fix typo
Jul 12, 2022
e44c80b
Merge pull request #87 from kubefirst/update_readme
Jul 12, 2022
e7f0890
reapply branch code
6za Jul 12, 2022
2d7157c
revert terraform action
6za Jul 12, 2022
4d74f47
fix err
6za Jul 12, 2022
c2092ae
re-ordering the functions (#89)
jarededwards Jul 12, 2022
c48813f
merge conflict
6za Jul 12, 2022
2f934f4
Merge pull request #88 from kubefirst/go-cli-reapply-branch
6za Jul 12, 2022
e4154e1
fix change
6za Jul 12, 2022
16c891d
fix change again
6za Jul 12, 2022
a3b5a14
Merge pull request #91 from kubefirst/go-cli-fix-20220712-v2
6za Jul 12, 2022
78fbe91
feat: expose argocd container port
Jul 12, 2022
1edef42
fix: fix aws profile config
Jul 12, 2022
f325a13
chore: expose gitlab and vault ports
Jul 12, 2022
171f3ee
Merge pull request #93 from kubefirst/fix_tf_profile
Jul 12, 2022
abb4352
Adding new ways to call shell
6za Jul 13, 2022
e938e41
Migrate most of terraforms
6za Jul 13, 2022
d46a034
Fix Var
6za Jul 13, 2022
a91d3de
Fix err
6za Jul 13, 2022
ce0b930
Fix err
6za Jul 13, 2022
6736de7
feat: update files and folders to follow rebranding
Jul 13, 2022
2e7b483
Merge pull request #94 from kubefirst/go-cli-0713-update-output
6za Jul 13, 2022
f3c50d5
chore: update naming left overs
Jul 13, 2022
1e11ff7
Merge pull request #95 from kubefirst/update_kubefirst_path
Jul 13, 2022
8555ff2
add new progress lib
6za Jul 13, 2022
980c464
create go-release-binaries workflow
johndietz Jul 13, 2022
5fe9b3c
add additional os/arch
johndietz Jul 13, 2022
810fb23
Merge pull request #97 from kubefirst/go-cli-0713-new-progress-bars
6za Jul 14, 2022
5e817ba
Explore a smaller create flow
6za Jul 14, 2022
78c99af
restore dry-run create
6za Jul 14, 2022
a8a5155
Merge pull request #99 from kubefirst/go-cli-0714-create-dry-run
6za Jul 14, 2022
69ce039
feat: implement handoff screen for clean command
Jul 14, 2022
3cecb27
chore: clean up
Jul 14, 2022
5ff4987
Merge pull request #100 from kubefirst/final_message_mvp
Jul 14, 2022
b821488
clean portforward noise
6za Jul 14, 2022
c0aefcd
Merge pull request #101 from kubefirst/go-cli-0714-silence-portforward
6za Jul 14, 2022
e241413
adding retry spec to argocd registry (#102)
johndietz Jul 14, 2022
7eab4c7
enable bucket version
6za Jul 14, 2022
31da92a
Merge pull request #103 from kubefirst/go-cli-versioning-buckets
6za Jul 14, 2022
3d890c1
refactor: update k1srt folder to k1
Jul 14, 2022
f1881f1
Merge pull request #104 from kubefirst/update_k1_folder
Jul 14, 2022
f5e963c
feat: add info styled screen
Jul 14, 2022
c9e3ee2
remove more noise
6za Jul 14, 2022
9ab686d
fixing conflict
6za Jul 14, 2022
60b24f8
fix vars
6za Jul 14, 2022
50e22b4
fix lint stuff
6za Jul 14, 2022
2f5a24d
Merge pull request #106 from kubefirst/go-cli-silence-cmds
6za Jul 14, 2022
17920fb
Merge pull request #105 from kubefirst/add_handoff_screen_to_info
Jul 14, 2022
7989ee4
feat: add handoff for create command
Jul 14, 2022
74ad990
feat: add non styled info command
Jul 15, 2022
0483782
chore: clean up
Jul 15, 2022
5b88920
Merge pull request #109 from kubefirst/non-blocker-info
Jul 15, 2022
6397725
chore: clean up, and add tests
Jul 15, 2022
30eff08
chore: enable docker colors, use 256 colors to avoid colors mismatch
Jul 15, 2022
2ccdf9d
Merge pull request #107 from kubefirst/add_handoff_screen_for_create
Jul 15, 2022
f529818
chore: update info screen to be non blocking screen
Jul 15, 2022
25a6a5a
static kubeconfig naming convention (#112)
johndietz Jul 15, 2022
7f70744
Merge pull request #111 from kubefirst/add_non_blocker_info_screen
Jul 15, 2022
5b18d82
k-ray cluster create updates (#108)
jarededwards Jul 15, 2022
8c196a1
fixes for vault unseal (#114)
johndietz Jul 15, 2022
3c00bd1
Vault unseal (#117)
johndietz Jul 15, 2022
0206148
Vault unseal (#118)
johndietz Jul 15, 2022
a7fae7c
Vault unseal (#119)
johndietz Jul 15, 2022
79d1281
Vault unseal (#120)
johndietz Jul 19, 2022
7691507
Merge branch 'main' into go-cli
johndietz Jul 19, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
strategy:
fail-fast: false
matrix:
language: [ 'javascript' ]
language: [ 'go' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
# Learn more about CodeQL language support at https://git.io/codeql-language-support

Expand Down
16 changes: 16 additions & 0 deletions .github/workflows/go-release-binaries.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,24 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: wangyoucao577/go-release-action@v1.29
with:
github_token: ${{ secrets.RELEASE_TOKEN }}
goos: darwin
goarch: amd64
- uses: wangyoucao577/go-release-action@v1.29
with:
github_token: ${{ secrets.RELEASE_TOKEN }}
goos: darwin
goarch: arm64
- uses: wangyoucao577/go-release-action@v1.29
with:
github_token: ${{ secrets.RELEASE_TOKEN }}
goos: linux
goarch: amd64
- uses: wangyoucao577/go-release-action@v1.29
with:
github_token: ${{ secrets.RELEASE_TOKEN }}
goos: linux
goarch: arm64

7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,9 @@ kubeconfig_*
*/cypress/screenshots/
*/cypress/videos/
.DS_Store
/git
/git
bin
.vscode/settings.json
logs/
/tmp
lint_log.txt
137 changes: 0 additions & 137 deletions Dockerfile

This file was deleted.

160 changes: 60 additions & 100 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,128 +1,88 @@
# nebulous
The Kubefirst Open Source Platform
# Kubefirst CLI

![images/nebulous-arch.png](images/nebulous-arch.png)
Kubefirst CLI is a cloud provisioning tool. With simple setup and few CLI calls, we spin up a full AWS cluster with full
GitOps integration, secrets management, production and development Kubernetes environments ready to be consumed.

## tl;dr:
- step 1: establish a new aws account with a single hosted zone that's configured to receive traffic from your domain name registrar
- step 2: add your 6 configuration values to kubefirst.env and run the nebulous container
- step 3: get a fully-functioning application delivery ecosystem, complete with kubernetes, gitops, vault, terraform, atlantis, gitlab, gitlab-runner, and a sample app that demonstrates how it all works.
- [Setup](#setup)
- [Start the container](#start-the-container)
- [Initialization](#initialization)
- [Creation](#creation)
- [Access ArgoCD](#access-argocd)
- [Destroy](#destroy)
- [Available Commands]()

---
## Setup

# user guide
The setup is extremely simple, create a `.env` file in the root folder, and add the following variables:

## docs
- [introduction](https://docs.kubefirst.com/)
- [installation](https://docs.kubefirst.com/nebulous/install.html)
- [getting familiar](https://docs.kubefirst.com/kubefirst/getting-started.html)
- [teardown](https://docs.kubefirst.com/nebulous/teardown.html)
- [faq](https://docs.kubefirst.com/nebulous/faq.html)
- [contact](https://docs.kubefirst.com/contact.html)
| Variable | example |
|--------------------|------------------|
| AWS_PROFILE | default |
| AWS_REGION | us-east-1 |
| CLOUD_PROVIDER=aws | aws |
| HOSTED_ZONE_NAME | example.com |
| ADMIN_EMAIL | john@example.com |

---
## Start the container

# contributor guide
We run everything on isolation with Docker, for that, start the container with:

The docs above are tailored to our end user's experience. However things are a little different if you're contributing to nebulous itself. The docs that follow are intended only for source contributors.
```bash
docker-compose up kubefirst-dev
```

## Initialization

### step 1 - setup nebulous.env
Some process requires previous initialization, for that, run:

For a first run, this step is no different than the guidance to our end users, you need to set up a `kubefirst.env` in the nebulous repo's root directory. You can create the file template by running this from your terminal, editing with your values with the normal settings.
```bash
mkdir -p ~/.kubefirst
go run . init --admin-email $ADMIN_EMAIL --cloud $CLOUD_PROVIDER --hosted-zone-name $HOSTED_ZONE_NAME --region $AWS_REGION
```

johndietz marked this conversation as resolved.
Show resolved Hide resolved
For subsequent executions, especially while debugging, it's sometimes helpful to use some additional environment variables that allow you to control the flow of execution. See the notes in each section for details on controlling your debugging.
## Creation

In addition to the flow controls, you'll also find some hack comments by the various terraform apply commands. This allows you to change apply commands to exiting deploy commands. This can also be valuable when you need a mulligan on a particular section.
At this point, everything is ready to start provisioning the cloud services, and for that we can run:

```bash
cat << EOF > kubefirst.env
###############################
# Access settings
# The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are your credentials to
# log into your AWS account, you can often find these in `~/.aws/credentials`
# The AWS_DEFAULT_REGION is the aws region that your new infrastructure will provision in -
# The AWS_HOSTED_ZONE_NAME is the domain name associated with your prerequesite hosted zone in route53 - it should look similar to yourdomain.com with no www. prefix and no . suffix

AWS_ACCESS_KEY_ID=YOUR_ADMIN_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=YOUR_ADMIN_AWS_SECRET_ACCESS_KEY
AWS_HOSTED_ZONE_NAME=yourdomain.com
AWS_DEFAULT_REGION=us-east-2


###################
# Admin settings
# The EMAIL_ADDRESS is used for the ssh key that's generated and for certificate expiration notifications
# The GITLAB_BOT_ROOT_PASSWORD is the password to use for the gitlab root user, change this to a value only you know

EMAIL_ADDRESS=YOUR_EMAIL_ADDRESS@yourdomain.com
GITLAB_BOT_ROOT_PASSWORD=123456ABCDEF!


###############################
# Users:
# The BUCKET_RAND needs to be set and uncommented before destroy, see the teardown
# docs for details.
#
# Contributors:
# The BUCKET_RAND has implications on bucket reuse when iterating
# once you successfully get past base terraform apply,
# take the random suffix that was generated, apply it to the
# next line, and start reusing the bucket for subsequent runs.
# if you don't set this value on subsequent runs, it will keep
# generating new buckets for you. You can find this value in the
# nebulous execution output.
#
# BUCKET_RAND=abc123


###############################
# Note: Operational Flow Controls - uncomment the items below
# when you want to skip over various sections. Leaving them
# all commented like they are here will execute everything.
#
#
# SKIP_HZ_CHECK=true
# SKIP_DETOKENIZATION=true
# SKIP_BASE_APPLY=true
# SKIP_GITLAB_RECONFIG=true
# SKIP_GITLAB_APPLY=true
# SKIP_ARGOCD_APPLY=true
# SKIP_VAULT_APPLY=true
# SKIP_SSH_STORAGE=true
# SKIP_USERS_APPLY=true
# SKIP_OIDC_PATCHING=true

EOF
go run . create
```

### step 2 - build nebulous locally

Come up with local tag name for your nebulous image. We'll use `foo` as our example local tag name in these docs. To build the `foo` tag of nebulous run the following from your local nebulous repo root directory.
## Access ArgoCD

```bash
docker build . -t nebulous:foo
aws eks update-kubeconfig --name kubefirst
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
kubectl -n argocd port-forward svc/argocd-server 8080:80
```

### step 3 - running nebulous
## Destroy

Once you have built the `nebulous:foo` image as shown above, you can kickoff the automated init script by running the following. The difference between this guidance and the end user guidance is that this mounts the `gitops`, `scripts`, and `git` directories to your localhost volume so you can negotiate changes to the runtime environment on the fly.
It will destroy the kubefirst management cluster, and clean up every change made in the cloud.

This is how you run the container with the volume mounts. Run this from your nebulous directory:
```
docker run -it --env-file=kubefirst.env -v $PWD/gitops:/gitops -v $PWD/metaphor:/metaphor -v $PWD/scripts:/scripts -v $PWD/git:/git --entrypoint /scripts/nebulous/init.sh nebulous:foo
```
```bash

### step 4 - teardown (once you're ready to tear it all back down, obviously)
go run . destroy
rm -rf ~/.kubefirst
rm ~/.flare
```

There are a few things to note about teardown.
## Available Commands

Nebulous creates a VPC, some subnets, a gitlab server, a kubernetes cluster, some policies, roles, and a few other things (complete list in the teardown docs). Terraform knows about all of these things, and if you only created these resources, you'll be able to run teardown without thinking too hard.
Kubefirst provides extra tooling for handling the provisioning work.

However, terraform is only able to destroy resources that are managed in terraform. It doesn't know about things you do manually. Anything you may have added through non-terraform operations must be manually removed before running the teardown script.
| Command | Description |
|------------|-----------------------------------------------------------|
| argocdSync | Request ArgoCD to synchronize applications |
| checktools | use to check compatibility of .kubefirst/tools |
| clean | removes all kubefirst resources locally for new execution |
| create | create a kubefirst management cluster |
| destroy | destroy the kubefirst management cluster |
| info | provides general Kubefirst setup data |
| init | initialize your local machine to execute `create` |
| version | print the version number for kubefirst-cli" |

Let's consider, for example, a scenario where you manually `helm install`ed an app to your new cluster, and that app spins up a new load balancer in your VPC. If you don't remove that app and its load balancer before running destroy, you won't be able to complete the terraform destroy operation. This is because you can't remove a VPC that still has a live load balancer running in it.
#### Notes:

With that context in mind, once you've removed the manual things you may have added to this environment, you can kickoff the automated destroy script by running:
```
docker run -it --env-file=kubefirst.env -v $PWD/gitops:/gitops -v $PWD/metaphor:/metaphor -v $PWD/scripts:/scripts -v $PWD/git:/git --entrypoint /scripts/nebulous/destroy.sh nebulous:foo
```
added gitlab.yaml to registry
pushing local to soft origin
Loading