Skip to content

Commit

Permalink
patch: reworked code to remove direct calls to viper.GetXXXX
Browse files Browse the repository at this point in the history
Previously there were calls to viper.GetSomething all over the code. This was prone to error if a value changed so now the code has been refactored to process any viper inputs.
Flags will always override the viper config file and any fields not supplied will use the defaults where applicable.

Added checksum validation for the Trivy cli download

Removed duplicate Trivy report structs and generally moved some bits around to begin work for allowing other clouds to be used. There is still work to be done to allow other clouds though as it's still generally OpenStack focussed.

Added new docs/readme & changelog

Fixes #32 & Fixes #34

updated README
  • Loading branch information
drew-viles committed Apr 12, 2023
1 parent 85821e0 commit dbe9815
Show file tree
Hide file tree
Showing 25 changed files with 712 additions and 572 deletions.
26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Changelog

## [ 12/04/2023 - v0.1.0-beta.1 ]

### ADDED
* Added changelog
* Refactored code to:
* Prevent using `viper.GetXXXX` across the codebase - now gets put into struct to allow one location to be updated rather than multiples.
* Begin work to allow more clouds to be added - still work to be done #36.
* Begin work to improve flags - still work to be done #33.
* Updated the config file requirements. *This is a breaking change and old configs will no longer work.*.

### Fixed
* Trivy checksum now used to validate trivy download if required #32.
* Added flags, which were previously missing, to support adding Trivy and Falco to the image #34.

### Deprecated
* The publish command will be reworked in an upcoming release to prevent the GitHub requirement. Instead, it will generate the files require to publish a single images scan results as an artifact with which the user can then decide what to do.

## [ Previous versions ]

* Up to this point, there has been no changelog supplied for previous versions as it was a rapid iterative process.
* With the release of v0.1.0-beta.1, any changes will be logged and should one be a breaking change, it will incur a version bump.
* Minor version bumps will be reserved for general changes
* Patch version bumps will be for fixes and patches
* The beta tags will be for superficial changes within a patch that require testing before a final release is created.
97 changes: 15 additions & 82 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,105 +12,38 @@ flags/config file, and it will do the rest for you.

⚠️Currently in beta at the moment.

# Prerequisites
# Supported clouds

### Openstack
| Cloud Provider |
|--------------------------------|
| [Openstack](docs/openstack.md) |

It is expected that you have a network and sufficient security groups in place to run this.<br>
It will not create the network or security groups for you.
*More clouds could be supported but may not be maintained by EscherCloudAI.*

For example:

```
openstack network create image-builder
openstack subnet create image-builder --network image-builder --dhcp --dns-nameserver 1.1.1.1 --subnet-range 10.10.10.0/24 --allocation-pool start=10.10.10.10,end=10.10.10.200
openstack router create image-builder --external-gateway public1
openstack router add subnet image-builder image-builder
OS_SG=$(openstack security group list -c ID -c Name -f json | jq '.[]|select(.Name == "default") | .ID')
openstack security group rule create "${OS_SG}" --ingress --ethertype IPv4 --protocol TCP --dst-port 22 --remote-ip 0.0.0.0/0 --description "Allows SSH access"
openstack security group rule create "${OS_SG}" --egress --ethertype IPv4 --protocol TCP --dst-port -1 --remote-ip 0.0.0.0/0 --description "Allows TCP Egress"
openstack security group rule create "${OS_SG}" --egress --ethertype IPv4 --protocol UDP --dst-port -1 --remote-ip 0.0.0.0/0 --description "Allows UDP Egress"
```
*EscherCloudAI will put the framework in place to the best of their availability/ability to facilitate more clouds being added.*

# Usage
Simply run the binary with the following flags (minimum required). See the example below.
You will also require a source image to reference for the build to succeed.
You must supply a clouds.yaml file for OpenStack connectivity.

The following is an example of the `baski.yaml` config file. This can be stored in `/tmp/`, `/etc/baski`, `$HOME/.baski` or the "current" directory.
```yaml
clouds-file: "~/.config/openstack/clouds.yaml"
cloud-name: "image-builder"
build:
verbose: true
build-os: "ubuntu-2204"
image-prefix: "kube"
#image-repo: ""
network-id: "network-id"
source-image: "source-image"
flavor-name: "spicy-meatball"
use-floating-ip: true
floating-ip-network-name: "Internet"
attach-config-drive: false
image-visibility: "private"
crictl-version: "1.25.0"
cni-version: "1.2.0"
kubernetes-version: "1.25.3"
extra-debs: "nfs-common"
enable-nvidia-support: true
nvidia-driver-version: "525.85.05"
nvidia-bucket-endpoint: "S3_ENDPOINT_URL"
nvidia-bucket-name: "nvidia"
nvidia-bucket-access: "ACCESS_KEY"
nvidia-bucket-secret: "SECRET_KEY"
nvidia-installer-location: "NVIDIA-Linux-x86_64-525.85.05-grid.run"
nvidia-tok-location: "client_configuration_token.tok"
gridd-feature-type: "4"
image-disk-format: "raw"
rootfs-uuid: "ROOT_FS_UUID" # The image in Openstack will be tagged with this. Useful for bare-metal in some use cases.
scan:
image-id: "" # Used for existing images
flavor-name: "spicy-meatball"
network-id: "network-id"
attach-config-drive: false
skip-cve-check: false
max-severity-score: 7.0 # Minimum severity score to check for
max-severity-type: MEDIUM # Minimum severity to check for
sign:
generate:
path: "." # Output path of any generated keys
vault:
url: "https://vault.ENDPOINT/"
token: "VAULT_TOKEN"
image-id: "" # Used for existing images
private-key: "" # Takes precedence over vault.
public-key: "" # Takes precedence over vault.
digest: "" # Used to verify a digest. Not required for image signing.
publish:
image-id: "" # Used for existing images
github:
user: "some-user"
project: "some-project"
token: "123456789"
pages-branch: ""
results-file: "/tmp/results.json"

```

Run the binary with a config file or see the help for a list of flags.
In the [example config](baski-example.yaml), not all fields are required and any fields that are not required are left
blank - unless the fields are enabled by a bool, for example in the Nvidia options where none are required
if `enable-nvidia-support` is set to false,

### More info

For more flags and more info, run `baski --help`

### GitHub Pages
### GitHub Pages - Deprecated

You will need to set up your target repo for the GitHub Pages in advanced.
It only requires a `gh-pages` branch for this to work.
GitHub Pages should be configured to point to a `docs` directory as this is where the resulting static site will be
placed.

# TODO
* Make this work for more than just Openstack so that it's more useful to the community around the Kubernetes Image Builder?

* Make this work for more than just Openstack so that it's more useful to the community around the Kubernetes Image
Builder?
* Remove dependency on GitHub Pages in the publish section - have this generate an artifact instead
* Fail on CVE critical discovery and remove any uploaded image
* Create all option to allow whole process?
Expand Down
60 changes: 60 additions & 0 deletions baski-example.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
cloud:
openstack:
clouds-file: "~/.config/openstack/clouds.yaml"
cloud-name: "image-builder"
network-id: "network-id"
source-image-id: "source-image"
flavor-name: "spicy-meatball"
attach-config-drive: false
use-floating-ip: true
floating-ip-network-name: "Internet"
image-visibility: "public"
image-disk-format: "raw"
volume-type: ""
rootfs-uuid: "ROOT_FS_UUID" # The image in Openstack will be tagged with this. Useful for bare-metal in some use cases.
build:
verbose: true
build-os: "ubuntu-2204"
image-prefix: "kube"
image-repo: "https://github.com/kubernetes-sigs/image-builder.git"
crictl-version: "1.26.0"
cni-version: "1.2.0"
kubernetes-version: "1.26.3"
extra-debs: "nfs-common"
add-trivy: true
add-falco: true
nvidia:
enable-nvidia-support: true
nvidia-driver-version: "525.85.05" # Currently used for tagging metadata - may be removed in future in favor of parsing the filename.
nvidia-bucket-endpoint: "S3_ENDPOINT_URL"
nvidia-bucket-name: "nvidia"
nvidia-bucket-access: "ACCESS_KEY"
nvidia-bucket-secret: "SECRET_KEY"
nvidia-installer-location: "NVIDIA-Linux-x86_64-525.85.05-grid.run"
nvidia-tok-location: "client_configuration_token.tok"
nvidia-gridd-feature-type: "4"
scan:
image-id: "" # Used for existing images
auto-delete-image: false
skip-cve-check: false
max-severity-score: 7.0 # Minimum severity score to check for
max-severity-type: MEDIUM # Minimum severity to check for
sign:
generate:
path: "." # Output path of any generated keys
vault:
url: "https://vault.ENDPOINT/"
token: "VAULT_TOKEN"
image-id: "" # Used for existing images
private-key: "" # Takes precedence over vault.
public-key: "" # Takes precedence over vault.
digest: "" # Used to verify a digest. Not required for image signing.
publish:
image-id: "" # Used for existing images
github:
user: "some-user"
account: "some-account" # Used for none personal accounts - leave blank if personal-account/project
project: "some-project"
token: "123456789"
pages-branch: ""
#results-file: "/tmp/results.json" # Not currently used
20 changes: 20 additions & 0 deletions docs/openstack.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
### Openstack guidelines

It is expected that you have a network and sufficient security groups in place to run this.<br>
It will not create the network or security groups for you.

For example:

```
openstack network create image-builder
openstack subnet create image-builder --network image-builder --dhcp --dns-nameserver 1.1.1.1 --subnet-range 10.10.10.0/24 --allocation-pool start=10.10.10.10,end=10.10.10.200
openstack router create image-builder --external-gateway public1
openstack router add subnet image-builder image-builder
OS_SG=$(openstack security group list -c ID -c Name -f json | jq '.[]|select(.Name == "default") | .ID')
openstack security group rule create "${OS_SG}" --ingress --ethertype IPv4 --protocol TCP --dst-port 22 --remote-ip 0.0.0.0/0 --description "Allows SSH access"
openstack security group rule create "${OS_SG}" --egress --ethertype IPv4 --protocol TCP --dst-port -1 --remote-ip 0.0.0.0/0 --description "Allows TCP Egress"
openstack security group rule create "${OS_SG}" --egress --ethertype IPv4 --protocol UDP --dst-port -1 --remote-ip 0.0.0.0/0 --description "Allows UDP Egress"
```

Then craft a `baski.yaml` file based on the [example](../baski-example.yaml) supplied and run the commands you require.
22 changes: 11 additions & 11 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,19 @@ go 1.19
require (
github.com/go-git/go-git/v5 v5.6.1
github.com/google/uuid v1.3.0
github.com/gophercloud/gophercloud v1.2.0
github.com/gophercloud/utils v0.0.0-20230316075016-e15d7ee3ba3b
github.com/gophercloud/gophercloud v1.3.0
github.com/gophercloud/utils v0.0.0-20230330070308-5bd5e1d608f8
github.com/hashicorp/vault/api v1.9.0
github.com/pkg/sftp v1.13.5
github.com/spf13/cobra v1.6.1
github.com/spf13/cobra v1.7.0
github.com/spf13/viper v1.15.0
golang.org/x/crypto v0.7.0
golang.org/x/crypto v0.8.0
gopkg.in/yaml.v3 v3.0.1
)

require (
github.com/Microsoft/go-winio v0.6.0 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230321155629-9a39f2531310 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230411080316-8b3893ee7fca // indirect
github.com/acomagu/bufpipe v1.0.4 // indirect
github.com/cenkalti/backoff/v3 v3.2.2 // indirect
github.com/cloudflare/circl v1.3.2 // indirect
Expand All @@ -38,7 +38,7 @@ require (
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
github.com/hashicorp/go-sockaddr v1.0.2 // indirect
github.com/hashicorp/hcl v1.0.1-vault-5 // indirect
github.com/imdario/mergo v0.3.14 // indirect
github.com/imdario/mergo v0.3.15 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
Expand All @@ -61,12 +61,12 @@ require (
github.com/spf13/pflag v1.0.5 // indirect
github.com/subosito/gotenv v1.4.2 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
golang.org/x/mod v0.9.0 // indirect
golang.org/x/net v0.8.0 // indirect
golang.org/x/sys v0.6.0 // indirect
golang.org/x/text v0.8.0 // indirect
golang.org/x/mod v0.10.0 // indirect
golang.org/x/net v0.9.0 // indirect
golang.org/x/sys v0.7.0 // indirect
golang.org/x/text v0.9.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/tools v0.7.0 // indirect
golang.org/x/tools v0.8.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/square/go-jose.v2 v2.6.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
Expand Down
Loading

0 comments on commit dbe9815

Please sign in to comment.