Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CN-147 refactor work #35

Merged
merged 1 commit into from
Apr 13, 2023
Merged

CN-147 refactor work #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Changelog

## [ 12/04/2023 - v0.1.0-beta.1 ]

### ADDED
* Added changelog
* Refactored code to:
* Prevent using `viper.GetXXXX` across the codebase - now gets put into struct to allow one location to be updated rather than multiples.
* Begin work to allow more clouds to be added - still work to be done #36.
* Begin work to improve flags - still work to be done #33.
* Updated the config file requirements. *This is a breaking change and old configs will no longer work.*.

### Fixed
* Trivy checksum now used to validate trivy download if required #32.
* Added flags, which were previously missing, to support adding Trivy and Falco to the image #34.

### Deprecated
* The publish command will be reworked in an upcoming release to prevent the GitHub requirement. Instead, it will generate the files require to publish a single images scan results as an artifact with which the user can then decide what to do.

## [ Previous versions ]

* Up to this point, there has been no changelog supplied for previous versions as it was a rapid iterative process.
* With the release of v0.1.0-beta.1, any changes will be logged and should one be a breaking change, it will incur a version bump.
* Minor version bumps will be reserved for general changes
* Patch version bumps will be for fixes and patches
* The beta tags will be for superficial changes within a patch that require testing before a final release is created.
99 changes: 16 additions & 83 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,107 +12,40 @@ flags/config file, and it will do the rest for you.

⚠️Currently in beta at the moment.

# Prerequisites
# Supported clouds

### Openstack
| Cloud Provider |
|--------------------------------|
| [Openstack](docs/openstack.md) |

It is expected that you have a network and sufficient security groups in place to run this.<br>
It will not create the network or security groups for you.
*More clouds could be supported but may not be maintained by EscherCloudAI.*

For example:

```
openstack network create image-builder
openstack subnet create image-builder --network image-builder --dhcp --dns-nameserver 1.1.1.1 --subnet-range 10.10.10.0/24 --allocation-pool start=10.10.10.10,end=10.10.10.200
openstack router create image-builder --external-gateway public1
openstack router add subnet image-builder image-builder

OS_SG=$(openstack security group list -c ID -c Name -f json | jq '.[]|select(.Name == "default") | .ID')
openstack security group rule create "${OS_SG}" --ingress --ethertype IPv4 --protocol TCP --dst-port 22 --remote-ip 0.0.0.0/0 --description "Allows SSH access"
openstack security group rule create "${OS_SG}" --egress --ethertype IPv4 --protocol TCP --dst-port -1 --remote-ip 0.0.0.0/0 --description "Allows TCP Egress"
openstack security group rule create "${OS_SG}" --egress --ethertype IPv4 --protocol UDP --dst-port -1 --remote-ip 0.0.0.0/0 --description "Allows UDP Egress"
```
*EscherCloudAI will put the framework in place to the best of their availability/ability to facilitate more clouds being added.*

# Usage
Simply run the binary with the following flags (minimum required). See the example below.
You will also require a source image to reference for the build to succeed.
You must supply a clouds.yaml file for OpenStack connectivity.

The following is an example of the `baski.yaml` config file. This can be stored in `/tmp/`, `/etc/baski`, `$HOME/.baski` or the "current" directory.
```yaml
clouds-file: "~/.config/openstack/clouds.yaml"
cloud-name: "image-builder"
build:
verbose: true
build-os: "ubuntu-2204"
image-prefix: "kube"
#image-repo: ""
network-id: "network-id"
source-image: "source-image"
flavor-name: "spicy-meatball"
use-floating-ip: true
floating-ip-network-name: "Internet"
attach-config-drive: false
image-visibility: "private"
crictl-version: "1.25.0"
cni-version: "1.2.0"
kubernetes-version: "1.25.3"
extra-debs: "nfs-common"
enable-nvidia-support: true
nvidia-driver-version: "525.85.05"
nvidia-bucket-endpoint: "S3_ENDPOINT_URL"
nvidia-bucket-name: "nvidia"
nvidia-bucket-access: "ACCESS_KEY"
nvidia-bucket-secret: "SECRET_KEY"
nvidia-installer-location: "NVIDIA-Linux-x86_64-525.85.05-grid.run"
nvidia-tok-location: "client_configuration_token.tok"
gridd-feature-type: "4"
image-disk-format: "raw"
rootfs-uuid: "ROOT_FS_UUID" # The image in Openstack will be tagged with this. Useful for bare-metal in some use cases.
scan:
image-id: "" # Used for existing images
flavor-name: "spicy-meatball"
network-id: "network-id"
attach-config-drive: false
skip-cve-check: false
max-severity-score: 7.0 # Minimum severity score to check for
max-severity-type: MEDIUM # Minimum severity to check for
sign:
generate:
path: "." # Output path of any generated keys
vault:
url: "https://vault.ENDPOINT/"
token: "VAULT_TOKEN"
image-id: "" # Used for existing images
private-key: "" # Takes precedence over vault.
public-key: "" # Takes precedence over vault.
digest: "" # Used to verify a digest. Not required for image signing.
publish:
image-id: "" # Used for existing images
github:
user: "some-user"
project: "some-project"
token: "123456789"
pages-branch: ""
results-file: "/tmp/results.json"

```

Run the binary with a config file or see the help for a list of flags.
In the [example config](baski-example.yaml), not all fields are required and any fields that are not required are left
blank - unless the fields are enabled by a bool, for example in the Nvidia options where none are required
if `enable-nvidia-support` is set to false,

### More info

For more flags and more info, run `baski --help`

### GitHub Pages
### GitHub Pages - Deprecated

You will need to set up your target repo for the GitHub Pages in advanced.
It only requires a `gh-pages` branch for this to work.
GitHub Pages should be configured to point to a `docs` directory as this is where the resulting static site will be
placed.

# TODO
* Make this work for more than just Openstack so that it's more useful to the community around the Kubernetes Image Builder?

* Make this work for more than just Openstack so that it's more useful to the community around the Kubernetes Image
Builder?
* Remove dependency on GitHub Pages in the publish section - have this generate an artifact instead
* Fail on CVE critical discovery and remove any uploaded image
* Add metrics/telemetry to the process
* Create all option to allow whole process?

# License
Expand Down
60 changes: 60 additions & 0 deletions baski-example.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
cloud:
openstack:
clouds-file: "~/.config/openstack/clouds.yaml"
cloud-name: "image-builder"
network-id: "network-id"
source-image-id: "source-image"
flavor-name: "spicy-meatball"
attach-config-drive: false
use-floating-ip: true
floating-ip-network-name: "Internet"
image-visibility: "public"
image-disk-format: "raw"
volume-type: ""
rootfs-uuid: "ROOT_FS_UUID" # The image in Openstack will be tagged with this. Useful for bare-metal in some use cases.
build:
verbose: true
build-os: "ubuntu-2204"
image-prefix: "kube"
image-repo: "https://github.com/kubernetes-sigs/image-builder.git"
crictl-version: "1.26.0"
cni-version: "1.2.0"
kubernetes-version: "1.26.3"
extra-debs: "nfs-common"
add-trivy: true
add-falco: true
nvidia:
enable-nvidia-support: true
nvidia-driver-version: "525.85.05" # Currently used for tagging metadata - may be removed in future in favor of parsing the filename.
nvidia-bucket-endpoint: "S3_ENDPOINT_URL"
nvidia-bucket-name: "nvidia"
nvidia-bucket-access: "ACCESS_KEY"
nvidia-bucket-secret: "SECRET_KEY"
nvidia-installer-location: "NVIDIA-Linux-x86_64-525.85.05-grid.run"
nvidia-tok-location: "client_configuration_token.tok"
nvidia-gridd-feature-type: "4"
scan:
image-id: "" # Used for existing images
auto-delete-image: false
skip-cve-check: false
max-severity-score: 7.0 # Minimum severity score to check for
max-severity-type: MEDIUM # Minimum severity to check for
sign:
generate:
path: "." # Output path of any generated keys
vault:
url: "https://vault.ENDPOINT/"
token: "VAULT_TOKEN"
image-id: "" # Used for existing images
private-key: "" # Takes precedence over vault.
public-key: "" # Takes precedence over vault.
digest: "" # Used to verify a digest. Not required for image signing.
publish:
image-id: "" # Used for existing images
github:
user: "some-user"
account: "some-account" # Used for none personal accounts - leave blank if personal-account/project
project: "some-project"
token: "123456789"
pages-branch: ""
#results-file: "/tmp/results.json" # Not currently used
20 changes: 20 additions & 0 deletions docs/openstack.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
### Openstack guidelines

It is expected that you have a network and sufficient security groups in place to run this.<br>
It will not create the network or security groups for you.

For example:

```
openstack network create image-builder
openstack subnet create image-builder --network image-builder --dhcp --dns-nameserver 1.1.1.1 --subnet-range 10.10.10.0/24 --allocation-pool start=10.10.10.10,end=10.10.10.200
openstack router create image-builder --external-gateway public1
openstack router add subnet image-builder image-builder

OS_SG=$(openstack security group list -c ID -c Name -f json | jq '.[]|select(.Name == "default") | .ID')
openstack security group rule create "${OS_SG}" --ingress --ethertype IPv4 --protocol TCP --dst-port 22 --remote-ip 0.0.0.0/0 --description "Allows SSH access"
openstack security group rule create "${OS_SG}" --egress --ethertype IPv4 --protocol TCP --dst-port -1 --remote-ip 0.0.0.0/0 --description "Allows TCP Egress"
openstack security group rule create "${OS_SG}" --egress --ethertype IPv4 --protocol UDP --dst-port -1 --remote-ip 0.0.0.0/0 --description "Allows UDP Egress"
```

Then craft a `baski.yaml` file based on the [example](../baski-example.yaml) supplied and run the commands you require.
22 changes: 11 additions & 11 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,19 @@ go 1.19
require (
github.com/go-git/go-git/v5 v5.6.1
github.com/google/uuid v1.3.0
github.com/gophercloud/gophercloud v1.2.0
github.com/gophercloud/utils v0.0.0-20230316075016-e15d7ee3ba3b
github.com/gophercloud/gophercloud v1.3.0
github.com/gophercloud/utils v0.0.0-20230330070308-5bd5e1d608f8
github.com/hashicorp/vault/api v1.9.0
github.com/pkg/sftp v1.13.5
github.com/spf13/cobra v1.6.1
github.com/spf13/cobra v1.7.0
github.com/spf13/viper v1.15.0
golang.org/x/crypto v0.7.0
golang.org/x/crypto v0.8.0
gopkg.in/yaml.v3 v3.0.1
)

require (
github.com/Microsoft/go-winio v0.6.0 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230321155629-9a39f2531310 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230411080316-8b3893ee7fca // indirect
github.com/acomagu/bufpipe v1.0.4 // indirect
github.com/cenkalti/backoff/v3 v3.2.2 // indirect
github.com/cloudflare/circl v1.3.2 // indirect
Expand All @@ -38,7 +38,7 @@ require (
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
github.com/hashicorp/go-sockaddr v1.0.2 // indirect
github.com/hashicorp/hcl v1.0.1-vault-5 // indirect
github.com/imdario/mergo v0.3.14 // indirect
github.com/imdario/mergo v0.3.15 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
Expand All @@ -61,12 +61,12 @@ require (
github.com/spf13/pflag v1.0.5 // indirect
github.com/subosito/gotenv v1.4.2 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
golang.org/x/mod v0.9.0 // indirect
golang.org/x/net v0.8.0 // indirect
golang.org/x/sys v0.6.0 // indirect
golang.org/x/text v0.8.0 // indirect
golang.org/x/mod v0.10.0 // indirect
golang.org/x/net v0.9.0 // indirect
golang.org/x/sys v0.7.0 // indirect
golang.org/x/text v0.9.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/tools v0.7.0 // indirect
golang.org/x/tools v0.8.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/square/go-jose.v2 v2.6.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
Expand Down
Loading