feat: added retroactive scanning command

* Adds retroactive scanning command which can pick up all images with a prefix and scan them.
* Refactored code to prevent duplication around scanning
* Moved all references to compute image service to image service in gopher cloud
* Allowed cobra to manage errors via RunE.

.github/ISSUE_TEMPLATE/bug.md

@@ -0,0 +1,8 @@
+# What happened?
+
+# What did you expect to happen?
+
+# Steps to reproduce the problem?
+
+# Additional information
+Baski Version: `baski version`

.github/ISSUE_TEMPLATE/feature.md

@@ -0,0 +1,5 @@
+# What do you need?
+
+# Why do you need it?
+
+# Any other Suggestions?

.github/pull_request_template.md

@@ -0,0 +1,8 @@
+# What's Changed
+
+# Why is it required?
+
+# PR checklist
+- [ ] Run tests locally
+- [ ] Updated Readme
+- [ ] Updated Changelog

.pre-commit-config.yaml

@@ -2,6 +2,6 @@ repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.4.0
     hooks:
-      -   id: trailing-whitespace
+      #-   id: trailing-whitespace
       -   id: end-of-file-fixer
       -   id: check-added-large-files

CHANGELOG.md

@@ -1,26 +1,108 @@
 # Changelog
 
-## [ 12/04/2023 - v0.1.0-beta.1 ]
+## [ Upcoming Release ]
+
+### Changed/Added
+
+### Fixed
+
+### Deprecated
+
+## [ 12/07/2023 - v0.1.0-beta.8 ]
+
+### Changed/Added
+
+* Added additional scan command to enable scanning multiple images.
+* Changed log.fatals to returns so that RunE can handle the error.
+
+### Fixed
+* Fixed code to support new repo changes in kubernetes.
+
+### Deprecated
+
+* Removed references to publish command so that it can no longer be called - code will be removed in coming release.
+
+## [ 12/07/2023 - v0.1.0-beta.7 ]
+
+### Changed/Added
+
+* Added ability to pass in a list of container images to bake in.
+
+### Fixed
+
+* Switched out panics for logging errors.
+* Corrected names in GitHub actions.
+* fixed date tag on image as it was just setting RFC3339 rather than using it as the format.
+
+## [ 12/07/2023 - v0.1.0-beta.6 ]
+
+### Changed/Added
+
+* Added ability to pass in a list of container images to bake in.
+
+### Fixed
+
+* Switched out panics for logging errors.
+* Corrected names in GitHub actions.
+* fixed date tag on image as it was just setting RFC3339 rather than using it as the format.
+
+## [ 16/05/2023 - v0.1.0-beta.5 ]
+
+### Changed/Added
+
+* Enabled support for S3 backends when using S3 buckets.
+
+### Fixed
+
+* Build command was missing some flags - these have been added.
+
+## [ 2023/05/15 - v0.1.0-beta.4 ]
+
+### Added
+
+* Support for trivyignore and adding lists of CVEs to ignore.
+
+## [ 2023/05/09 - v0.1.0-beta.3 ]
+
+### Fixed
+
+* Ensured FIP creation and removal rather than just looking for one in the account to prevent race condition when
+  attaching an IP.
+
+## [ 2023/04/28 - v0.1.0-beta.2 ]
+
+### Fixed
+
+* Fixed Nvidia and security inclusions.
+
+## [ 2023/04/12 - v0.1.0-beta.1 ]
+
+### Changed/Added
 
-### ADDED
 * Added changelog
 * Refactored code to:
-  * Prevent using `viper.GetXXXX` across the codebase - now gets put into struct to allow one location to be updated rather than multiples.
-  * Begin work to allow more clouds to be added - still work to be done #36.
-  * Begin work to improve flags - still work to be done #33.
+    * Prevent using `viper.GetXXXX` across the codebase - now gets put into struct to allow one location to be updated
+      rather than multiples.
+    * Begin work to allow more clouds to be added - still work to be done #36.
+    * Begin work to improve flags - still work to be done #33.
 * Updated the config file requirements. *This is a breaking change and old configs will no longer work.*.
 
 ### Fixed
- * Trivy checksum now used to validate trivy download if required #32.
- * Added flags, which were previously missing, to support adding Trivy and Falco to the image #34.
+
+* Trivy checksum now used to validate trivy download if required #32.
+* Added flags, which were previously missing, to support adding Trivy and Falco to the image #34.
 
 ### Deprecated
-* The publish command will be reworked in an upcoming release to prevent the GitHub requirement. Instead, it will generate the files require to publish a single images scan results as an artifact with which the user can then decide what to do.
+
+* The publish command will be reworked in an upcoming release to prevent the GitHub requirement. Instead, it will
+  generate the files require to publish a single images scan results as an artifact with which the user can then decide
+  what to do.
 
 ## [ Previous versions ]
 
 * Up to this point, there has been no changelog supplied for previous versions as it was a rapid iterative process.
-* With the release of v0.1.0-beta.1, any changes will be logged and should one be a breaking change, it will incur a version bump.
-* Minor version bumps will be reserved for general changes
-* Patch version bumps will be for fixes and patches
+* With the release of v0.1.0-beta.1, any changes will be logged and should one be a breaking change, it will incur a
+  version bump.
+* Minor version bumps will be reserved for general changes.
+* Patch version bumps will be for fixes and patches.
 * The beta tags will be for superficial changes within a patch that require testing before a final release is created.

README.md

@@ -51,19 +51,11 @@ docker run --name baski -it --rm --env OS_CLOUD=some-cloud -v /path/to/openstack
 baski build / scan / sign
 ```
 
-### GitHub Pages - Deprecated
-
-You will need to set up your target repo for the GitHub Pages in advanced.
-It only requires a `gh-pages` branch for this to work.
-GitHub Pages should be configured to point to a `docs` directory as this is where the resulting static site will be
-placed.
-
 # TODO
-
+* Automatically clear up resources when ctrl-c is pressed.
 * Make this work for more than just Openstack so that it's more useful to the community around the Kubernetes Image
   Builder?
-* Remove dependency on GitHub Pages in the publish section - have this generate an artifact instead
-* Add metrics/telemetry to the process
+* Add metrics/telemetry to the process.
 * Create all option to allow whole process?
 
 # License

baski-example.yaml

@@ -29,10 +29,6 @@ cloud:
     volume-size: 0
     # The rootfs-UUID. In testing this has only been required for bare-metal instances as the bare-metal instance needs know about the rootfs, so it knows what to boot.
     rootfs-uuid: "ROOT_FS_UUID"
-    # This exists because sometimes a race condition happens in OpenStack where an instance is created and removed too quickly for it to happen. It's rare, but it occurs.
-    # The scan step has been identified as something that essentially runs too fast for OpenStack so this temporary measure has been put in place.
-    # If this is true a 5 minutes pause is placed on the scan step to prevent the server spinning up and down too quickly.
-    add-pause: true
 
 # Baski presumes there is an S3 endpoint available for the pulling of any items such as the .trivyignore and nvidia files.
 # This is because the image builder defaults to this for the Nvidia support, and it is presumed that most if not all people can have/can set up an S3 endpoint for authenticated storage.
@@ -93,8 +89,18 @@ build:
 
 # Scan stage options
 scan:
-  # The ID of the image to be scanned.
-  image-id: ""
+  # Used to scan a single image - takes an image ID and scans it.
+  single:
+    # The ID of the image to be scanned.
+    image-id: ""
+  # Used in the `existing` command - takes a wildcard and scans all images that match it.
+  multiple:
+    # The image-search is used to filter images. If this string is within the name of the image, it'll be selected for scanning.
+    image-search: "kube-"
+    # How many concurrent scans to perform.
+    concurrency: 2
+  # Override the cloud.[provider].flavor for the scan. This can help avoid using a large or gpu enabled node just for scanning.
+  flavor-name: "not-so-spicy-meatball"
   # Whether to auto-delete the image. This has been added for automation purposes in the scenario where the image is built then scanned right away.
   # Should the scan fail then the image shouldn't be available publicly and so can be automatically deleted from the infrastructure to make sure a vulnerable image is not deployed.
   auto-delete-image: false

pkg/cmd/build/build.go

@@ -17,12 +17,12 @@ limitations under the License.
 package build
 
 import (
+	"fmt"
 	"github.com/eschercloudai/baski/pkg/cmd/util/data"
 	"github.com/eschercloudai/baski/pkg/cmd/util/flags"
 	"github.com/eschercloudai/baski/pkg/constants"
 	ostack "github.com/eschercloudai/baski/pkg/openstack"
 	"github.com/spf13/cobra"
-	"log"
 	"path/filepath"
 )
 
@@ -47,13 +47,13 @@ images such as ones with GPU/HPC drivers/tools.
 
 To use baski to build an image, an Openstack cluster is required.`,
 		TraverseChildren: true,
-		Run: func(cmd *cobra.Command, args []string) {
+		RunE: func(cmd *cobra.Command, args []string) error {
 			o.SetOptionsFromViper()
 
 			cloudsConfig := ostack.InitOpenstack(o.CloudsPath)
 			packerBuildConfig := ostack.InitPackerConfig(o)
 			if !checkValidOSSelected(o.BuildOS) {
-				log.Fatalf("an unsupported OS has been entered. Please select a valid OS: %s\n", constants.SupportedOS)
+				return fmt.Errorf("an unsupported OS has been entered. Please select a valid OS: %s\n", constants.SupportedOS)
 			}
 
 			buildGitDir := CreateRepoDirectory()
@@ -62,7 +62,7 @@ To use baski to build an image, an Openstack cluster is required.`,
 			metadata := ostack.GenerateBuilderMetadata(o)
 			err := ostack.UpdatePackerBuildersJson(buildGitDir, metadata)
 			if err != nil {
-				log.Fatalln(err)
+				return err
 			}
 			capiPath := filepath.Join(buildGitDir, "images", "capi")
 			packerBuildConfig.GenerateVariablesFile(capiPath)
@@ -73,18 +73,19 @@ To use baski to build an image, an Openstack cluster is required.`,
 
 			err = BuildImage(capiPath, o.BuildOS, o.Verbose)
 			if err != nil {
-				log.Fatalln(err)
+				return err
 			}
 
 			imgID, err := data.RetrieveNewImageID()
 			if err != nil {
-				log.Fatalln(err)
+				return err
 			}
 
 			err = SaveImageIDToFile(imgID)
 			if err != nil {
-				log.Fatalln(err)
+				return err
 			}
+			return nil
 		},
 	}
 

pkg/cmd/publish/process.go

@@ -29,7 +29,7 @@ import (
 	"github.com/go-git/go-git/v5/plumbing"
 	"github.com/go-git/go-git/v5/plumbing/object"
 	"github.com/gophercloud/gophercloud/openstack"
-	"github.com/gophercloud/gophercloud/openstack/compute/v2/images"
+	"github.com/gophercloud/gophercloud/openstack/imageservice/v2/images"
 	"html/template"
 	"log"
 	"os"
@@ -104,7 +104,7 @@ func FetchExistingReports(gitDir string) ([]string, error) {
 	return reportPaths, nil
 }
 
-// Image represents an Image returned by the Compute API.
+// Image represents an Image returned by the API.
 type Image struct {
 	ID        string `json:"id"`
 	CreatedAt string `json:"created_at"`

pkg/cmd/root.go

@@ -18,7 +18,6 @@ package cmd
 
 import (
 	"github.com/eschercloudai/baski/pkg/cmd/build"
-	"github.com/eschercloudai/baski/pkg/cmd/publish"
 	"github.com/eschercloudai/baski/pkg/cmd/scan"
 	"github.com/eschercloudai/baski/pkg/cmd/sign"
 	"github.com/eschercloudai/baski/pkg/cmd/util/config"
@@ -46,7 +45,6 @@ It could be extended out to provide images for a variety of other builders howev
 		build.NewBuildCommand(),
 		sign.NewSignCommand(),
 		scan.NewScanCommand(),
-		publish.NewPublishCommand(),
 	}
 	cmd.AddCommand(commands...)