Create asset graph engine #120

staebler · 2018-08-10T18:17:27Z

This PR establishes the initial framework for generating assets using a graph engine (https://jira.coreos.com/browse/CORS-758).

Adds an openshift-install binary for generating assets. This is not hooked up to the build process yet. To build the executable, run go build ./cmd/openshift-install.
Adds an asset.Store type that generates an asset by walking the tree of dependencies for the asset ensuring that each asset of a dependency has been generated prior to generating the dependent asset.
Adds an asset.Stock type that establishes the available assets that can be generated.
Adds the install-config asset and its dependent assets (https://jira.coreos.com/browse/CORS-760, https://jira.coreos.com/browse/CORS-759). To generate the install-config asset, run ./openshift-install install-config. The user-provided assets as crude with respect to UX and still need further refinement. The install-config asset does not fully populate the install-config.yml yet.
Vendors https://github.com/stretchr/testify to provide some convenience functions for use in unit tests.
Vendors https://github.com/pborman/uuid to generate a random UUID for the cluster ID.

abhinavdahiya · 2018-08-10T21:41:49Z

installer/pkg/nextgen/asset.go

+
+type Asset interface {
+	Dependencies() []Asset
+	Generate([]*AssetState) (*AssetState, error)


It would nice if we Generate could take entire State https://github.com/openshift/installer/blob/master/Documentation/design/assetgeneration.md#state

What would be nice about getting the entire State? The asset has already stated what its dependencies are. If it needs anything from State beyond those dependencies, then it is using a dependencies that it did not declare. The AssetStore has already fetched all of the states for all of the dependencies, so there is no need for the asset to relocate them from the entire State.

abhinavdahiya · 2018-08-10T22:02:13Z

installer/pkg/nextgen/asset/graph.go

@@ -0,0 +1,35 @@
+package asset


We should try to avoid defining this graph defining.

abhinavdahiya · 2018-08-10T22:30:15Z

installer/pkg/nextgen/asset/installconfig.go

+		return nil, fmt.Errorf("unknown platform type %q", platform)
+	}
+
+	json, err := jsoniter.ConfigFastest.Marshal(installConfig)


not sure why specfically marshal to json then, marshal to yaml is required. "github.com/ghodss/yaml".Marshal() already does it?

abhinavdahiya · 2018-08-13T17:36:58Z

installer/pkg/nextgen/asset/graph.go

+
+type Graph struct {
+	// Targetable assets
+	InstallConfig nextgen.Asset


Why do we need this specialization.

Targetable assets vs Non-targetable assets

wking · 2018-08-11T13:34:58Z

installer/pkg/nextgen/asset/graph.go

+	// Targetable assets
+	InstallConfig nextgen.Asset
+
+	// Non-targetable assets


Do we need a targetable/non-targetable distinction? Why couldn't the password asset, etc. correspond to on-disk files?

We don't need the distinction. There is no code that differentiates a targetable asset from a non-targetable asset. This is just a helpful delineation to make clear which asset are the ones that the user would directly generate as opposed to being indirectly generated by other assets.

I don't see why the password asset could not correspond to on-disk files.

wking · 2018-08-11T13:41:24Z

installer/pkg/nextgen/asset.go

+package nextgen
+
+type Asset interface {
+	Dependencies() []Asset


Related to @abhinavdahiya's comment, this should return a slice of parent hashes, not the parent assets themselves. That makes it a bit easier to get the asset's Merkle hash, and you would Fetch the parent by hash if you needed more detail about it.

So you explain a little more how the assetstore might use the hash from Dependencies to actually generate the corresponding asset.

Dependencies() defines the assets it requires. asset store ensures that those assets are in state before calling generate on the it asset.

I missed the discussion related to using a Merkle DAG for the asset store. Could someone elaborate on how we plan on using the properties of a Merkle DAG? Maybe this could be added to the design document.

wking · 2018-08-11T13:47:48Z

installer/pkg/nextgen/assetstore.go

+package nextgen
+
+type AssetStore interface {
+	Fetch(Asset) (*AssetState, error)


nit: Maybe I'm biased by Git, but for me "fetch" hints of remote access. Can we use the more generic Get? And adjust this to be content-addressable retrieval:

Get(hash string) (Asset, error)

I also think the store needs a property for the root hash (or maybe a slice of root hashes?) so we can walk the current DAG rebuilding/extracting assets without tripping over stale entries.

wking · 2018-08-11T13:53:01Z

installer/pkg/nextgen/asset/platform.go

+	LibvirtPlatformType = "libvirt"
+)
+
+type Platform struct{}


It feels like the platform asset should be an instance of UserProvided instead of a type in its own right. Maybe add a Validate property holding a validation function to UserProvided?

I don't think it much matters whether it is one type or two separate types. In the end, I decided on having Platform as a separate type because there was extra code that needed to go somewhere to define the Validate for the platform and to define the acceptable platforms. Either I put this extra code in stock.go, which will tend to make that file grow too large, or I create a separate file for the extra platform code. If I create a separate file, then I may as well create a separate type, too, which has the extra benefit of keeping the UserProvided type smaller.

yifan-gu · 2018-08-15T00:44:01Z

installer/cmd/nextgen/main.go

@@ -0,0 +1,50 @@
+package main


Can we put it under just ${installer_root}/pkg?
I was expecting something like:

openshift-installer/ ├── cmd │ └── openshift-install └── pkg └── graph(or asset?)

We should just create a greenfield for the new installer binary.

yifan-gu · 2018-08-15T01:36:46Z

installer/pkg/nextgen/asset/graph.go

+	g.License = &UserProvided{Prompt: "License: "}
+	g.PullSecret = &UserProvided{Prompt: "Pull Secret: "}
+	g.Platform = &UserProvided{Prompt: "Platform: "}
+	g.EmailAddress = &UserProvided{Prompt: "Email Addres: "}


s/Email Addres/Email Address

yifan-gu · 2018-08-15T02:31:12Z

installer/pkg/nextgen/asset/installconfig.go

+}
+
+func (a *InstallConfig) Generate(dependencies []*nextgen.AssetState) (*nextgen.AssetState, error) {
+	//emailAddress := string(dependencies[0].Contents[0].Data)


maybe we should pass a map?

yifan-gu · 2018-08-15T18:58:35Z

@staebler lgtm so far

wking · 2018-08-15T21:44:24Z

pkg/types/installconfig.go

+	metav1.ObjectMeta `json:"metadata"`
+
+	// ClusterID is the ID of the cluster.
+	ClusterID uuid.UUID `json:"clusterID"`


I'd rather have this be a []byte (or hex-encoded string?). Is there a reason to preserve the UUID type in this config? Using a UUID algorithm to generate a unique ID makes sense, but once it's generated I think we can treat it as an opaque ID and won't need access to any UUID-specific methods.

In the design (https://github.com/openshift/installer/blame/master/Documentation/design/installconfig.md#L246), it is specified as a uuid.UUID, so that is what I used. I agree that we can likely treat is as an opaque ID, but in the short-term I will leave it as a uuid.UUID to match the design.

wking · 2018-08-15T23:17:51Z

cmd/openshift-installer/main.go

+	log "github.com/Sirupsen/logrus"
+	"gopkg.in/alecthomas/kingpin.v2"
+
+	"github.com/openshift/installer/pkg/asset"


I like these new package paths, avoiding our current github.com/openshift/installer/installer/... redundancy. But to get CI tests for these new locations, we'll need updates to go-lint.sh, go-vet.sh, ci-operator (also here), etc. I'm not sure about the best way to roll those out. Maybe something like:

Land an installer PR with a stub cmd/openshift-installer/main.go and pkg/ directory with the associated local CI updates.

Land a release PR with updates to add tests for the new locations.

Come back and rebase this PR around the stubs from step 1.

Will it be OK to add the CI updates after this PR lands?

Will it be OK to add the CI updates after this PR lands?

I've filed openshift/release#1289 bumping unit and go-lint to cover the new directories. And I've filed #185 with the local hack/ changes.

abhinavdahiya · 2018-08-16T01:12:27Z

cmd/openshift-installer/main.go

@@ -0,0 +1,49 @@
+package main


nit: maybe just rename this folder to cmd/openshift-install/main.go,
So the binary will be openshift-install.
Then we can say things like
openshift-install installconfig
openshift-install manifests
openshift-install ignconfigs
openshift-install cluster

Then we can say things like

openshift-install installconfig openshift-install manifests

👍 to cmd/openshift-install, but I don't see why we'd want sub-commands for subsets of the graph. Why not generate to (re)generate all the assets, install to push them out, and destroy to tear down?

@wking The sub-commands are used so that the user has an opportunity to make manual modifications to the generated assets from the previous step before they are used by subsequent steps.

wking · 2018-08-16T17:00:19Z

pkg/asset/userprovided.go

+	for {
+		fmt.Print(prompt)
+		var input string
+		if _, err := fmt.Scanln(&input); err != nil {


It seems like Scanln is breaking on whitespace (regardless of newline-ness). I think we may need to use a Scanner.Scan (although there are other options as well). Here's a play showing the issue (note the "expected newline" error on the first Fscanln).

For most of our inputs (email addresses, domain names, ...), we don't want spaces in the input. But the pull secret is JSON, which may contain spaces. And folks could have spaces in their password(/phrase). So where we want to enforce "this line contains no spaces" (or "this line contains an @ and other valid email-address structure"), I'd rather leave that to explicit validation, instead of having it be a limit imposed by the generic queryUser.

wking · 2018-08-16T20:04:46Z

pkg/asset/stock.go

+	"os"
+)
+
+// Stock is the stock of assets that cen be generated.


nit: typo "cen" -> "can"

wking · 2018-08-17T19:44:26Z

pkg/asset/userprovided.go

+			input = input[:len(input)-1]
+		}
+		if a.validation != nil {
+			validatedInput, ok := a.validation(input)


I'm fine returning a cleaned version of the raw input (your validatedInput), but I think the UX would be better if you returned an error with a message instead of the boolean ok. One benefit of that approach is that you can use a number of our existing validators (with a wrapper if you keep validatedInput). For example, see the ValidateInput type in my mock-up, used here and configured here.

yifan-gu · 2018-08-17T22:02:21Z

@staebler Was implementing the tls assets generation (#145) based on this PR, but then I realized the baseDomain is used not in the installconfig?
Thought that the installconfig should store those user inputs?

abhinavdahiya · 2018-08-18T01:49:31Z

@yifan-gu There are few other missing fields, Admin.Email Admin.Password SSHKey(public key) BaseDomain We should add these fields iteratively ? Or atleast add these mentioned above and update installconfig.md ?

- stretchr/testify - pborman/uuid

openshift-ci-robot · 2018-08-24T20:01:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: staebler, yifan-gu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [yifan-gu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wking · 2018-08-24T20:11:58Z

/lint

^ this should run checks on pkg/... as well, since we're currently not covering those in our golint Prow job.

openshift-ci-robot

@wking: 12 warnings.

In response to this:

/lint

^ this should run checks on pkg/... as well, since we're currently not covering those in our golint Prow job.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2018-08-24T20:12:08Z

pkg/asset/state.go

@@ -0,0 +1,36 @@
+package asset
