CI: Use specific version of xurls utility #856

jodh-intel · 2018-10-25T10:39:11Z

Build the version of xurls specified in the runtimes versions
database to avoid problems; the latest version of xurls requires
golang 1.10.3 but that is not available in the stable branches.

Includes rework to the .ci/lib.sh functions to support building a particular version of a package.

Fixes #855.

Signed-off-by: James O. D. Hunt james.o.hunt@intel.com

jodh-intel · 2018-10-25T10:39:24Z

Blocked on kata-containers/runtime#855.

grahamwhaley

Well, that was slightly more convoluted than I expected ;-)
lgtm

jodh-intel · 2018-10-25T13:42:05Z

@chavafg - branch updated with the changes from kata-containers/runtime#855.

jodh-intel · 2018-10-25T13:43:21Z

No longer blocked on kata-containers/runtime#855 as this PR now contains those changes.

chavafg

lgtm

chavafg · 2018-10-25T13:50:57Z

.ci/lib.sh

+
+	[ ! -f "$versions_file" ] && die "cannot find $versions_file"
+
+	# This is needed in order to retrieve the version for qemu-lite


I think this comment can now be removed...

Zapped! 😄

chavafg · 2018-10-25T13:53:17Z

and travis now failing on ppc64

jodh-intel · 2018-10-25T14:08:14Z

@nitkon - any ideas? I don't think I've changed anything architecture dependent on this PR, but from the Travis ppc64 logs...

INFO: Installing xurls utility
/home/travis/gopath/bin/yq: line 1: Not: command not found
/home/travis/gopath/bin/yq: line 1: Not: command not found
ERROR: need version to build

grahamwhaley · 2018-10-25T14:11:38Z

@jodh-intel - did you drop an install_yq, which although it said it was needed for qemu, was maybe also needed elsewhere?

jodh-intel · 2018-10-25T14:29:41Z

@grahamwhaley - not that I can see, no.

It appears that install_yq() doesn't actually support ppc64le yet:

https://github.com/kata-containers/tests/blob/master/.ci/lib.sh#L70..L83

... although yq is available fwics:

The question is, how was this working before on Travis?!?

grahamwhaley · 2018-10-25T14:32:47Z

:-) OK @jodh-intel - we'll wait to see what @nitkon thinks.
What I saw in the changed files was:

function get_version(){
	dependency="$1"
	GOPATH=${GOPATH:-${HOME}/go}
-	# This is needed in order to retrieve the version for qemu-lite
-	install_yq >&2
``

jodh-intel · 2018-10-25T14:36:06Z

@grahamwhaley - ah - you've only got part of the context I think:

$  git diff master | grep install_yq
@@ -100,20 +123,43 @@ function install_yq() {
+       install_yq >&2
-       install_yq >&2

grahamwhaley · 2018-10-25T14:38:02Z

Ah, I see it - not obvious on the github view as the addition was somewhat above. OK, np

chavafg · 2018-10-25T14:52:20Z

seems like failing here, we haven't added the ppc64le support:

tests/.ci/lib.sh

Lines 93 to 106 in cb7a9d3

    
           case "${sysInfo[1]}" in 
        
           "aarch64") 
        
           	goarch=arm64 
        
           	;; 
        
           "x86_64") 
        
           	goarch=amd64 
        
           	;; 
        
           "s390x") 
        
           	goarch=s390x 
        
           	;; 
        
           "*") 
        
           	die "Arch ${sysInfo[1]} not supported" 
        
           	;; 
        
           esac

jodh-intel · 2018-10-25T15:05:25Z

@chavafg - #859 raised to resolve that. Do we want to fix that first, or just land this since?

chavafg · 2018-10-25T15:54:58Z

I'll prefer to fix #859 first, since travis will start failing on every other PR.
Opened #860 to address the issue.

chavafg · 2018-10-25T16:36:31Z

/test

GabyCT · 2018-10-25T16:37:36Z

lgtm

chavafg · 2018-10-25T16:59:49Z

See failures in the CI:

11:45:38 INFO: Building github.com/kata-containers/agent
11:45:38 make: *** empty string invalid as file name.  Stop.
11:45:38 Build step 'Execute shell' marked build as failure

Sort the entries in the tests versions database. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Stop `yamllint` bleating. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Add a description to the tests versions database. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Add the [`xurls`](https://mvdan.cc/xurls/cmd/xurls) utility to the versions database. This tool is used by the CI scripts in the `tests` repo to check URLs for validity. However, the latest version of `xurls` requires golang 1.10.3+, but we don't need to use the latest version and doing so is causing us problems (see kata-containers/kata-containers#32). Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Fix `build()` to use the provided make target variable. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Update `build()` to only run the `autogen.sh` script if it exists. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Change `build()` to use `go get -d` and an explicit `go install` rather than a `go get` to support a future change that will allow a specific version of a go project to be built. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Rework `build()` into `build_version()` and add a new version of `build()` that will build the latest package version by default. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Refactor `get_version()` into `get_dep_from_yaml_db()` and provide a new `get_test_version()` to allow querying the versions database in the tests repo rather than the runtime one. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Build the version of `xurls` specified in the runtimes versions database to avoid problems; the latest version of `xurls` requires golang 1.10.3 but that is not available in the stable branches. Fixes kata-containers#855. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

jodh-intel · 2018-10-26T08:28:46Z

/test

This was referenced Oct 25, 2018

create/run: Make bundle path default to cwd kata-containers/runtime#836

Merged

create/run: Make bundle path default to cwd kata-containers/runtime#838

Merged

grahamwhaley approved these changes Oct 25, 2018

View reviewed changes

jodh-intel force-pushed the ci-build-versioned-xurls branch from d171946 to a5dd7da Compare October 25, 2018 11:08

jodh-intel mentioned this pull request Oct 25, 2018

versions: Add xurls utility kata-containers/runtime#855

Closed

jodh-intel force-pushed the ci-build-versioned-xurls branch from a5dd7da to be0a0f7 Compare October 25, 2018 13:40

chavafg approved these changes Oct 25, 2018

View reviewed changes

jodh-intel force-pushed the ci-build-versioned-xurls branch from be0a0f7 to cb7a9d3 Compare October 25, 2018 13:55

jodh-intel mentioned this pull request Oct 25, 2018

install_yq() doesn't support ppc64 #859

Closed

jodh-intel force-pushed the ci-build-versioned-xurls branch 2 times, most recently from 2a3fb1d to 8e7873f Compare October 25, 2018 16:21

jodh-intel added 3 commits October 26, 2018 09:23

versions: Sort file

8d24555

Sort the entries in the tests versions database. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

versions: Add missing document separator

b32c10f

Stop `yamllint` bleating. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

versions: Add a description

eb69a4d

Add a description to the tests versions database. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

jodh-intel added 7 commits October 26, 2018 09:23

CI: Actually use the make target

1bb7476

Fix `build()` to use the provided make target variable. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

CI: Check if autogen.sh exists before running

cfd5e17

Update `build()` to only run the `autogen.sh` script if it exists. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

CI: Use explicit "go install"

5ac9637

Change `build()` to use `go get -d` and an explicit `go install` rather than a `go get` to support a future change that will allow a specific version of a go project to be built. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

CI: Allow specific project version to be built

389d570

Rework `build()` into `build_version()` and add a new version of `build()` that will build the latest package version by default. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

CI: Support tests repo versions database query

f430f47

Refactor `get_version()` into `get_dep_from_yaml_db()` and provide a new `get_test_version()` to allow querying the versions database in the tests repo rather than the runtime one. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

jodh-intel force-pushed the ci-build-versioned-xurls branch from 8e7873f to 3d61716 Compare October 26, 2018 08:27

jodh-intel merged commit d0ef722 into kata-containers:master Oct 26, 2018

marcov mentioned this pull request Oct 29, 2018

Really fix xurls URL #841

Closed

jodh-intel mentioned this pull request Nov 16, 2018

Add xurls to versions database kata-containers/runtime#854

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Use specific version of xurls utility #856

CI: Use specific version of xurls utility #856

jodh-intel commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

grahamwhaley left a comment

jodh-intel commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

chavafg left a comment

chavafg Oct 25, 2018

jodh-intel Oct 25, 2018

chavafg commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

grahamwhaley commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

grahamwhaley commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

grahamwhaley commented Oct 25, 2018

chavafg commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

chavafg commented Oct 25, 2018

chavafg commented Oct 25, 2018

GabyCT commented Oct 25, 2018 •

edited by amshinde

Loading

chavafg commented Oct 25, 2018

jodh-intel commented Oct 26, 2018


		[ ! -f "$versions_file" ] && die "cannot find $versions_file"

		# This is needed in order to retrieve the version for qemu-lite

CI: Use specific version of xurls utility #856

CI: Use specific version of xurls utility #856

Conversation

jodh-intel commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

grahamwhaley left a comment

Choose a reason for hiding this comment

jodh-intel commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

chavafg left a comment

Choose a reason for hiding this comment

chavafg Oct 25, 2018

Choose a reason for hiding this comment

jodh-intel Oct 25, 2018

Choose a reason for hiding this comment

chavafg commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

grahamwhaley commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

grahamwhaley commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

grahamwhaley commented Oct 25, 2018

chavafg commented Oct 25, 2018

jodh-intel commented Oct 25, 2018

chavafg commented Oct 25, 2018

chavafg commented Oct 25, 2018

GabyCT commented Oct 25, 2018 • edited by amshinde Loading

chavafg commented Oct 25, 2018

jodh-intel commented Oct 26, 2018

GabyCT commented Oct 25, 2018 •

edited by amshinde

Loading