Kata 1.5.0 Installation and documentation errors

[fredbcode]

Documentation about installation is missing

https://github.com/kata-containers/runtime/blob/1.5.0/installation link from https://github.com/kata-containers/runtime/releases

And Ubuntu package is broken

apt-update
apt-get install kata-runtime
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
kata-runtime : Depends: kata-shim (>= 1.5.0+git.efbf3bb-42) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

[fredbcode]

Shim is not updated

Package: kata-containers-image
Version: 1.5.0-40
Architecture: amd64
Maintainer: Kata containers team https://github.com/kata-containers/
Installed-Size: 536410
Filename: ./amd64/kata-containers-image_1.5.0-40_amd64.deb
Size: 31128476
MD5sum: e50429fc3ab9af1aae72634451fb3616
SHA1: 37a78086b7a9965a65fe4862881c10adb94f5681
SHA256: 16dc1228fcaa27a5b20729ee028da6976d1cc9bd84af50d9d8c34336f3142870
Section: devel
Priority: optional
Homepage: https://katacontainers.io
Description: Kata containers image
Description-md5: ca1a3f858d55e2f6b0af272a60d93bc5

Package: kata-ksm-throttler
Version: 1.5.0.git+3dd4c9f-41
Architecture: amd64
Maintainer: Kata containers team https://github.com/kata-containers/
Installed-Size: 16558
Filename: ./amd64/kata-ksm-throttler_1.5.0.git+3dd4c9f-41_amd64.deb
Size: 4055784
MD5sum: 9bbda1d26aa43dc6c5c5af466ce94fea
SHA1: cc962e3bb6615f1dd7a961a75b842ccfca8e913a
SHA256: 083e88989533207ed19a1ed5a97e5fca3b63d32fc74bf7cf515b91caa0ff0119
Section: devel
Priority: optional
Homepage: https://katacontainers.io
Description: This project implements a Kernel Same-page Merging throttling daemon.
Its goal is to regulate KSM by dynamically modifying the KSM sysfs entries, in order to minimize memory duplication as fast as possible while keeping the KSM daemon load low.
Description-md5: 46b69168f9c57ac9e4261179c1a19b36

Package: kata-linux-container
Version: 4.14.67.22-143
Architecture: amd64
Maintainer: Kata containers team https://github.com/kata-containers/
Installed-Size: 34042
Filename: ./amd64/kata-linux-container_4.14.67.22-143_amd64.deb
Size: 11660888
MD5sum: ed2df7d75491a17d3d6f9bb2b3e43ff0
SHA1: a3c902be283e4dbc42727e7a778faf2ad9513068
SHA256: 7ac8a9ca422500c0c67bcb5fb5d788910d6cc709c2d650d04f59bfb8e7f5a2af
Section: devel
Priority: optional
Homepage: https://katacontainers.io
Description: linux kernel optimised for container-like workloads.
Linux kernel optimised for container-like workloads
Description-md5: ad1e538ac61beea6f4028c77055b504b

Package: kata-linux-container-debug
Source: kata-linux-container
Version: 4.14.67.22-143
Architecture: amd64
Maintainer: Kata containers team https://github.com/kata-containers/
Installed-Size: 2748
Filename: ./amd64/kata-linux-container-debug_4.14.67.22-143_amd64.deb
Size: 439420
MD5sum: bdc3c95f2b87dd77f6a89ec45f0bdc35
SHA1: bb24884bedb40c03afcbbe603b7553ccede42de4
SHA256: c841106d0d276509436411dea100230c65d8a4fac59a304b33e6f69bb13715c9
Section: devel
Priority: optional
Homepage: https://katacontainers.io
Description: Debug components for the kata-linux-container package.
This package includes the kernel config and the kernel map.
Description-md5: 9aeb16ea97cfa05105f8263b05516016

Package: kata-proxy
Version: 1.5.0+git.9e77a0b-41
Architecture: amd64
Maintainer: Kata containers team https://github.com/kata-containers/
Installed-Size: 3624
Filename: ./amd64/kata-proxy_1.5.0+git.9e77a0b-41_amd64.deb
Size: 926196
MD5sum: 195282089194aff7c8375332c3ddcd29
SHA1: 386513ce2e76279e04628babf90baa018db35dc1
SHA256: 7dc269ce54de97e9b8395984318a1824a0d8a4b0e10c8a8d283aa45eb851d7d2
Section: devel
Priority: optional
Homepage: https://katacontainers.io
Description: kata-proxy works alongside the Kata Containers runtime and shim to provide a VM-based OCI runtime solution.
kata-proxy is a daemon offering access to the hyperstart VM agent to both the runtime and shim processes.
Description-md5: 541109035d3eeca8ae8ac7e6a065eb11

Package: kata-runtime
Version: 1.5.0+git.5f7fcd7-56
Architecture: amd64
Maintainer: Kata containers team https://github.com/kata-containers/
Installed-Size: 45607
Depends: kata-containers-image (>= 1.5.0-40), kata-linux-container (>= 4.14.67.22-143), kata-proxy (>= 1.5.0+git.9e77a0b-41), kata-shim (>= 1.5.0+git.efbf3bb-42), kata-ksm-throttler (>= 1.5.0+git.3dd4c9f-41), qemu-lite (>= 2.11.0+git.87517af-57), qemu-vanilla (>= 2.11.2+git.0982a56-54)
Filename: ./amd64/kata-runtime_1.5.0+git.5f7fcd7-56_amd64.deb
Size: 10752284
MD5sum: 7ec09339efa61c6804a7a15151d4796e
SHA1: c08fa51916cf310b8a848f749a23235a55484d16
SHA256: d8db25b06b4166e63c515d49e38eb396f971a515f804a284559ffec078fff333
Section: devel
Priority: optional
Homepage: https://katacontainers.io
Description: An Open Containers Initiative (OCI) "runtime" that launches an Intel VT-x secured Kata Containers hypervisor, rather than a standard Linux container.
Description-md5: 9bd55bc436d874a08e7719aaaf8b70a8

Package: kata-shim
Version: 1.4.0+git.b02868b-41
Architecture: amd64
Maintainer: Kata containers team https://github.com/kata-containers/
Installed-Size: 11677
Filename: ./amd64/kata-shim_1.4.0+git.b02868b-41_amd64.deb
Size: 2540216
MD5sum: c2d32a458d2e7445cb7e45bb510311e0
SHA1: 94a7fd91268741acdd90292f88de5300026b4cf9
SHA256: 65cfd0cca6bbb0e069d35a4e182eba1b5cd8181a28178de6731058f9f9fc9a23
Section: devel
Priority: optional
Homepage: https://katacontainers.io
Description: kata-shim is a process spawned by the Intel VT-x secured Kata Containers runtime per container workload.
The runtime provides the pid of the kata-shim process to containerd-shim on OCI create command.
Description-md5: d4335e76a97063e91d5f223a022255ce

[grahamwhaley]

I think I saw something similar on Friday around the 'held broken packages', but thought it was just my system as I'd not updated packages for many many months...
/cc @jcvenegas @marcov

[jodh-intel]

Hi @fredbcode - thanks for reporting.

What happens if you explicitly run:

$ sudo apt -y install kata-runtime kata-shim

I didn't see this problem locally but I tend to apt-get upgrade.

/cc @jcvenegas.

@egernst, @chavafg, @marcov - We really need to look at kata-containers/ci#10 since this issue would have been caught.

[fredbcode]

Same error, anyway it can't works because shim still in 1.4.0 in mirror

[jodh-intel]

Sounds like you hit a bad OBS mirror maybe as...

• https://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/xUbuntu_18.04/amd64/
• https://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/Debian_9/amd64/

Both have kata-shim_1.5.0+git.efbf3bb-14_amd64.deb.

/cc @marcov, @egernst.

[fredbcode]

There is something wrong here : https://download.opensuse.org/repositories/home:/katacontainers:/release/xUbuntu_18.04/

[jodh-intel]

@fredbcode - yes, you are using the wrong repo I'm afraid - if you look at the doc below for example:

• https://github.com/kata-containers/documentation/blob/master/install/ubuntu-installation-guide.md

... you'll see that the URL is releases, not release. The new URL also includes the architecture as we plan to offer packages for other architectures.

If I was you, I would:

• Remove the existing Kata packages on your system
• If you're on Ubuntu/Debian, delete /etc/apt/sources.list.d/kata-containers.list.
• Re-install Kata from the official docs.

[fredbcode]

I found this mirror from previous documentation.

Now the installation link is also broken (as 1.5.0) https://github.com/kata-containers/runtime/blob/1.3.0-rc1/installation

[jodh-intel]

@jcvenegas, @marcov - can we just delete https://download.opensuse.org/repositories/home:/katacontainers:/release/ ? I'm not clear how it is being maintained but the problem @fredbcode is reporting makes it seem like OBS is copying the files from https://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/xUbuntu_18.04/amd64/ to https://download.opensuse.org/repositories/home:/katacontainers:/release/xUbuntu_18.04/. But in this case, the copy seems to have failed?

[jodh-intel]

@fredbcode - I'm confused - the link you've provided above in invalid but where is that being used? It's also clearly for an older version of Kata, so...?

[jodh-intel]

/cc @mssola, @flavio as I keep forgetting Marco is not about atm.

[fredbcode]

Thank you, I guess the old mirror should be removed (at least not with old and new packages)

[fredbcode]

From https://github.com/kata-containers/runtime/releases at the end (Installation) the links are wrong (404) for 1.5.0 or 1.3.0

[jodh-intel]

Thanks @fredbcode - I've raised an issue to resolve that:

• Fix install doc links in runtime-release-notes.sh packaging#320

[fredbcode]

Ok installation fixed for me now, but still this issue can be very dangerous for production server with an old source list (a 404 must be better than an incomplete packages installation)

[jodh-intel]

Agreed. I think we need to either remove that URL entirely ("forcing function" to make people upgrade :), or sym-link https://download.opensuse.org/repositories/home:/katacontainers:/release/xUbuntu_18.04 to https://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/xUbuntu_18.04 if that sort of thing is possible with OBS. If so, this should be a reasonable thing to do given that initially we only provided amd64 binaries.

[marcov]

The problem here is that shim wasn't building in home:katacontainers:release for any of the distributions, probably because of an hiccup.

As a consequence the kata-shim package left to download was the last one that built successfully, 1.4.0

I retriggered a build of shim, and it built successfully for all distros but Ubuntu 16.04.

I think we need to either remove that URL entirely ("forcing function" to make people upgrade :),

@jcvenegas any reasons to keep this subproject around?

or sym-link https://download.opensuse.org/repositories/home:/katacontainers:/release/xUbuntu_18.04 to https://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/xUbuntu_18.04 if that sort of thing is possible with OBS.

@jodh-intel Kind of. It's not possible to symlink but it's possible to use _aggregate to copy built packages from one repository to another

We really need to look at kata-containers/ci#10 since this issue would have been caught.

Agree and, for packages installation testing, building from a Dockerfile containing installation instructions would be enough.

can we just delete https://download.opensuse.org/repositories/home:/katacontainers:/release/ ? I'm not clear how it is being maintained

This subproject is updated from time to time by @jcvenegas.

it seem like OBS is copying the files from https://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/xUbuntu_18.04/amd64/ to https://download.opensuse.org/repositories/home:/katacontainers:/release/xUbuntu_18.04/. But in this case, the copy seems to have failed?

Nope, as I explained above it was a build failure.

[marcov]

/cc @mssola, @flavio as I keep forgetting Marco is not about atm.

I'm tempted to replace my avatar with an x-ray of my broken wrist 😆

[jodh-intel]

@marcov - ouch! Hope it's healing well!

@egernst - any thoughts on the above?

[grahamwhaley]

@jcvenegas @jodh-intel @mssola, @flavio
I'm a bit confused then - should we be using the obs dir release or releases?
I think all our install docs point to releases, but:

• the last modified timestamps at http://download.opensuse.org/repositories/home:/katacontainers:/
  show that release was updated on Dec 3rd, but releases was last updated on August 20th? That could be a red herring 🐟 though, as maybe only sub sub dirs updated, and the dirs thus maybe didn't...
• http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/xUbuntu_16.04/amd64/ has a kernel image file updated Jan 22nd, named kata-linux-container_4.14.67.22-18_amd64.deb
• http://download.opensuse.org/repositories/home:/katacontainers:/release/xUbuntu_16.04/amd64/ has a kernel image file updated 23rd Jan, named kata-linux-container_4.14.67.22-143_amd64.deb

(which has been throwing me somewhat sideways whilst trying to reproduce and analyse kata-containers/ci#102)

[jodh-intel]

We should be using "releases/" and not "release/" since the former then goes on to specify the architecture, such as:

• http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/xUbuntu_16.04/amd64/

The old "release" didn't include an architecture as at that point Kata only worked on x86_64 but I vote we kill that URL since:

• It's confusing everyone.
• Once the world is using "releases/", no further changes should be necessary (famous last words...)

[jodh-intel]

As for wtf is going on wrt timestamps and contents, @jcvenegas and @marcov can comment 😄

[grahamwhaley]

If we should be using releases, then the repo setups in the ci scripts worry me:
https://github.com/kata-containers/tests/blob/master/.ci/setup_env_ubuntu.sh#L61

	obs_url="http://download.opensuse.org/repositories/home:/katacontainers:/release/xUbuntu_$(lsb_release -rs)/"

Does that, and all of its cousins, need updating?

[jodh-intel]

@grahamwhaley - I think "release" and "releases" are supposed to be in sync, to ensure users of the old URL continue to get a good experience. But that premise failed recently as shown above. Once fixed, we're back to working, but I still think we should delete "release/" which would mean updating the ubuntu, fedora and centos install scripts to use releases/.

/cc @chavafg, @GabyCT, @jcvenegas.

[marcov]

about the releases/ vs release/ ambiguity:

• release/ used to be the only Kata repo on OBS.
• when we opted to have release for two stable and a master branch, new subprojects with the releases/ prefix started to be created for master and all stable releases.
• however some of the documentation / tests infrastructure was still using the old release/ naming, so @jcvenegas from time to time updates this subproject with the "current" stable release.

Given all the confusion arised, I'd propose to either kill release/ alltogether or create a link from release/ to releases/master as @jodh-intel proposed.

[jcvenegas]

@jodh-intel @marcov

• Lets remove the repository
• Update any reference in our docs or scripts

[jcvenegas]

Or even better let our scripts use static binaries.

[jcvenegas]

See kata-containers/tests#1107

[jcvenegas]

@fredbcode @jodh-intel are we missing something else? Docs ,release notes scripts, and CI updated.

[jodh-intel]

@jcvenegas - I think the only outstanding issue is to delete the old "release" URL?

wdyt about doing that @egernst, @gnawux?

[fredbcode]

Yes I agree, remove this URL is most "secure" for users, I broken a test machine with partial packages