Create UBI based images for che-theia #13765

evidolob · 2019-05-15T11:17:57Z

We need eventually move to UBI images
With this issue we can start creating POC images for che-theia
Currently there are ubi8/nodejs-10, so we can start POC using that image as base image.
I propose to start with:

The text was updated successfully, but these errors were encountered:

mmorhun · 2019-06-14T06:24:38Z

To build theia image we need first build theia-dev because it is used for building of Che Theia from sources.

To build theia-dev based on UBI (nodejs-10) we lack at least two packages:

jq which is used for github quota analyzing
libxkbfile-devel - used to be able to build electron part of Theia

I've tried to leave theia-dev image based on Alpine node, and rework runtime theia image to UBI, but this doesn't work because of differences in system libs against which Theia was compiled and ran (musl libc vs glibc at least).

Using official nodejs image as a base for theia-dev is not a good option, because we need to patch it, at least git, which fails to clone with fatal: Unable to find remote helper for 'https'. Decided not to spend time on that.

The approach which works is to use CentOS as a base for theia-dev image (but in this case we must handle nodejs and yarn installation ourselves). Then use UBI (nodejs-10) for runtime theia image.

nickboldt · 2019-09-28T15:31:31Z

benoitf · 2019-09-30T13:25:17Z

generator was in a separated repository (ws-skeleton) and then published to npmjs

as it's now in che-theia repository, it should not be retrieved through npmjs

benoitf · 2019-09-30T13:33:28Z

why hardcoded version @theia/generator-plugin@0.0.1-1540209403 vs. using latest?

to be sure that the version used is working as expected.

also let say you run the build 2 months later, it may not work at all

AndrienkoAleksandr · 2019-09-30T14:52:47Z

Collect the contents of these for use in Brew:
. /home/theia-dev/.yarn-global/node_modules/
. /usr/local/share/.cache/yarn/
. /opt/app-root/src/.npm-global/

I think from time to time content these folders changes and become outdated. Also If you want these folders in the some another images I think multi-stage build could help.

AndrienkoAleksandr · 2019-09-30T14:57:08Z

Figure out what the MONACO_CDN_PREFIX and CDN_PREFIX stuff is about

To speed up che-theia loading we are caching che-theia scripts and push it to the CDN registry like artifact. Then on start theia browser downloads scripts from the closest CDN server, that why theia loading faster.

nickboldt · 2019-09-30T14:59:00Z

why hardcoded version @theia/generator-plugin@0.0.1-1540209403 vs. using latest?

to be sure that the version used is working as expected.

also let say you run the build 2 months later, it may not work at all

Right but why a DIFFERENT version in che-theia/theia vs che-theia/theia-dev?

theia-dev: @theia/generator-plugin@0.0.1-1540209403
theia: @theia/generator-plugin@0.0.1-1562578105

Surely you want the same version in both? Seems like hardcoding this in the dockerfiles vs. in a package.json or yarn.lock is what's causing the mismatch (update one, forget the other) ?

nickboldt · 2019-09-30T15:00:08Z

To speed up che-theia loading we are caching che-theia scripts and push it to the CDN registry like artifact. Then on start theia browser downloads scripts from the closest CDN server, that why theia loading faster.

I'm not sure we can do that for the product unless we have a RH-approved repo where we're publishing. What's the full URL of the CDN we're using?

nickboldt · 2019-09-30T15:01:34Z

I think from time to time content these folders changes and become outdated. Also If you want these folders in the some another images I think multi-stage build could help.

No, I need to pull the contents of these folders so that I can push them into the Dockerfile build inside Brew. Brew can't see the outside world, so I need to tar up these node_modules and then push the tarball into pkgs.devel.redhat.com so we have a stored copy of the sources/binaries from which to build offline.

nickboldt · 2019-09-30T15:03:04Z

generator was in a separated repository (ws-skeleton) and then published to npmjs

as it's now in che-theia repository, it should not be retrieved through npmjs

Do you mean @eclipse-che/theia-generator or @theia/generator-plugin ?

If you mean @eclipse-che/theia-generator... should we also be building @theia/generator-plugin instead of installing it from npm?

Ref:

https://github.com/redhat-developer/codeready-workspaces/blob/master/dependencies/che-theia/dockerfiles/theia/Dockerfile#L127
and
https://github.com/redhat-developer/codeready-workspaces/blob/master/dependencies/che-theia/dockerfiles/theia-dev/Dockerfile#L71

benoitf · 2019-09-30T15:14:29Z

@nickboldt

do we need to build generator/eclipse-che-theia-generator.tgz ?

yes it's part of the source code of this project.

should we also be building @theia/generator-plugin instead of installing it from npm?

no, because the source code is external to this repository: https://github.com/eclipse/theia-generator-plugin

nickboldt · 2019-09-30T15:40:31Z

If "it's in another repo" was a reason not to build something, then the whole che-theia repo wouldn't exist. :P

I'm asking if it's better to use a specific binary, or to build from master. Since the version of the generator plugin is different in theia-dev vs. theia, I'd think at least we should standardize on a single version, if not using latest.

Would you agree? or is there some technical reason to have them different?

benoitf · 2019-09-30T15:43:14Z

it should be the same version coming from npmjs for theia-generator-plugin in theia-dev and in theia

nickboldt · 2019-09-30T19:04:40Z

@benoitf so you mean:

Instead of using
@theia/generator-plugin@0.0.1-1562578105
and
@theia/generator-plugin@0.0.1-1540209403

we should just pull the latest from npm ? or both should be using newer 1562578105 timestamp?

benoitf · 2019-09-30T19:13:39Z

1562578105 timestamp

benoitf · 2019-09-30T19:15:12Z

latest may pull unwanted versions and is not tagged/ reproducible build

nickboldt · 2019-10-04T04:22:16Z

I've got a working build of theia-dev in Brew, pushed to Quay.

Sync/trigger job: https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/crw-theia-dev_sync-github-to-pkgs.devel/configure
Latest tags: https://quay.io/repository/crw/theia-dev-rhel8?tab=tags

nickboldt · 2019-10-06T04:11:54Z

Sync/trigger job for theia: https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/crw-theia_sync-github-to-pkgs.devel/ is now working.

Latest tags (built locally, pushed manually): https://quay.io/repository/crw/theia-rhel8?tag=latest&tab=tags

However the build is still dying in Brew because node-gyp is trying to download stuff:

2019-10-06 03:54:41,669 - atomic_reactor.plugins.imagebuilder - INFO - --> RUN if [ -z $GITHUB_TOKEN ]; then unset GITHUB_TOKEN; fi &&     yarn
2019-10-06 03:54:42,190 - atomic_reactor.plugins.imagebuilder - INFO - yarn install v1.17.3
2019-10-06 03:54:42,279 - atomic_reactor.plugins.imagebuilder - INFO - $ node-gyp install
2019-10-06 03:54:42,543 - atomic_reactor.plugins.imagebuilder - INFO - gyp http GET https://nodejs.org/download/release/v10.14.1/node-v10.14.1-headers.tar.gz
2019-10-06 03:54:42,591 - atomic_reactor.plugins.imagebuilder - INFO - gyp WARN install got an error, rolling back install
2019-10-06 03:54:42,596 - atomic_reactor.plugins.imagebuilder - INFO - gyp ERR! stack Error: connect ECONNREFUSED 104.20.22.46:443
2019-10-06 03:54:42,596 - atomic_reactor.plugins.imagebuilder - INFO - gyp ERR! stack     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1113:14)
2019-10-06 03:54:42,596 - atomic_reactor.plugins.imagebuilder - INFO - gyp ERR! System Linux 3.10.0-957.21.2.el7.x86_64
2019-10-06 03:54:42,596 - atomic_reactor.plugins.imagebuilder - INFO - gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "install"
2019-10-06 03:54:42,596 - atomic_reactor.plugins.imagebuilder - INFO - gyp ERR! cwd /home/theia-dev/theia-source-code
2019-10-06 03:54:42,597 - atomic_reactor.plugins.imagebuilder - INFO - gyp ERR! node -v v10.14.1
2019-10-06 03:54:42,597 - atomic_reactor.plugins.imagebuilder - INFO - gyp ERR! node-gyp -v v3.8.0
2019-10-06 03:54:42,597 - atomic_reactor.plugins.imagebuilder - INFO - gyp ERR! not ok
2019-10-06 03:54:42,614 - atomic_reactor.plugins.imagebuilder - INFO - error Command failed with exit code 1.

-- http://download.eng.bos.redhat.com/brewroot/work/tasks/3375/23873375/x86_64.log

So probably have to prefetch https://nodejs.org/download/release/v10.14.1/node-v10.14.1-headers.tar.gz and use

npm config set nodedir /abs/path/to/node/sources
  or
npm config set tarball /abs/path/to/headers.tgz

Unless there's a better way, @benoitf ?

nickboldt · 2019-10-06T16:57:56Z

OK, got past the node-gyp download problem.

Next issue:

Couldn't find package "axios@0.18.1" required by "@eclipse-che/theia-plugin-ext@7.2.0" on the "npm" registry.
Couldn't find package "clean-webpack-plugin@^0.1.19" required by "@eclipse-che/theia-plugin-ext@7.2.0" on the "npm" registry.
Couldn't find package "@eclipse-che/api@^7.0.0-beta-4.0-80162ea6d2" required by "@eclipse-che/theia-preferences-provider-extension@7.2.0" on the "npm" registry.
Couldn't find package "@eclipse-che/api@latest" required by "@eclipse-che/che-theia-hosted-plugin-manager-extension@7.2.0" on the "npm" registry.
Couldn't find package "@eclipse-che/api@latest" required by "@eclipse-che/plugin@7.2.0" on the "npm" registry.
Couldn't find package "@eclipse-che/api@latest" required by "@eclipse-che/theia-dashboard-extension@7.2.0" on the "npm" registry.
Couldn't find package "@eclipse-che/api@latest" required by "@eclipse-che/theia-terminal@7.2.0" on the "npm" registry.
Couldn't find package "@eclipse-che/workspace-client@latest" required by "@eclipse-che/theia-plugin-ext@7.2.0" on the "npm" registry.
Couldn't find package "@eclipse-che/workspace-client@latest" required by "@eclipse-che/theia-terminal@7.2.0" on the "npm" registry.
Couldn't find package "html-webpack-plugin@^3.2.0" required by "@eclipse-che/theia-assembly@0.11.0" on the "npm" registry.
Couldn't find package "ini@^1.3.5" required by "@eclipse-che/theia-git-provisioner@7.2.0" on the "npm" registry.
Couldn't find package "js-yaml@3.13.1" required by "@eclipse-che/theia-plugin-ext@7.2.0" on the "npm" registry.
Couldn't find package "ts-loader@^4.1.0" required by "@eclipse-che/theia-plugin-ext@7.2.0" on the "npm" registry.
Couldn't find package "typescript-formatter@7.2.2" required by "@eclipse-che/theia-plugin-ext@7.2.0" on the "npm" registry.
Couldn't find package "typescript-formatter@7.2.2" required by "@eclipse-che/theia-remote@7.2.0" on the "npm" registry.
Couldn't find package "typescript-formatter@7.2.2" required by "@eclipse-che/theia-terminal@7.2.0" on the "npm" registry.
Couldn't find package "@types/js-yaml@3.11.2" required by "@eclipse-che/theia-plugin-ext@7.2.0" on the "npm" registry.
Couldn't find package "webpack@^4.20.2" required by "@eclipse-che/theia-plugin-ext@7.2.0" on the "npm" registry.
Couldn't find package "webpack-cli@^3.1.1" required by "@eclipse-che/theia-plugin-ext@7.2.0" on the "npm" registry.
Couldn't find package "xterm@3.9.2" required by "@eclipse-che/theia-terminal@7.2.0" on the "npm" registry.

Can we predeclare these dependencies in a yarn.lock or package.json and use that when setting up theia-dev so these will already be in the dev env? Or add them to the theia package.json and prefetch them before building theia?

For easier pasting, here's the uniq'd sort'd list:

 "axios@0.18.1" 
 "clean-webpack-plugin@^0.1.19" 
 "@eclipse-che/api@^7.0.0-beta-4.0-80162ea6d2" 
 "@eclipse-che/api@latest" 
 "@eclipse-che/workspace-client@latest" 
 "html-webpack-plugin@^3.2.0" 
 "ini@^1.3.5" 
 "js-yaml@3.13.1" 
 "ts-loader@^4.1.0" 
 "typescript-formatter@7.2.2" 
 "@types/js-yaml@3.11.2" 
 "webpack@^4.20.2" 
 "webpack-cli@^3.1.1" 
 "xterm@3.9.2"

nickboldt · 2019-10-06T17:01:37Z

Also can we use a version other than @latest ? That's great for CI but terrible for products. :) If so, how do we override that, and where?

benoitf · 2019-10-06T17:01:55Z

I think it's solved with yarn offline / cache

nickboldt · 2019-10-06T17:03:56Z

when/where do I have to run yarn to perform the caching? and from what folder do I then need to tar up the results? I know it's the weekend ... but your answers are not helpful :(

benoitf · 2019-10-06T19:38:46Z

the yarn cache folder: https://github.com/benoitf/che-theia-brew/blob/master/build.sh#L70
(or yarn cache dir command)

for yarn offline:
https://github.com/benoitf/che-theia-brew/blob/master/conf/theia-dev/ubi8-brew/setup-extra.env

it will append --offline on every yarn commands

benoitf · 2019-10-08T15:23:03Z

I've reworked che-theia docker image builds:
eclipse-che/che-theia#474

Found this issue with offline mode:
#14804

Here is a script to generate 'offline/ubi8 / brew compliant (hopefully) images'
https://github.com/benoitf/che-theia-brew

The build script is mostly composed of 3 parts:

we clone che-theia repository (for now it’s my branch) and run yarn
then, theia-dev, we generate a dry-run and use TMP_BUILDER_NAME to build an image “online” with ubi8. Once it is built, we extract cache/downloaded stuff and create assets. Then we create “real image” for ubi8 (ubi8-brew) with the name CHE_THEIA_DEV_IMAGE_NAME=“quay.io/codeready-workspaces/che-theia-dev:next” (configured at the top of the file)
then theia: again, dry run. then two builds: one for the builder target, one for the runtime target. Extract assets from builder and runtime, generate offline/ubi8-brew image and build it with CHE_THEIA_IMAGE_NAME=“quay.io/codeready-workspaces/che-theia:next”

nickboldt · 2019-10-09T19:23:36Z

Thanks to Florent's hard work we have theia, theia-dev, theia-endpoint images now for CRW 2.0:

quay.io/crw/theia-rhel8:2.0-3 286.7 MB
quay.io/crw/theia-dev-rhel8:2.0-46 288.7M
quay.io/crw/theia-endpoint-rhel8:2.0-1 275.4 MB

l0rd · 2019-10-10T21:49:22Z

@nickboldt @benoitf is there still something missing or we can close this issue?

nickboldt · 2019-10-11T03:22:17Z

Once Florent's code is in https://github.com/redhat-developer/codeready-workspaces-theia (including whatever magic is used to produce the Dockerfile and asset*.gz files for the theia-endpoint), we can resolve this. Currently the code in https://github.com/benoitf/che-theia-brew/tree/master/conf only covers Theia and Theia dev. Need all three.

Followup work in #14797

nickboldt · 2019-10-16T19:08:19Z

Code is live in https://github.com/redhat-developer/codeready-workspaces-theia but not synced with pkgs.devel repos yet

nickboldt · 2019-10-30T22:20:51Z

https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/CRW_CI/view/Pipelines/job/crw-theia_master/ builds theia-dev, theia, and the endpoint runtime binary, then triggers three brew builds and pushed them to brew. Resolving.

mmorhun self-assigned this Jun 10, 2019

benoitf transferred this issue from eclipse-che/che-theia Jul 5, 2019

benoitf added area/editor/theia Issues related to the che-theia IDE of Che team/ide2 labels Jul 5, 2019

mmorhun mentioned this issue Aug 16, 2019

Add dockerfile for Che Theia runtime image based on UBI 8 eclipse-che/che-theia#279

Closed

l0rd added this to the 7.3.0 milestone Sep 26, 2019

l0rd added team/productization severity/P1 Has a major impact to usage or development of the system. labels Sep 26, 2019

nickboldt mentioned this issue Sep 28, 2019

Cannot build che-theia generator from master branch #14706

Closed

3 tasks

nickboldt mentioned this issue Sep 30, 2019

use the same version of... eclipse-che/che-theia#458

Merged

benoitf mentioned this issue Oct 4, 2019

feat(docker): Provides ubi8 based images for che-theia and che-theia-dev eclipse-che/che-theia#467

Closed

This was referenced Oct 6, 2019

add --image-name: flag so that full image... eclipse-che/che-theia#470

Closed

allow slashes in ORGANIZATION field by using... eclipse-che/che-theia#471

Merged

benoitf mentioned this issue Oct 8, 2019

feat(docker): Provides ubi8 based images for che-theia and che-theia-dev eclipse-che/che-theia#474

Merged

benoitf mentioned this issue Oct 11, 2019

Provides new ubi8 images and fixes on docker images eclipse-che/che-theia#481

Merged

nickboldt closed this as completed Oct 30, 2019

nickboldt modified the milestones: 7.3.0, 7.3.1 Oct 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create UBI based images for che-theia #13765

Create UBI based images for che-theia #13765

evidolob commented May 15, 2019

mmorhun commented Jun 14, 2019 •

edited

Loading

nickboldt commented Sep 28, 2019 •

edited

Loading

benoitf commented Sep 30, 2019

benoitf commented Sep 30, 2019 •

edited

Loading

AndrienkoAleksandr commented Sep 30, 2019 •

edited

Loading

AndrienkoAleksandr commented Sep 30, 2019

nickboldt commented Sep 30, 2019

nickboldt commented Sep 30, 2019

nickboldt commented Sep 30, 2019

nickboldt commented Sep 30, 2019 •

edited

Loading

benoitf commented Sep 30, 2019

nickboldt commented Sep 30, 2019

benoitf commented Sep 30, 2019

nickboldt commented Sep 30, 2019

benoitf commented Sep 30, 2019

benoitf commented Sep 30, 2019

nickboldt commented Oct 4, 2019

nickboldt commented Oct 6, 2019 •

edited

Loading

nickboldt commented Oct 6, 2019 •

edited

Loading

nickboldt commented Oct 6, 2019

benoitf commented Oct 6, 2019

nickboldt commented Oct 6, 2019

benoitf commented Oct 6, 2019

benoitf commented Oct 8, 2019

nickboldt commented Oct 9, 2019 •

edited

Loading

l0rd commented Oct 10, 2019

nickboldt commented Oct 11, 2019 •

edited

Loading

nickboldt commented Oct 16, 2019

nickboldt commented Oct 30, 2019

Create UBI based images for che-theia #13765

Create UBI based images for che-theia #13765

Comments

evidolob commented May 15, 2019

mmorhun commented Jun 14, 2019 • edited Loading

nickboldt commented Sep 28, 2019 • edited Loading

Theia-dev

Theia

benoitf commented Sep 30, 2019

benoitf commented Sep 30, 2019 • edited Loading

AndrienkoAleksandr commented Sep 30, 2019 • edited Loading

AndrienkoAleksandr commented Sep 30, 2019

nickboldt commented Sep 30, 2019

nickboldt commented Sep 30, 2019

nickboldt commented Sep 30, 2019

nickboldt commented Sep 30, 2019 • edited Loading

benoitf commented Sep 30, 2019

nickboldt commented Sep 30, 2019

benoitf commented Sep 30, 2019

nickboldt commented Sep 30, 2019

benoitf commented Sep 30, 2019

benoitf commented Sep 30, 2019

nickboldt commented Oct 4, 2019

nickboldt commented Oct 6, 2019 • edited Loading

nickboldt commented Oct 6, 2019 • edited Loading

nickboldt commented Oct 6, 2019

benoitf commented Oct 6, 2019

nickboldt commented Oct 6, 2019

benoitf commented Oct 6, 2019

benoitf commented Oct 8, 2019

nickboldt commented Oct 9, 2019 • edited Loading

l0rd commented Oct 10, 2019

nickboldt commented Oct 11, 2019 • edited Loading

nickboldt commented Oct 16, 2019

nickboldt commented Oct 30, 2019

mmorhun commented Jun 14, 2019 •

edited

Loading

nickboldt commented Sep 28, 2019 •

edited

Loading

benoitf commented Sep 30, 2019 •

edited

Loading

AndrienkoAleksandr commented Sep 30, 2019 •

edited

Loading

nickboldt commented Sep 30, 2019 •

edited

Loading

nickboldt commented Oct 6, 2019 •

edited

Loading

nickboldt commented Oct 6, 2019 •

edited

Loading

nickboldt commented Oct 9, 2019 •

edited

Loading

nickboldt commented Oct 11, 2019 •

edited

Loading