Need to pre-pull images required for 'java-maven' Che 7 devfile used on che.openshift.io and compare workspace startup time with non-prepulled devfiles

[ibuziuk]

Potential candidates from e2e happy path - eclipse-che/che#12728
I believe this list needs to be reconsidered and updated:

• eclipse/che-jwtproxy:latest
• quay.io/eclipse/che-java11-maven:nightly
• eclipse/che-machine-exec:7.0.0
• eclipse/che-theia:7.0.0
• eclipse/che-remote-plugin-runner-java8:7.0.0

[amisevsk]

If we mean pre-pull via k8s image puller, we'd need some work on that side. Currently there's a limit to how many images it can work on (5Mi per container, per node).

[ibuziuk]

@l0rd looks like we have no capacity for this task in the next sprint. Are you ok if we will work on it after GA release (according to the metrics workspace startup is relatively fast atm, so optimization is not crucial atm)

[skabashnyuk]

Currently there's a limit to how many images it can work on (5Mi per container, per node).

What does it mean 5Mi?

[l0rd]

@ibuziuk I am ok with postponing that

[amisevsk]

What does it mean 5Mi?

The minimum memory request/limit for kubernetes containers is 5 Mebibytes. Since the containers started by a daemonset belong to a single namespace, this can cause problems. E.g. 5Mi per container, times 5 containers in the daemonset, times 75 nodes in the cluster = 1875Mi quota used on that namespace. Each additional container we'd want to add would increase the request by 375Mi.

The solution to this would likely be grouping containers into daemonsets that will fit and removing them once they are ready. As long as the daemonsets are started frequently enough the images should always be cached [1].

The other thing that would need investigation is how many images can be stored in the cache; if caching the 21st image means dropping the 1st, then we would have to think about prioritization.

[1] - This is probably a good idea to implement regardless, since without manual intervention I don't think a daemonset will rollout new images when they are pushed.

[aditya-konarde]

I've added a parameter to the template so that we can control replicas of this pod in different environments

• Set replicas via template parameter kubernetes-image-puller#10
• Add new replicas parameter to k8s image puller openshiftio/saas-openshiftio#1317

[ibuziuk]

blocked by #1549

[amisevsk]

PR updating devfiles to be cached has been merged. The current list is

eclipse/che-jwtproxy:latest
quay.io/eclipse/che-java11-maven:nightly
eclipse/che-machine-exec:7.0.0
eclipse/che-theia:7.0.0
eclipse/che-remote-plugin-runner-java8:7.0.0

The image puller is currently only running on prod-preview and caching images in cluster 2a (and is still out of date since we need to rollout a new deployment). Since these images are used for Che 7.0.0 workspaces, we likely need to wait for 7.0.0 rollout to get meaningful data from production.

[ibuziuk]

@amisevsk since prod has been updated could we close this issue now (e.g. for analyzing the data between pre-pulled & non-pre-pulled devfiles)?

[amisevsk]

@ibuziuk The pre-pull part of the issue is completed but I left it open because part of the task is comparing pulled and non-pulled times. I think we could likely do this for prod-preview now, since 7.0.0 is available there.

If you think the comparison should be handled in a separate issue, feel free to close and open a followup.

[ibuziuk]

makes sense let's add it to the next sprint and do comparison once Hosted Che is updated to GA

[amisevsk]

The current plan for testing this is

1. Wait for 7.1.0 to roll out to production
2. Get timing data for starting a java-maven workspace
3. Update puller to cache images used in 7.1.0 workspaces (tagged 7.1.0)
4. Re-do timing tests with images cached

The only concern with this approach is if there are any shared layers between 7.0.0 and 7.1.0 images, we might show less of an effect than otherwise; there's no real way around this though, since we can't easily control the images used in workspaces (apart from the main container).

[amisevsk]

All done, email sent out to mailing lists.