Elektra Web jenkins deployment #2110

omnidan · 2018-06-25T11:20:32Z

I copied the definitions from the homepage tasks, so it should work. Is there any way I can test the Jenkinsfile locally?

manually building docker images

make sure to change 1.6 to a new version if it was updated. base image (bold) has to be built first, others can be ran in parallel.

in scripts/docker/webui/base run: docker build --no-cache -t elektra/web-base:latest -t elektra/web-base:1.6 .
in scripts/docker/webui/elektrad run: docker build -t elektra/elektrad:latest -t elektra/elektrad:1.6 .
in scripts/docker/webui/webd run: docker build -t elektra/webd:latest -t elektra/webd:1.6 .
in scripts/docker/webui/elektrad-demo run: docker build -t elektra/elektrad-demo:latest -t elektra/elektrad-demo:1.6 .
in scripts/docker/webui/webd-demo run: docker build -t elektra/webd-demo:latest -t elektra/webd-demo:1.6 .
in scripts/docker/webui/web run: docker build -t elektra/web:latest -t elektra/web:1.6 .

to publish these images to docker hub (the main docker registry), simply run:

docker push elektra/web-base
docker push elektra/elektrad
docker push elektra/webd
docker push elektra/elektrad-demo
docker push elektra/webd-demo
docker push elektra/web

ingwinlu

Is there any way I can test the Jenkinsfile locally?

unfortunately only if you set up a jenkins installation.

Please remove any duplicate code that is introduced with the current commits. there is no need for functions that do exactly the same that just have a different name.

omnidan · 2018-06-25T11:48:25Z

@ingwinlu I re-used the idHomepage function now

ingwinlu · 2018-06-25T11:50:29Z

scripts/jenkins/Jenkinsfile

@@ -902,6 +923,49 @@ def buildHomepage() {
  return homepageTasks
 }

+def deployWebUI() {
+  node("elektrad") {


did you add this tag? does whatever this tag describe provide the means to execute the code below?

@ingwinlu how do I add tags? or do you mean a git tag?

ingwinlu · 2018-06-25T11:51:04Z

scripts/jenkins/Jenkinsfile

+      backend.pull()
+      frontend.pull()
+
+      sh "docker stop -t 5 ${backendName} || /bin/true"


this functionality could be moved into a function as it is now used 4 times in a row

ingwinlu · 2018-06-25T11:52:05Z

scripts/jenkins/Jenkinsfile

+  node("elektrad") {
+    docker.withRegistry("https://${REGISTRY}",
+                        'docker-hub-elektra-jenkins') {
+      def backendName = "elektrad"


keep in mind those names are deployed without elektra context. if i am a system administrator and i see a webd container i no nothing about it.

but only on our local registry, right? then the elektra context is there, because it is the elektra docker registry.

ingwinlu · 2018-06-25T11:52:47Z

scripts/jenkins/Jenkinsfile

@@ -902,6 +923,49 @@ def buildHomepage() {
  return homepageTasks
 }

+def deployWebUI() {


i had hoped we could reuse some of the code in deployHomepage instead of just having a 1:1 copy with some names exchanged.

I will try to refactor it a bit

markus2330

Please improve and set "ready to merge" only if the web ui is actually reachable and works.

markus2330 · 2018-06-25T12:02:01Z

doc/news/_preparation_next_release.md

@@ -110,6 +110,7 @@ Try it out now on: http://webui.libelektra.org:33334/
 - added "create array" button to easily create arrays
 - removed confirmation dialog before deletion (undo can be used instead)
 - created a docker image: `elektra/web`
+- created a task to deploy the demo


needs (name)

markus2330 · 2018-06-25T12:02:47Z

scripts/jenkins/Jenkinsfile

@@ -152,6 +152,10 @@ maybeStage("Deploy Homepage", isMaster()) {
  deployHomepage()
 }

+maybeStage("Deploy Web UI", isMaster()) {


@ingwinlu Is it possible/useful to deploy Homepage/Web UI Demo in parallel?

not really as changes to the docker environment are serialised anyway. plus the maximum speed up that can be gained are around 5s.

@omnidan I mean this "isMaster" here.

markus2330 · 2018-06-25T12:03:45Z

scripts/jenkins/Jenkinsfile

+    DOCKER_IMAGES.webui_elektrad = createDockerImageDesc(
+      "elektra/elektrad-demo", this.&idHomepage,
+      ".",
+      "./scripts/docker/web/elektrad-demo/Dockerfile",


Please add some more mountpoints.

did we really add a web directory? super confusing as homepage is not part of web. can you rename to webui?

@markus2330 I added some new mountpoints:

RUN kdb mount /etc/networks system/networks hosts RUN kdb mount /etc/ssh/ssh_config system/ssh line RUN kdb mount /etc/ldap/ldap.conf system/ldap line RUN kdb mount /var/lib/dpkg/available system/dpkg/available dpkg RUN kdb mount /var/lib/dpkg/status system/dpkg/status dpkg

is this enough or do you have any ideas for other mountpoints?

@ingwinlu I renamed the folder to webui

ingwinlu · 2018-06-25T12:15:46Z

your approach also is failing to provide your base image. that has to be built as well. your contexts are also wrong.

your base image should not git clone but use the existing workspace.

omnidan · 2018-06-25T12:23:34Z

@ingwinlu what do you mean by "your contexts are wrong"? how can I use the existing workspace in the Dockerfile?

ingwinlu · 2018-06-25T12:30:46Z

https://github.com/ElektraInitiative/libelektra/blob/master/scripts/docker/homepage/backend/Dockerfile#L71

This line copies the already checked out elektra into the docker build context.
You should do something similar with your base image (and not require a fresh checkout if you are already assuming you are in the repo).

Afterwards your other images do not the whole directory as a docker build context.

omnidan · 2018-06-25T12:42:17Z

@ingwinlu I fixed it (I think). How do I get jenkins to build the webui?

markus2330 · 2018-06-25T13:05:18Z

It gets build automatically. But it seems there are errors because of the release notes: https://build.libelektra.org/jenkins/blue/organizations/jenkins/libelektra/detail/PR-2110/8/pipeline

omnidan · 2018-06-25T14:24:52Z

@markus2330 @ingwinlu jenkins says:

Stage skipped: Deploy Web UI

How can I get it to not skip this step?

markus2330 · 2018-06-25T14:34:16Z

Maybe you cannot trigger it from the PR because of the isMaster(). You need to pass something that always returns true instead.

@ingwinlu What is the best way to do that?

omnidan · 2018-06-25T14:44:50Z

@markus2330 I will temporarily turn the maybeStage into a stage so that we can see if it works.

markus2330 · 2018-06-27T09:32:15Z

I assume it is for the internal communication: Is it really necessary to have this DNS entry?

Nevertheless, elektrad-demo now points to a7.

ingwinlu · 2018-06-27T09:54:27Z

7962229 does not what you think it does. it now pulls from hub.docker instead of using the image you build earlier in the pipeline...

docker assumes hub.docker if you omit adding a registry to an image tag.

…/web-base" This reverts commit 7962229.

omnidan · 2018-06-27T10:02:04Z

@ingwinlu thanks for the hint! I reverted the commit.

@markus2330 thanks for adding the DNS. yes, it is just for internal communication.

Edit: I noticed it still does not seem to resolve the DNS properly, but maybe it will take a while until a7 refreshes its dns cache. https://webdemo.libelektra.org/instances/56755a3c-27f5-45f1-92bd-8140fe1780f0 should work when it's refreshed.

omnidan · 2018-06-27T10:05:56Z

I also re-enabled full builds and changed the trues back to isMaster() (including the one for the docker image you told me about @ingwinlu). Let me know if everything looks fine now.

ingwinlu · 2018-06-27T10:07:48Z

scripts/docker/webui/elektrad-demo/Dockerfile

+RUN kdb mount /etc/ssh/ssh_config system/ssh line
+RUN kdb mount /etc/ldap/ldap.conf system/ldap line
+RUN kdb mount /var/lib/dpkg/available system/dpkg/available dpkg
+RUN kdb mount /var/lib/dpkg/status system/dpkg/status dpkg



you really want to join those statements together to speed up builds and reduce image size

this is true for all your images

ingwinlu · 2018-06-27T10:09:25Z

scripts/jenkins/Jenkinsfile

@@ -859,22 +889,20 @@ def buildPackageDebianStretch() {
  }]
 }

-def deployHomepage() {


why the strange notation of having to deploy 2 images?

just split it up to 1 image

this could probably be further improved:

Instead of passing the imageId I think it is easier to just pass the image description.

host should probably be renamed to hostname.

ingwinlu · 2018-06-27T10:09:49Z

scripts/jenkins/Jenkinsfile

+
+def buildWebUI() {
+  def webuiTasks = [:]
+  webuiTasks << buildImageStage(DOCKER_IMAGES.webui_base)


remove here you build it earlier via autobuild

omnidan · 2018-06-27T10:15:11Z

@ingwinlu should be fine now!

ingwinlu · 2018-06-27T10:19:42Z

@omnidan hard to say as you already removed the builds from the pr again (and deployment was never tested)

omnidan · 2018-06-27T10:21:50Z

@ingwinlu it already deployed earlier, but I can let it run again with the adjustments now.

… full build" This reverts commit b3a392e.

ingwinlu · 2018-06-27T12:54:48Z

lgtm when the isMaster stuff is back in. thanks for your work on this and for implementing my feedback.

markus2330 · 2018-06-27T13:55:42Z

doc/news/_preparation_next_release.md

+- added 404 page for invalid instance ids *(Daniel Bugl)*
+- implement drag & copy by holding the Ctrl or Alt key *(Daniel Bugl)*
+- add button to show error details *(Daniel Bugl)*
+- allow deleting all keys in a namespace *(Daniel Bugl)*


I think it would be enough to add your name once at the very end of the highlight. But this is okay, too.

Do not forget to add the instructions of manual building+uploading of the docker images. (inside some README of the Web UI)

@markus2330 oh okay, the error message told me to add (Name) after each line. I will add the information to the README of scripts/docker/webui/.

…e-enable full build"" This reverts commit e81ae25.

omnidan · 2018-06-27T15:40:39Z

@ingwinlu @markus2330 done!

markus2330 · 2018-06-27T15:44:31Z

Thank you, great job!

omnidan requested review from ingwinlu and markus2330 June 25, 2018 11:20

ingwinlu suggested changes Jun 25, 2018

View reviewed changes

omnidan added the ready to merge label Jun 25, 2018

ingwinlu suggested changes Jun 25, 2018

View reviewed changes

markus2330 removed the ready to merge label Jun 25, 2018

markus2330 suggested changes Jun 25, 2018

View reviewed changes

omnidan mentioned this pull request Jun 25, 2018

create live demo of web-ui #1901

Closed

omnidan added 9 commits June 25, 2018 14:40

web ui: implement deployment via Jenkinsfile

c597f9e

web ui: add info about jenkins deployment to release notes

73a0144

web ui jenkins: remove duplicate code

18062bd

web ui jenkins: refactor Jenkinsfile

9a20279

web ui jenkins: update release notes

f320ea4

web ui docker: add more mountpoints to demo

192f61c

web ui docker: rename folder to webui/

6fdfdb0

web ui docker: use existing build context

c34d04d

web ui jenkins: build base image

214dd7b

omnidan force-pushed the elektra-web-jenkins branch from 6d5837f to 214dd7b Compare June 25, 2018 12:40

web ui docker: move realworld.ini to new folder

c89442f

web ui: add name to release notes

6c70a45

web ui jenkins: temporarily deploy web UI for non-master branches

22d8c9b

web ui jenkins/docker: attempt renaming image back to elektra/web-base

7962229

Revert "web ui jenkins/docker: attempt renaming image back to elektra…

b780c29

…/web-base" This reverts commit 7962229.

web ui jenkins: change true back to isMaster(), re-enable full build

b3a392e

ingwinlu reviewed Jun 27, 2018

View reviewed changes

omnidan added 3 commits June 27, 2018 12:10

web ui docker: group statements together

e9eb1dc

web ui jenkins: refactor deploy function

1d9a038

web ui docker: group statements together

8e608f8

web ui jenkins: fix typo

04da458

omnidan added 2 commits June 27, 2018 12:22

Revert "web ui jenkins: change true back to isMaster(), re-enable…

e81ae25

… full build" This reverts commit b3a392e.

web ui jenkins: improve deployDockerContainer function

8b560b5

ingwinlu approved these changes Jun 27, 2018

View reviewed changes

markus2330 reviewed Jun 27, 2018

View reviewed changes

omnidan added 3 commits June 27, 2018 17:38

web ui docker: add guide to manually build docker images

4bd3ae1

web ui jenkins: adjust release notes

d858da8

Revert "Revert "web ui jenkins: change true back to isMaster(), r…

1484fd2

…e-enable full build"" This reverts commit e81ae25.

omnidan added the ready to merge label Jun 27, 2018

markus2330 approved these changes Jun 27, 2018

View reviewed changes

markus2330 merged commit e82b55f into ElektraInitiative:master Jun 27, 2018

Elektra Web jenkins deployment #2110

Elektra Web jenkins deployment #2110

Conversation

omnidan commented Jun 25, 2018 • edited Loading

manually building docker images

ingwinlu left a comment

Choose a reason for hiding this comment

omnidan commented Jun 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markus2330 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ingwinlu commented Jun 25, 2018

omnidan commented Jun 25, 2018

ingwinlu commented Jun 25, 2018

omnidan commented Jun 25, 2018

markus2330 commented Jun 25, 2018

omnidan commented Jun 25, 2018

markus2330 commented Jun 25, 2018

omnidan commented Jun 25, 2018

markus2330 commented Jun 27, 2018

ingwinlu commented Jun 27, 2018

omnidan commented Jun 27, 2018 • edited Loading

omnidan commented Jun 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omnidan commented Jun 27, 2018

ingwinlu commented Jun 27, 2018

omnidan commented Jun 27, 2018

ingwinlu commented Jun 27, 2018

markus2330 Jun 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omnidan commented Jun 27, 2018

markus2330 commented Jun 27, 2018

omnidan commented Jun 25, 2018 •

edited

Loading

omnidan commented Jun 27, 2018 •

edited

Loading

markus2330 Jun 27, 2018 •

edited

Loading