create live demo of web-ui

[markus2330]

The web-ui is now in a quite stable state (only little bugs are open, see #1892) so we could now start to create a live demo.

The deployment script should be very similar to what we already have for our homepage, it is basically:

• compile with -DTOOLS=..;web
• make install
• kdb run-elektrad &
• kdb run-web &

We obviously need a completely separated container for that, the web-ui basically allows remote execution of kdb commands with the user elektrad is running.

Such a live demo might be one of the best advertisement iff it works well. What do you think?

[markus2330]

The only realistic plan for this release is to build and host a docker image with a manual build of web-ui.

@omnidan Can you do that?

[markus2330]

apiary of the elektrad+web would be also part of this.

[markus2330]

@omnidan did you have time to work on this or the video?

[omnidan]

I already made a video: https://drive.google.com/open?id=13dSC0ereCQGusrwlIodaDostW6eEes72

I will try building a docker image with the web-ui installed. Is there already a docker image that has libelektra installed? Where would we host the docker image?

[markus2330]

Thank you, the video is great! If you happen to do another video sometimes, having a video without students as example would be excellent 😄 (e.g. editing /etc/hosts)

What is our best option to host the video, so that playing the video works smoothly across browsers? (In the link above two browsers were not able to play it directly.)

I have a nextcloud installation, we could try it there.

Or is it only a matter of the video format it is in?

Is there already a docker image that has libelektra installed

See scripts/docker for building images.

Where would we host the docker image?

We have our own docker registry at https://hub.libelektra.org but as far as I know it is not yet publicly available. @ingwinlu knows more about that.

[omnidan]

@markus2330 I uploaded it to youtube (unlisted), that should work in all browsers: https://youtu.be/lLg9sk6Hx-E

[markus2330]

Why unlisted?

[omnidan]

@markus2330 I can make it public, I just thought you only wanted to use it for embedding on the website.

[markus2330]

Public is fine for me. We can embed it into the next release notes or even post it as comment to the current reactions of the release notes?

[markus2330]

What is the standard youtube license?

[omnidan]

I don't know, I changed it to CC BY

[markus2330]

I think a docker image with a full (and latest) Elektra preinstalled could be very useful for many (demo) use cases (not only the Web UI).

So I am quite keen to have such a docker image 😉

[ingwinlu]

@omnidan you can use a stretch base image and install the packages from our repo. That should be fairly fast and clean.

However you will not have access to any features that do not exist in debian packages yet.

[markus2330]

Thanks for the input.

Unfortunately, the Debian packages do not include the Web UI yet. And it would be a rather difficult thing to do (any nodejs packaging experts around here?).

And having parts of Elektra installed as packages, and other parts by "make install" does not really sound clean to me (the versions can easily mismatch).

[markus2330]

@ingwinlu would it possible for you to to host a Docker image at a7 which was created by @omnidan? Or do you consider to write a Dockerfile for the Web-UI yourself?

Having a dockerfile+image (maybe in a public Docker registry if we want to keep our registry private?) with the latest Elektra installed would be really great.

[ingwinlu]

a7's repository is not configured for public access. And if it is not needed by the build system it makes more sense to put it in the public docker registry.

[markus2330]

So the plan is that devs rebuild their own docker images? Or should we put all images in a public repo?

Is there a potential harm if we allow read-only access?

[ingwinlu]

So the plan is that devs rebuild their own docker images?

That is an option, or to manually push the image from one of the build slaves to the public repository. Or to reconfigure the reverse-proxy in front of the registry to allow anonymous get requests through to the endpoint.

Or should we put all images in a public repo?

That will slow down the build system on image updates a lot.

Is there a potential harm if we allow read-only access?

Right now no as there should be no sensitive credentials in the images. However miss configuration could mean any keys / access needed for the build system might be leaked into the images.

[markus2330]

@ingwinlu Yes, a job that uploads to a public registry sounds like a good option. Or can we have two registries?

@omnidan Is it okay for you to write a Dockerfile for the Web UI?

[markus2330]

@ingwinlu Thank you so much!
webui.libelektra.org webdemo.libelektra.org should point to a7 shortly.

@omnidan The webui might be more interesting if you create some mountpoints in the images. kdb mount-info and kdb mount --with-recommends /etc/hosts system/hosts hosts would be a good start!

[omnidan]

@ingwinlu you can already use :latest, it uses :1.4 right now (will publish :1.5 on latest when the PR gets merged)

[markus2330]

I would prefer if we immediately can have 1.5, there is little use of testing 1.4 now. I thought merging is not necessary?

[ingwinlu]

@omnidan you seem to be misunderstanding how the latest tag works: https://medium.com/@mccode/the-misunderstood-docker-tag-latest-af3babfd6375

https://hub.docker.com/r/elektra/web/tags/ does not show latest. You need to explicitly push it.

[markus2330]

Thank you for pointing this out!

Can you start the docker image 1.5.0-beta on a7 so that we have something to try out tomorrow in the meeting?

[ingwinlu]

http://a7.complang.tuwien.ac.at:33334/

I could not reverse proxy it as the ports are hardcoded (and that will not work with the current setup). Also note that elektra is running as root user in the container which makes me uncomfortable as hell.

[markus2330]

http://webui.libelektra.org:33334/ also works! Thank you 💯 times.

Also note that elektra is running as root user in the container which makes me uncomfortable as hell.

Do not worry, it is not root (but 999) and it is not even possible to write to system keys: could not create configuration directory: Could not create directory "/etc/kdb", because: "Permission denied" uid: 999, euid: 999, gid: 999, egid: 999

[ingwinlu]

Yeah but it is not proxied so also no https. It is just rigged to run for now but would not keep it that way in the long run. markus2330 <notifications@github.com> schrieb am Fr., 1. Juni 2018, 10:38:

…

http://webui.libelektra.org:33334/ also works! Thank you 💯 times. Also note that elektra is running as root user in the container which makes me uncomfortable as hell. Do not worry, it is not root (but 999) and it is not even possible to write to system keys: could not create configuration directory: Could not create directory "/etc/kdb", because: "Permission denied" uid: 999, euid: 999, gid: 999, egid: 999 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1901 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEOv-lxXTk85wCl8ipo7okci3PKztfWhks5t4P1jgaJpZM4TRVIf> .

[omnidan]

the ports are hardcoded

The problem here is that we run elektrad and webd in one container, which is against docker principles. So I can only make one of the two ports adjustable (which should be enough though, as long as we are fine with elektrad being on :33333)

Also note that elektra is running as root user in the container which makes me uncomfortable as hell.

I specifically made sure to create and use an elektra user in the Dockerfile, unless I did something wrong?

[ingwinlu]

Yeah sorry for the rushed message (I am completely full with appointments this weekend + my last pr broke master builds).

There are two big problems with the current Dockerfile:

multiple processes:
while you already said it is against 'principles' to do so it is mainly said to be a principle because if you break the rule you need to understand what is happening and why it is a problem.
You create a shell script that starts two processes. This shell script is run via PID 1. This is kind of special for Docker as it is used as the 'init' system you know from traditional OSs. Hence this is the pid that needs to control / restart / log all other processes that you start.
For single process containers this is trivial as the started process usally does that anyway for itself. For multiple process you need a wrapper.
There is some documentation available https://docs.docker.com/config/containers/multi-service_container/
http://supervisord.org/ is very popular to solve the issue as well.
Or you take a look at how the current homepage containers work I wrote the last month to split it up into single service containes.

ports:
I tried proxying 33334 and it had trouble connecting through to the backend even when it was exposed.
While I did not fully troubleshoot the problem I suspect that is because the location for the backend is hardcoded to something like localhost:33334 which breaks in that case as the container is not actually exposing it on localhost but on its randomly generated hostname.
A long term solution would probably be to either set the backend to the containers hostname in your start script or again split up the containes and just point the frontend to the backend.
Again you might find inspiration for the second case in our existing homepage containers (which are not perfect but they solve that problem).

I have more time next week and can help you rewrite the files if you get stuck.

[omnidan]

For single process containers this is trivial as the started process usally does that anyway for itself.

@ingwinlu unfortunately, it does not seem like Node.js does this :/ I think this could be solved by passing the --init flag when running containers, though. (https://www.elastic.io/nodejs-as-pid-1-under-docker-images/)

I suspect that is because the location for the backend is hardcoded to something like localhost:33334 which breaks in that case as the container is not actually exposing it on localhost but on its randomly generated hostname.

The location is actually not hardcoded. The client is always served on the same host/port as the backend (webd), and it simply accesses /api/ on the same host.

I think the best way to go about this is separating the elektrad and webd images, then setting the PORT env var will also work properly.

[ingwinlu]

node.js on pid 1

I would not trust --init. But a wrapper process like they introduced in the article sounds fine.

The location is actually not hardcoded. The client is always served on the same host/port as the backend (webd), and it simply accesses /api/ on the same host.

Again I did not have time to debug it. I setup 33334 to be reverse proxyied on one of the addresses we pointed to a7. It resulted in failures to load the website because it could not access an address it needed to load. This was also the case for exposing 33333:33333 directly with 33334 beeing reverse proxied on 443 (https).

[markus2330]

Ok, so the new plan is:

1. @ingwinlu creates a minimal Elektra docker file and a build job that automatically builds the image and publishes it to docker.com (@omnidan can you give us access to the docker.com registry for elektra? Do you need something else other than account names?)
2. Based on that image, @omnidan (with reviews by @ingwinlu) creates three more Docker files/images that are built by the same build job (or extensions of it):
   1. extended image with full Elektra (to make Web UI more interesting),
   2. extended image with elektrad (for live demo, part 1)
   3. extended image with client/web (for live demo, part 2)

As end product we have a build job that builds four Elektra images for every commit in master:

1. a minimal one for minimalists (or with little bandwidth)
2. a full Elektra test image (for people trying out Elektra)
3. an image that starts up elektrad (for our live demo and people trying out Web UI)
4. an image that starts up client/web (for our live demo and people trying out Web UI)

Is that okay for you two?

[omnidan]

Do you need something else other than account names?

I need your Docker Hub usernames, that's all.

Based on that image

Is it ready yet?

[markus2330]

Thank you, my Docker Hub Username (Docker ID) is markus2330

https://hub.docker.com/u/elektra/ says that elektra/web already had 10k+ pulls. Is it so popular or is our setup doing something wrongly?

[markus2330]

@ingwinlu Can we retrigger the build server so that "webui.libelektra.org" is updated to 1.5? Or can you please implement such a job?

[ingwinlu]

the buildserver has absolutely nothing to do with webui.libelektra.org.

it was set up to always pull the latest 1.5.0-beta tag. I modified it so it uses :latest now.

[markus2330]

See above, I proposed a plan that docker images are built and published, so they should be related.

Do you mean that the build server does not deploy webui? Is there any difference in deploying webui and the homepage? And if so, why?

it was set up to always pull the latest 1.5.0-beta tag. I modified it so it uses :latest now.

Thank you! Is it automatically updated? When?

[ingwinlu]

See above, I proposed a plan that docker images are built and published, so they should be related.

Do you mean that the build server does not deploy webui? Is there any difference in deploying webui and the homepage?

Because nobody has done it. I only quickly set it up as seen in the comments above.

And if so, why?

Because nobody was interested in doing so, at least I am not aware of it. There are also several questions to answer first:

• Where we should get the tags for the images from?
• How to set up the upload to the public registry?
• How to setup access rights to the public registry?
• When should be builds be run?
• ...

I currently have no interest in implementing it as I have more urgent matters to attend to.

[markus2330]

And if so, why?

The question referred to why there are differences between webui and homepage deployment. We could push the web ui images to our private registry, which should resolve the questions?

Pushing to docker.com can be done manually for now (for the releases). But to automate this would be highly appreciated.

[markus2330]

@omnidan Have you already added us to hub.docker.com? Username: markus2330

[ingwinlu]

the new split up docker images (the ones that only run 1 process) could be build, pushed and deployed identically to the homepage. probably.

[omnidan]

@markus2330 I added you as owner, you should be able to add new members on https://hub.docker.com/u/elektra/dashboard/teams/?team=owners now

[markus2330]

I would say we keep webui.libelektra.org to have the public Docker Image.

And webdemo.libelektra.org gets rebuild on every commit in master.

@ingwinlu What do you think?

[omnidan]

I changed it to webdemo.libelektra.org in the #2110 PR

[markus2330]

I think we can close this now?

@omnidan Can you please also update the front page? The "Upcoming:" is now longer true 😄 (Screenshot can be improved, too.)

Great work by you two, thank you!

[omnidan]

@markus2330 yes, we can close this. I will send you a quick PR for the homepage tomorrow!

Thank you both too, especially @ingwinlu for explaining docker and jenkins to me and correcting my dockerfiles (I learned a lot about docker from you!)