New Dockerfile and update on Readme

[cyberkov]

Hi!

I started the Dockerfile based on the ubuntu image with java8 (as suggested in the forum).

The exposed volumes are /config and /userdata.

Please let me know if we can improve it anyhow.

Cheers
Hannes

[cyberkov]

It might make sense to have $DEBUG Environment variable to start with debug instead os server

[kaikreuzer]

Thanks for making a start here! I will leave it to the Docker experts to review and merge the PRs.
I suggest that on every PR at least some other person adds a "lgtm" before it gets merged (I will do the merging for now).

[cyberkov]

@christianwaite Would you mind taking a look at it? From what i saw in your repo it should be quite similar in behavior with yours, except that I did use ubuntu:trusty instead of phusion/baseimage. The reason for that is that it would improve portability to arm architecture since one would only need to split the architecture-specifics into it's own Dockerfile (i couldn't find baseimage for arm). On the other hand I have no idea if Java is doing proper reaping as PID 1 or if there are any zombies at all, which I guess has been the reason for pusion/baseimage at first :)

I have it running on my server with this docker-compose.yml since a few days and for me it seems to work properly.

I would really love to get this going as it would improve updating openhab drastically, since dockerhub's autobuilds are only triggered on pushes to the current repo (only updates on the Dockerfile in this case), while creating it as a downstream job of the openhab-distro would be almost instantaneous.

[kaikreuzer]

@christianwaite, @tdeckers, @WetwareLabs & @umiddelb, I just want to make you aware of this repo/PR. It would be wonderful if one of you could have a look and comment a "lgtm" here.
If you could even provide further PRs to the repo to get a good automatically-built docker container for x86 and armhf, it would be perfect - I count on you! Thanks in advance 🙇

[cyberkov]

Hello!

I might have found a solution to create automated arm builds with https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/
I'll try to incorporate this in the next days.

Cheers
Hannes

[umiddelb]

Hm, as looking to this PR from an ARM perspective:

wget -nv https://github.com/jwilder/dockerize/releases/download/v0.2.0/dockerize-linux-amd64-v0.2.0.tar.gz &&\

I need to check to which extend this will break/run on ARM as well. If there are platform specific dependencies inside, dockerize should be better established as a Docker image (like baseimage-docker).

Why are you installing Java 8 inside the Dockerfile instead of picking one of the existing Java 8 Docker images? This might add more flexibility, e.g. switching to OpenJDK and would follow the 'segregation of concerns' principle.

I might have found a solution to create automated arm builds with https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/ I'll try to incorporate this in the next days.

You might get in touch with @DieterReuter or @StefanScherer from the @hypriot team. They found a smart solution to build/run ARM Docker images with Travis CI automatically.

[cyberkov]

Hi @umiddelb !
Thanks a lot for your input! Right now I'm trying to update the PR according to your suggestions and try to implement a Proof-of-concept on Dockerhub.

• dockerize: Yes I noticed that as well. The arm build I referenced on the forum contains the arm version. in all cases I would suggest staying in control of the whole build chain as it would ease the switch between Java versions and architectures.
  Unfortunately it is not possible to "program" dockerfiles like jenkin's matrices, so I ended up creating 6 Dockerfiles (openhab-online, openhab-offline x armv7hf, x86)
• hypriot seems to base it's images on resin's work as well :) (i just checked a few of the repos e.g. https://github.com/hypriot/rpi-mysql/blob/master/Dockerfile ), in some cases they seem to use drone.io but I am not sure how this actually works.

Cheers
Hannes

[cyberkov]

Ok. I updated the PR now with all different versions. I only tested the openhab:offline for x86 since I do not have a docker installation on arm, maybe someone can test this please.

For the setup of automated builds:
I created two autobuild-repositories openhab-base and openhab2 which both point to the same github repo.
The build settings for an official repo would be (branch|file|tag):
openhab-base:

• master | /baseimages/x86 | latest
• master | /baseimages/arm7hf | arm7hf
  openhab2:
• master | /openhab-online/x86 | latest
• master | /openhab-offline/x86 | offline
• master | /openhab-online/arm7hf | online-arm7hf
• master | /openhab-offline/arm7hf | offline-arm7hf

In the cloudbees instance you could set up a curl trigger to promote a successful build to a docker image:

# Trigger all tags/branchs for this automated build.
$ curl -H "Content-Type: application/json" --data '{"build": true}' -X POST https://registry.hub.docker.com/u/cyberkov/openhab2/trigger/xxxx-xxxx-xxx-xxx-xxxxx/

Of course the current PR contains my repos for testing in the FROM field, but I would happily change these and squash the commits if you like it.

[cyberkov]

I noticed a problem when the volumes are mounted on the host (as I described in the README) that the /openhab/userdata and /openhab/conf directory will be empty.

This can be resolved through an entrypoint script which should check whether it's empty and copies the files over. I will update the PR in the next hours.

I'm also thinking about removing dockerize as it is only used for printing the logfile to stdout (which is supposed to be docker standard), while it could of course be used as well for waiting on a mysql database for instance. But I am unsure to which standard we should adhere, Dockers (stdout, would need reconfiguration of Karaf) or keep it in openhab's way (logging to /openhab/userdata/logs?

[christianwaite]

I'm finding it hard to find time to look into this, so I'll leave it to you guys. I'll test when I can.

[kaikreuzer]

But I am unsure to which standard we should adhere, Dockers (stdout, would need reconfiguration of Karaf) or keep it in openhab's way (logging to /openhab/userdata/logs?

Just an idea: For debian, we won't use openhab/userdata/logs either and we are discussing a new env variable ${log.dir} or similar, which Karaf would use as a log destination, if set.
For Docker, this variable could then simply be set to /dev/stdout, what do you think?

[umiddelb]

Moin,

• dockerize: I might have a deeper look inside understanding the differences between the armhf and the x86 release and why it is necessary to differentiate distinct platforms. BTW aarch64 is missing at this time.
• logging: Logging to stdout /stderr is the Docker way how to transport logging information to the 'ouside' and the logging architecture relies on this.
• build time arguments: There have been several initiatives to make things easier, this one was successful at the end.
• Java: There might be some nasty implications when we start to distribute Oracle's Java binaries within Docker images.
• ${log.dir}: What about extending this concept to more general logging destinations, like remote syslog servers?

Cheers
Uli

[cyberkov]

• "stdout" or some similar switch would be fine for me. Advanced logging functionality like taking care of remote syslog targets etc should be a concern of a "proper" syslog server, not the application. If I would need to do this, there'd be a chance that I would do this either on the docker host itself (logspout can do this iirc) or for a local installation rsyslog/syslog-ng will take care.
• Java: Yes I had the same thought. If we would get rid of dockerize and Oracle Java it would come down to one Dockerfile for each arch we are planning to built. In fact I was hoping to find some easy to use templating system for Dockerfiles to remove the redundancy in the repo but dockerhub only supports "Dockerfile" as name, so I had to put them into their own subdirectory. In addition the docker daemon does not allow adding files outside of the build context. This issue could be solved with Jenkins, as we could then use a build matrix with custom Dockerfile names (and the aforementioned Jenkins plugin).
• the build arg option would not be usable for the FROM (so to say architecture) but for the online and offline version which would bring the number of needed Dockerfiles down to $number_of_architectures_we_plan_to_support :)

[cyberkov]

I created a Jenkins Matrixbuild using the build-args method and it seems work quite well. The gist of the config can be found here.
The files for it are in the dev branch for now cyberkov/openhab-docker@2d45e80 but I plan to add when we figured all architectures that are going to be needed.
@umiddelb I was wondering whether you might like to provide crosscompile images (static compiled qemu) for the other archs? Unfortunately I'm very bad in all of this arm-based things :( I would like to provide an easy way of building nightlies and stable versions on cloudbees without any additional (or external) dependencies.

@kaikreuzer is it ok for you to go the openJDK path? We could then use the official java image (openjdk-8) for x86

One more thing that came to my mind:
How would you expect things like speedtest-cli should be handled? Should we provide a directory for hooks that can be executed on startup (e.g. pip install, apt-get install etc in a shellscript) or should this be built into a subsequent container?

Cheers
Hannes

[kaikreuzer]

@kaikreuzer is it ok for you to go the openJDK path?

I am aware of the license problems with the OracleJDK and I don't have a good solution. The issue is that I have seen quite some problems when using openHAB on OpenJDK - especially on ARM (like the RaspPi), the OpenJDK is terribly slow and almost unusable for openHAB. So in general, our regular advice is to use OracleJDK instead.

Now for the docker images, the problem exists only if we offer pre-build containers for download, right? I wonder if we can use OpenJDK in the pre-built containers, while still also offering docker files with OracleJDK, which people then have to build themselves locally (as this seems to be a pretty normal way of doing it as well, right?). What do you think?

[christianwaite]

Hey. I tried out your docker image cyberkov, but it's not working under Unraid.
We're getting a new version soon which should support compose, so I'll try again when that's up and running.

[ioft]

Now for the docker images, the problem exists only if we offer pre-build containers for download, right? I wonder if we can use OpenJDK in the pre-built containers, while still also offering docker files with OracleJDK, which people then have to build themselves locally (as this seems to be a pretty normal way of doing it as well, right?). What do you think?

It depends on the dockerhub's terms & conditions. The Docker image containing the OracleJDK binaries is made available for download at the dockerhub repository. So dockerhub might be hold liable in the first run, but then dockerhub's T&C apply. The best way to handle this issue is to build the OpenHAB image on top of an existing Java image provided by someone else (and there are some). Docker itself provides OpenJDK library images only and even Oracle as well.

I'm afraid the Docker Oracle JDK image has to be built from source by any individual.

[kaikreuzer]

I'm afraid the Docker Oracle JDK image has to be built from source by any individual.

Yes, I agree. So for the start, we should probably let go the idea of Docker hub and only provide the container description (using Oracle JDK).
Afterwards, we can also step into building an image with OpenJDK, but this will probably mean that we have to fix some bugs and problems first that occur when using OpenJDK - so I would not want to rush this...

Is this ok for you all? What is the status of this PR then in this light?

[umiddelb]

There is a common way to install Oracle 8 JDK inside a Docker container. I'm using a different base image (derived from Ubuntu Core) which makes it necessary to install software-properties-common first. In both cases the base image is derived from Ubuntu, so we may start with the official Ubuntu images (ubuntu, armhf/ubuntu, aarch64/ubuntu, and so on).

[cyberkov]

I sent an email to FSF Europe in the hope they may be able to help us in this matter :)

[kaikreuzer]

I sent an email to FSF Europe in the hope they may be able to help us in this matter :)

As stated on the Oracle-docker issue, there is hardly a chance to be able to re-distribute OracleJDK for ARM (and that is really the one that matters to us most, right?), because some more strict T&Cs apply there (the user actually has to confirm that he uses it only for development & eval purposes). So I am not sure that it will bring much to follow up on the legal side - but please let us know, what comes back from the FSF!

@cyberkov Are you meanwhile ok to create the docker file with JDK installation as @umiddelb describes?

@umiddelb Is there anything specific that one needs to consider in order to make the Docker file also work on Snappy Ubuntu Core? Do you know?

[umiddelb]

Is there anything specific that one needs to consider in order to make the Docker file also work on Snappy Ubuntu Core? Do you know?

No, the only prerequisite is the Docker daemon itself (and a Docker enabled Linux kernel). You can use your preferred Linux distribution as host environment (btw. Arch comes with excellent Docker support). AFAIK Snappy Ubuntu Core should work as well, but I haven't tried it yet.

[cyberkov]

Hi!

Sorry I've been busy the last days with my dayjob. I wanted to make sure we are on the same page as the thing with the Dockerfiles came up:

I wonder if we can use OpenJDK in the pre-built containers, while still also offering docker files with OracleJDK, which people then have to build themselves locally (as this seems to be a pretty normal way of doing it as well, right?)
My understanding is that Dockerfiles are supposed to be OpenSource as they describe a way to build your software (in that case a layer of the operatingsystem) and should be easily accessible to see what got "baked in". But unfortunately building the Dockerfile is something you usually do not do on your own (Think more of an .deb or .rpm package) rather it is done upfront for you.

So to achieve these two things (accessibility of the Dockerfile and ready-made images, one would need to go through dockerhub's autobuild feature or provide a clean documentation in the Readme from where this image came from. In both cases building the image means we can assume that the building docker daemon (cloudbees or dockerhub's autobuild) is going to run on x86 architecture and needs to be able to run the given arch in an emulated environment ("apt-get install", "unzip" etc) or we are going to have a buildfarm of docker daemons somewhere for each arch in someones basement.

@cyberkov Are you meanwhile ok to create the docker file with JDK installation as @umiddelb describes?
Yes of course. So I'll remove the base image (as it is only java and dockerize) and put the Java installation to the openhab Dockerfile for compliance.

For all other architectures than x86 I will try to get something with qemu going (as is now with the arm image) but I cannot test it which makes me more or less blind if this is going to work :(

On the other hand I found some things that I want to put into the container and hope they might be useful:

• configurable UID and GID for the openhab user
• proper initialization of the mounted host volumes (i'm still not confident it will work all the time)
• optional: proper Tests with Serverspec to make sure all the commands are actually working.

I hope i will have something in the next days and keep you posted :)
cheers Hannes

[kaikreuzer]

Thanks for your efforts!

[cyberkov]

@umiddelb Thank you! :D I cannot describe how happy I am! In the last few days I learned A LOT about crosscompiling and Linux, despite I've been using it since 2008 :) Especially your first post about hypriots approach on travis pointed me towards the right direction.
I've set up a new branch called multiarch-support and a travis-ci file to build amd64, arm64 and armhf containers (even 386 and ppc would be possible) and it seems to work quite well. There's still some finetuning to be done but I just wanted to drop this here.
The only downside on the build approach is, that it is not possible to use Dockerhubs autobuild as you need to change binfmt_misc features on the building dockerhost, but I think travis-ci should be a enough trusted source to be ok with that. You can see the build at https://travis-ci.org/cyberkov/openhab-docker/builds/117825426 (it's failing, but that's because I asked it to run the image afterwards).

By moving Java to the main Dockerfile I didn't see a reason for a baseimage at all (removed dockerize) and this brought the image down to 1 Dockerfile for all architectures and Flavors :)

I also started on the serverspec tests and they do work quite well. Still there's more to be done like checking on the different ways to invoke the image (with and without userdata as hostvolume, data-volume container pattern, demo, debugmode and so on...). Unfortunately the documentation on docker in combination with serverspec is very sparse.

When I finalized the travis config, including the uploading to dockerhub, I will update the PR to reflect my changes.

Cheers
Hannes

[cniweb]

Hi Hannes @cyberkov,
your .travis.yml:
https://github.com/cyberkov/openhab-docker/blob/multiarch-support/.travis.yml
works?

Cheers,
Chris

[cyberkov]

Hello @cniweb !

Yes it does. For instance in the armhf job you can see taht it's downloading and installing armhf packages:
Get:1 http://ports.ubuntu.com/ubuntu-ports/ wily/main libffi6 armhf 3.2.1-3 [16.2 kB]

This would normally not work on x86 :)

[cniweb]

OK, and this:
https://travis-ci.org/cyberkov/openhab-docker/jobs/117825430#L862
fail the Travis build, can you fix it?

[cyberkov]

Hello @kaikreuzer! Any luck changing the account type? I saw that the account still exists :)
Cheers Hannes

[kaikreuzer]

Yes, support has answered me today and told me that openhab was actually already an organization, but I didn't have any rights on it. It was Thomas who created it and he added me as an admin today - and I have just added @cyberkov and @cniweb as well, you should have received an according notification.

[cniweb]

Great, I create the Repo:
https://hub.docker.com/r/openhab/openhab/
@cyberkov can you configure this Repo?

[cyberkov]

Hi! @cniweb @kaikreuzer
I updated the README on dockerhub, merged the changes and travis seems to be ready as well.
As soon as the PR is accepted it should publish automatically to dockerhub :D

Next step would be the job on cloudbees. You need to set the TRAVIS_TOKEN and as an environment variable and execute shell build step with make trigger.

Cheers Hannes

[kaikreuzer]

Thanks! Before merging, could you please squash the 3 commits into a single one and sign-off the commit?
As it contains the work of others, please also list the other authors with an Also-By line in the commit.

[cyberkov]

Hi! I updated the message and squashed the commit. I am not used to the procedures of Eclipse projects so this is a bit unusual for me :)
The Makefile is based on multiarchs Dockerfile proof-of-concept. Would this mean that @moul would need to sign the commit as well?

[kaikreuzer]

I am not used to the procedures of Eclipse projects so this is a bit unusual for me

It is not that much an "Eclipse procedure" than rather a mechanism that I suggest to keep the repository clean from an IP perspective. We should make sure that code that ends up in the repo is clearly contributed under the EPL by all copyright holders (=authors).

Would this mean that @moul would need to sign the commit as well?

It depends on what "based on" exactly means. If you took his Makefile as an example to write your own version for our purposes, then all is fine. If you have copied some of his work (which is under his copyright), we would need his approval that he is fine to contribute it here unter the EPL as well.

[cyberkov]

Hello!
Despite one could argue that Makefile's are an implementation detail, I would prefer to go the safe route. I added @moul to the Also-By list. Do I need to inform him via Mail as well or is there anything to do for him?
Cheers Hannes

[kaikreuzer]

Copying code from @moul and adding him as Also-By means that you must have made sure that he agrees to contribute this code under the EPL. There is no license given for the Makefile in his own repo, so we will need an explicit confirmation from him. If he reads this and would comment on this issue that he is fine with it, that would be the simplest solution.

[cyberkov]

I sent him a mail as well :) Would be great to have his ok.

[moul]

License added :) have fun

[kaikreuzer]

Many thanks, @moul. As we are using EPL and not MIT in this repo, could you just briefly confirm that this is ok for you (i.e. that you contribute this file under the EPL to this repo by being a co-author of this PR)?

[moul]

I confirm this is OK for me, I choose MIT because it was listed as compatible with EPL on the EPL website :)

I'm not sure if this message is enough or if I need to commit something somewhere ?

[kaikreuzer]

This is enough, thanks again!

[cyberkov]

Hello @cniweb / @kaikreuzer!
Someone with write access to this repo needs to schedule and reschedule the failed builds on travis-ci. Unfortunately I cannot do this ;)
For the daily updates to work we are going to need the Jenkins instance do the triggering. Can you please deactivate the "build on pull requests" switch as well? Since there are no tests yet this would release a new version on dockerhub every time someone creates a PR.

Cheers
Hannes

[cyberkov]

@kaikreuzer thanks for adding :) I reconfigured Travis and restarted the one erroneous build.
Until we have it running in Cloudbees I added the trigger to my own Jenkins instance (worst case would be that we trigger the builds twice a day) to create the daily builds.

Cheers
Hannes

[kaikreuzer]

@cyberkov Just granted you access to the Cloudbees Jenkins as well - so feel free to set up a new job there!

[cyberkov]

@kaikreuzer thank you! I created the job and subscribed it as a downstream to openHAB-Distribution.
I ran into an issue with the naming as GIT_BRANCH was already taken by the Jenkins Github Plugin so I took the liberty to rename it and approve my PR, hope that's ok.
If everything works out we should see new docker images spawning as soon as a new openHAB-Distribution is built :)

Cheers Hannes

[kaikreuzer]

I took the liberty to rename it and approve my PR, hope that's ok.

Absolutely - imho small fixes can be merged by the committer himself. For other changes we should stick to a "lgtm" comment from someone else, just to follow the 4-eye pronciple.

[umiddelb]

@cyberkov please give me a short ping, if there is anything to be tested by myself on armhf or aarch64. I'm considering to open my 'lab' environment for external access, in order to run necessary tests automatically, but I need to learn more about OpwnWRT first ;)