Add Dockerfile and a docker-compose to run Skosmos with Fuseki

[kinow]

Not sure if others would be interested. I started a simple Docker environment as I am on a new notebook and didn't want to install PHP + Apache + Fuseki etc just for developing SKOSMOS again.

To get everything up and running, once in the project folder:

$ docker-compose up
$ curl -I -u admin:admin -XPOST --data "dbName=skosmos&dbType=tdb" -G http://localhost:3030/$/datasets/
$ curl -I -X POST -H Content-Type:text/turtle -T vocabularies.ttl -G http://localhost:3030/skosmos/data

And with that SKOSMOS should be running on http://localhost:8000.

Feel free to close if that's not interesting at this point :-) just in case someone else was thinking about doing something along these lines. It can be easily modified to also support nginx.

[Jefwillems]

I am trying to get this to work, but when i add new data from other .ttl files, the data is not shown on the skosmos website.

I ran your commands first, then some more of the same curl requests:

$ curl -I -X POST -H Content-Type:text/turtle -T ./stadGentTTL/diensten.ttl -G http://localhost:3030/skosmos/data

as you can see here the data is not showing up and there is an internal server error on an api request.

my config.ini file is included here

I am running this environment on mac os 10.13.3.
Any idea how i can get the data to show on the skosmos website? I know it is showing up on the apache jena fuseki panel.

EDIT: when i search for data in the top right corner of the skosmos site, i get an error: 127.0.0.1:3030 connection refused

[kinow]

I can try it on a different computer in a few hours. Let me try it and see if there's something missing.

[osma]

@kinow Thanks for the PR! Many people have been interested in a Dockerized Skosmos.

@mhabsaoui Is this similar to what you've done to get Skosmos running in Docker?

I'd be happy if the people using Docker to run Skosmos could review and agree on this PR, then it can be merged. I've used Docker very little so it's difficult for me to comment.

[Jefwillems]

I got working now, connecting to 'fuseki' instead of localhost seems to work just fine.

[mhabsaoui]

Hi,

@osma Yep, I was using about the same docker-compose file running two containers : stain/jena-fuseki + a built one including php-apache which had a bind-mount volume (-v ./Skosmos:/var/www/html) in order to mount the skosmos web app (e.g. the Skosmos directory created by the git clone of skosmos repository) inside the skosmos container.

PS : Sorry, I was very ill late;y and didn't had the chance to modify the Tutorial as intended. Is it still needed or all info about Docker will be added to the Readme after this PR ?

@Jefwillems Indeed, 'localhost' is local (internally) to any docker container.So the querys fail because it doesn't go outside the skosmos container. Using the container name (e.g. fuseki) makes the querys to go outside the skosmos container, and to be delivered to the jena-fuseki container : all this thanks to the docker network (docker0).

@kinow We don't need all those extensions, no need more than this to make skosmos work (locale language install is though not madatory, but just to make Skosmos show vocabularies in the chosen default language...).

Cheers.

[kinow]

@Jefwillems got sidetracked at work and couldn't look on it, sorry. I believe you got it all working, and @mhabsaoui also included a great answer to your issue above.

@mhabsaoui owe you an apology. I hadn't seen there was a previous work on running Skosmos + Fuseki with Docker. Saw in issue #635 some notes about it too. I've confirmed that we didn't need all those extensions (PR updated). The main difference I see now between your example and this pull request is that you are using Docker Compose version 2 syntax, so things like the links and container-name seem to have been marked as deprecated. I'm happy to update the README, or perhaps provide a page for the Wiki, with simple instructions for running it. It would be grand if you could review it too :-)

[mhabsaoui]

@kinow You're welcome on any improvements to Skosmos 👍

Docker-Compose version 2 syntax

yeah docker-engine API version 18.02 / docker-compose version 3.6 now aleady released! (version 2 was just a constraint relative to my WIP project...)

the links and container-name properties seem to have been marked as deprecated

container_name seems still supported on latest reference doc

links seems still to be supported. But, i quote,

It may eventually be removed.

So with latest API version no more need of links as it is implicitly created by docker-compose. Rather creating user-defined-networks is a good practice...

And using depends_on is the right way to declare the depency of Skosmos on Jena-fuseki TDB 👍

I'm happy to update the README, or perhaps provide a page for the Wiki, with simple instructions for running it. It would be grand if you could review it too :-)

=> My pleasure to review it ASAP ;-)

Cheers.

[mhabsaoui]

@osma just FYI, as promised I have initiated a wiki page for Docker use on Skosmos based on @kinow 's fork. Did not found the merged branch dedicated to Docker on NatLibFi...

Cheers.

[kinow]

@mhabsaoui excellent doc!!! I'm updating this pull request to include a locale in the installation steps of the Dockerfile. If I remember well, yours had at least the French locale. I didn't copy that part from your previous example, and only when @osma pointed out in #704 that that's the reason why I was seeing in_this_language instead of the right text.

1/ Not sure if we should include a note about it in the wiki?
2/ I think some users might be interested in running just one of the containers. Like starting up just Fuseki, which for this case we could just add a link to the stain/jena-fuseki docker-hub docs. Or for starting up just Skosmos, and point to an existing endpoint. I find this latter example quite practical, especially when you have a running instance, and want to test if skosmos would work. WDYT?

Props again for the amazing - and quick - job with the docs!
Bruno

[kinow]

Finished updating the Dockerfile! Had some trouble with the order of commands, as setting the locale by last, it seemed Apache was not able to detect the three locales installed. Found this issue docker-library/php#318 that had a good example similar to what we needed.

Adapted the part of locales to support what comes in the config.inc.dist by default, en_GB, fi_FI, and sv_SE, but happy to add other locales if necessary.

Now top menu working as expected! Translations working with no issues so far.

Cheers
Bruno

[mhabsaoui]

@kinow Yep, indeed I remember playing some with locales as following

FROM php:apache RUN apt-get update && apt-get install locales && \ echo "fr_FR.UTF-8 UTF-8" >> /etc/locale.gen && \ echo "en_GB.UTF-8 UTF-8" >> /etc/locale.gen && \ locale-gen fr_FR.UTF-8 && \ locale-gen en_GB.UTF-8 UTF-8 && \ a2enmod rewrite && \ docker-php-ext-install gettext mbstring

1/ Not sure if we should include a note about it in the wiki?

Always usefull to add as tip for extended languages needs.

2/ I think some users might be interested in running just one of the containers..., and want to test if skosmos would work. WDYT?

Why not, could be interesting to add as a suggestion which would make the Dockerfile part clearer maybe... For example ==>

$ docker run --rm --name jena-fuseki -p 3030:3030 -e ADMIN_PASSWORD=jfadminpass -e JVM_ARGS=-Xmx2g -v "$PWD"/fuseki:/fuseki stain/jena-fuseki

$ docker run --rm -p 8080:80 --name skosmos -v "$PWD":/var/www/html/ php:7.0-apache
/var/www/html/.htaccess: Invalid command 'RewriteEngine', perhaps misspelled or defined by a module not included in the server configuration

Cheers.

[kinow]

@osma @henriyli @mhabsaoui I believe this is ready for review now.

This pull request provides the Dockerfile for building a Skosmos container with Apache httpd, some PHP extensions, and the locales for the default languages in the config.inc.dist file.

The documentation was done by @mhabsaoui , and if this PR gets merged, we just need to re-word the following

You need to first clone the Skosmos-Docker version from kinow's github repository. Once cloned, go inside this cloned directory and proceed as following.

To reflect that the repository to be cloned is this one 🎉

Ta
B

ps: @mhabsaoui I noticed reviewing the docs tonight again the trick for running composer in a Docker container! Sometimes devs at $work have issues with composer (especially after fiddling with extensions like xdebug), so will add a note in the internal Wiki with your tip :D thanks heaps!

[osma]

@mhabsaoui Can you please check whether this is OK from your perspective? I would like to merge this sooner rather than later (definitely for Skosmos 2.0) but don't really have the time to delve into Docker now. If it's working for at least two people then it could go in and be improved later if necessary.

[mhabsaoui]

@osma Checked Dockerfile + docker-compose for skosmos/fuseki ==> OK.

1. docker-compose- up => OK
2. creating dataset => OK

curl -I -u admin:admin -XPOST --data "dbName=skosmos&dbType=tdb" -G http://localhost:3030/$/datasets/

3. adding YSA and YSO vocabs data to previously created skosmos fuseki TDB => OK

curl -I -X POST -H Content-Type:text/turtle -T ysa-skos.ttl -G http://localhost:3030/skosmos/data

As we can see below, tripleCount/triples is the same as ysa'one 👍

Yet, here some additionnal notes :

• It may be useful to add the contqiner_name property so that containers get by default names : skosmos and fuseki...

• Volume for fuseki to persist TDB ? Could of course confuse docker beginners as data will still be there until the volume directory is explicitly removed...

• For Skosmos app to work properly, of course the config.inc must exist (simply rename config.inc.dist to config.inc).
  Same goes for vocabularies.ttl (simply rename vocabularies.ttl.dist to vocabularies.ttl). BTW, the beginner user must be aware that this vocabularies.ttl file points to http://api.dev.finto.fi/sparql sparqlEndpoint by default, and not the localhost fuseki TDB ! In fact, this vocabularies.ttl file has to be edited with the localhost fuseki TDB sparqlEndpoint, e.g. http://localhost:3030/ds/sparql. And then execute the special commands to create dataset and populate it with vocabulary data...
  And finally, be also aware that Skosmos languages need to be activated by adding in config.inc as following

Otherwise, Skosmos shows error about url failing as following...

FR not activated in Skosmos config

FR activated in Skosmos config

EN selected on Skosmos

• for skosmos to point correctly to fuseki TDB server and get data from it, in config.inc the "localhost" must be replaced by fuseki docker container name, e.g. by default "fuseki"
  ==> define("DEFAULT_ENDPOINT", "http://fuseki:3030/ds/sparql");

Also, in vocabularies.ttl file the same thing, plus chaning all URL pointing to finto so that it points to localhost fuseki TDB...

Cheers.

[kinow]

Thanks for the great and detailed reply @mhabsaoui !

It may be useful to add the contqiner_name property so that containers get by default names : skosmos and fuseki...

Agreed. Adding in a new commit. After that, here's the result of docker-compose up:

~/Development/php/workspace/Skosmos$ docker-compose up
Recreating skosmos_fuseki_1 ... 
Recreating skosmos_fuseki_1 ... done
Creating skosmos-web ... 
Creating skosmos-web ... done
Attaching to skosmos-fuseki, skosmos-web
skosmos-web | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.19.0.3. Set the 'ServerName' directive globally to suppress this message
skosmos-web | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.19.0.3. Set the 'ServerName' directive globally to suppress this message
skosmos-web | [Mon Mar 12 09:33:16.348831 2018] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.10 (Debian) PHP/7.0.27 configured -- resuming normal operations
skosmos-web | [Mon Mar 12 09:33:16.348873 2018] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
skosmos-fuseki | [2018-03-12 09:33:16] Server     INFO  Apache Jena Fuseki 3.6.0
skosmos-fuseki | [2018-03-12 09:33:16] Config     INFO  FUSEKI_HOME=/jena-fuseki
skosmos-fuseki | [2018-03-12 09:33:16] Config     INFO  FUSEKI_BASE=/fuseki
skosmos-fuseki | [2018-03-12 09:33:16] Config     INFO  Shiro file: file:///fuseki/shiro.ini
skosmos-fuseki | [2018-03-12 09:33:17] Config     INFO  Configuration file: /fuseki/config.ttl
skosmos-fuseki | [2018-03-12 09:33:17] Config     INFO  Load configuration: file:///fuseki/configuration/skosmos.ttl
skosmos-fuseki | [2018-03-12 09:33:17] Config     INFO  Register: /skosmos
skosmos-fuseki | [2018-03-12 09:33:17] Server     INFO  Started 2018/03/12 09:33:17 GMT on port 3030

Volume for fuseki to persist TDB ? Could of course confuse docker beginners as data will still be there until the volume directory is explicitly removed...

Indeed. In my case I use Docker with Skosmos to reproduce issues and experiment a few things. Normally I prefer that it's not persisted. But other users may prefer to use a volume.

What about the following comment in the Dockerfile? Pushing in a commit here, but happy to revert it later.

  # You can uncomment the line below to have a local volume bound onto the container, or
  # visit https://hub.docker.com/r/stain/jena-fuseki/ for other alternatives
  # volumes:
  #  - ${PWD}/fuseki:/fuseki

Though volumes is an option, there's another alternative in the stain/jena-fuseki docs, that some users may prefer. And, of course, we add a note to the documentation about it.

For Skosmos app to work properly, of course the config.inc must exist (simply rename config.inc.dist to config.inc). (...)

I believe all of these final notes can be addressed in the Wiki documentation. So there would be no change in this PR. WDYT?

[kinow]

Pushed two more commits, and also rebase+push-force'd.

[mhabsaoui]

@kinow Except the container naming I would use as a preference fuseki / skosmos (instead of skosmos-fuseki / skosmos-web to avoid confusion ?), it's OK 👍

[kinow]

I use a fuseki container for Jena development already, so I thought about giving it a more specific name so I know which container is which. Perhaps skosmos and skosmos-fuseki?

[mhabsaoui]

@kinow Of course we can use any whiched container name by modifying compose file.
I was aiming rather the beginner user to make clearer for him to understand that fuseki is independent from skosmos...

[osma]

skosmos and skosmos-fuseki sounds good to me.

[kinow]

skosmos and skosmos-fuseki sounds good to me.

@osma @mhabsaoui names update to skosmos and skosmos-fuseki

Notes about volume data persistence for Fuseki added in this revision of the Wiki

Notes about the configuration file changes required to use local Docker fuseki added in this revision

There were already notes about locale (we also provide the same set that is included in the default configuration file).

(I didn't put the configuration file specific name (config.inc) as I believe it may change near 2.x release?) 🎉

[kinow]

Pushed once again after rebasing against the master branch 👍 ready for review once Scrutinizer+Travis are OK building it

[osma]

Thanks for your hard work on this! I think it's good enough to go into 2.0. After all it's not going to do any harm for non-Docker usage. I will merge this as it is and if new problems turn up, please open another issue or PR.

[kinow]

Thanks for merging and for preparing 2.0. Kudos to @mhabsaoui who had started it before, and had patience to review my pull request with great suggestions, and also started the Wiki page!