feat(update): Add new repository update framework

[benjamingeer]

This PR replaces the Python-based repository update tool with one written in Scala, to improve performance.

Resolves #1451.

[subotic]

I've moved the upgrade code to its own subproject.

[subotic]

lol, did the same commit at almost the same time :-)

[benjamingeer]

Great minds think alike. :)

[benjamingeer]

And thanks for making the subproject!

Before I update the documentation, how do you want to run the Main program? Is it going to be in Docker? If so, should we document how to run it in SBT as well as in Docker?

[lrosenth]

🤪 Von meinem iPhone gesendet Am 04.10.2019 um 09:11 schrieb Benjamin Geer <notifications@github.com>:  Great minds think alike. :) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#1458>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABJX3TFQKRGFGAKZOTEEBWLQM3UAHANCNFSM4I4VA75Q>.

[subotic]

And thanks for making the subproject!

You're welcome :-)

Before I update the documentation, how do you want to run the Main program? Is it going to be in Docker?

Yes, it is going to run in its own Docker image. I would like to run it as the python verson before, i.e., with parameters for GraphDB hostname, repository name and work folder.

If so, should we document how to run it in SBT as well as in Docker?

Documenting Docker is difficult as it depends on the way how it is orchestrated. The way we run it in Docker is just one way of doing it. It works for us at the moment, but will change over time. I think it is enough to document it on how to run it from SBT.

[benjamingeer]

I would like to run it as the python verson before, i.e., with parameters for GraphDB hostname, repository name and work folder.

I was thinking of giving the sysadmin a bit more control over the process, by having the Scala upgrade program just read an input file and write an output file. In upgrade/graphdb-se there are scripts for dumping a repository to a TriG file, emptying a repository, and uploading a TriG file to a repository, and these scripts take the usual parameters.

This way, if the TriG file was very large, you could use GraphDB's LoadRDF instead of the upload script.

And if Knora rejects your TriG file and you need to fix some things manually in it, you could just do that and upload it again.

Also, the same Scala upgrade program would work with TriG files from other triplestores.

How does that sound?

[subotic]

I would prefer a full-auto mode, i.e., run upgrade script and wait until it finishes.

Being able to do the upgrade in manual steps is good for debugging if the full-auto mode does not work.

So if possible, then please add both options. Manually working with trig files is error-prone (I have this "pleasure" on each deployment). Having to deal with them additionally for upgrades, is not so ideal.

[benjamingeer]

please add both options.

OK, will do.

[subotic]

Could you please add the option to optionally provide the path to the directory where stuff is getting downloaded to?

In the old python script there was an option for this: "-t", "--tempdir", help="temporary directory"

Also, it seems that for auto-upgrade.sh the -u and -p options are mandatory, even if they are not needed locally.

[benjamingeer]

Will do.

[benjamingeer]

I think I need to get you a wizard costume for Halloween.

[subotic]

I think I need to get you a wizard costume for Halloween.

lol :-)

[subotic]

I finally got it working. There is a Travis test covering the sbt upgrade use case. The Docker use case is still missing, but I'm going to add it.

[benjamingeer]

Also, it seems that for auto-upgrade.sh the -u and -p options are mandatory, even if they are not needed locally.

This is also true with the old upgrade framework. If you don't specify a password, you get a password prompt:

parser.add_argument("-u", "--username", help="GraphDB username", type=str, required=True)
parser.add_argument("-p", "--password", help="GraphDB password (if not provided, will prompt for password)",
                    type=str)

I could add an option --no-auth.

[subotic]

I could add an option --no-auth.

no, it‘s fine.

[subotic]

I think that this PR is good now. Thanks for making the upgrade faster!

[benjamingeer]

Thanks for the review and the SBT magic!