Flysystem Install

[dannylamb]

Install code for Islandora/documentation#769

Using this PR to install with Flysystem support

• Pull this code into claw-playbook
• vagrant up
  • It'll throw a wall of red text on you at the end, but that's because of the fix for claw-playbook unmet dependencies Islandora/documentation#873
• vagrant ssh
• curl https://gist.githubusercontent.com/dannylamb/27ad287ae92afd7139a98435446550d6/raw/6e9f369139528d3445e19e187e6a22a5a1386d4d/gistfile1.txt | sudo bash
  • It'll throw another message at you about some core config still referencing the article bundle, but that's ok. It's benign, but I have to sort that out in islandora_core_feature.

Testing Flysystem Support

I've been testing by adding an Image with a Preservation Master and watching it go. It's easiest to see what's going on by checking Gemini. Before doing anything, I confirm that the taxonomy terms are in and mapped.

vagrant@claw:~$ mysql -u root -pislandora gemini
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 119
Server version: 5.7.22-0ubuntu0.16.04.1 (Ubuntu)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select fedora_uri, drupal_uri from Gemini;
+------------------------------------------------------------------------------------+------------------------------------------------------+
| fedora_uri                                                                         | drupal_uri                                           |
+------------------------------------------------------------------------------------+------------------------------------------------------+
| http://localhost:8080/fcrepo/rest/03/ad/74/0c/03ad740c-12b5-4b6b-950d-f26068dc16fe | http://localhost:8000/taxonomy/term/3?_format=jsonld |
| http://localhost:8080/fcrepo/rest/05/b3/33/25/05b33325-c6c3-4173-8ca7-ab892e173af7 | http://localhost:8000/taxonomy/term/9?_format=jsonld |
| http://localhost:8080/fcrepo/rest/06/96/8c/b0/06968cb0-318d-4ac9-a544-7e088cf29589 | http://localhost:8000/taxonomy/term/5?_format=jsonld |
| http://localhost:8080/fcrepo/rest/08/f4/c4/7d/08f4c47d-b1ec-4fb9-983c-9bbc39675b18 | http://localhost:8000/taxonomy/term/4?_format=jsonld |
| http://localhost:8080/fcrepo/rest/13/5c/04/b1/135c04b1-d6c7-490f-b4f1-280dc0aa06c0 | http://localhost:8000/taxonomy/term/1?_format=jsonld |
| http://localhost:8080/fcrepo/rest/2d/4f/70/f9/2d4f70f9-3c92-483e-a5df-fb4990e6d039 | http://localhost:8000/taxonomy/term/7?_format=jsonld |
| http://localhost:8080/fcrepo/rest/49/82/d2/75/4982d275-f612-4886-a527-b7d03986b391 | http://localhost:8000/taxonomy/term/8?_format=jsonld |
| http://localhost:8080/fcrepo/rest/cc/53/e4/01/cc53e401-350f-4ce4-9882-f21d243a8c35 | http://localhost:8000/taxonomy/term/6?_format=jsonld |
| http://localhost:8080/fcrepo/rest/fe/47/a0/bc/fe47a0bc-29b3-4d6b-9bf5-7cb7d3b89604 | http://localhost:8000/taxonomy/term/2?_format=jsonld |
+------------------------------------------------------------------------------------+------------------------------------------------------+
9 rows in set (0.00 sec)

Then go add an Image node and give it a Preservation Master media.

• From the landing page, click "+ Add Content"
• Select "Repository Item"
• Fill out the form and give it a tag of Image
• After submitting the form, view the Image object and click on its "Media" tab
• Click "+ Add Media"
• Click "Image"
• Fill out the form, upload a file, and give it a tag of "Preservation Master"
• Hover over the image in the Media's display, and you can see the link to the file is http://localhost:8000/_flysystem/fedora/2018-07/...

The file you uploaded will wind up in Fedora, but the derivatives will land in Drupal's public filesystem. Check out Gemini again and it will have all the relevant nodes/media/files mapped.

mysql> select fedora_uri, drupal_uri from Gemini;
+--------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| fedora_uri                                                                                             | drupal_uri                                                                                                   |
+--------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| http://localhost:8080/fcrepo/rest/03/ad/74/0c/03ad740c-12b5-4b6b-950d-f26068dc16fe                     | http://localhost:8000/taxonomy/term/3?_format=jsonld                                                         |
| http://localhost:8080/fcrepo/rest/05/b3/33/25/05b33325-c6c3-4173-8ca7-ab892e173af7                     | http://localhost:8000/taxonomy/term/9?_format=jsonld                                                         |
| http://localhost:8080/fcrepo/rest/06/96/8c/b0/06968cb0-318d-4ac9-a544-7e088cf29589                     | http://localhost:8000/taxonomy/term/5?_format=jsonld                                                         |
| http://localhost:8080/fcrepo/rest/08/f4/c4/7d/08f4c47d-b1ec-4fb9-983c-9bbc39675b18                     | http://localhost:8000/taxonomy/term/4?_format=jsonld                                                         |
| http://localhost:8080/fcrepo/rest/0d/66/06/83/0d660683-551c-4037-b49b-38634f4e97f0                     | http://localhost:8000/media/3?_format=jsonld                                                                 |
| http://localhost:8080/fcrepo/rest/13/5c/04/b1/135c04b1-d6c7-490f-b4f1-280dc0aa06c0                     | http://localhost:8000/taxonomy/term/1?_format=jsonld                                                         |
| http://localhost:8080/fcrepo/rest/2d/4f/70/f9/2d4f70f9-3c92-483e-a5df-fb4990e6d039                     | http://localhost:8000/taxonomy/term/7?_format=jsonld                                                         |
| http://localhost:8080/fcrepo/rest/49/82/d2/75/4982d275-f612-4886-a527-b7d03986b391                     | http://localhost:8000/taxonomy/term/8?_format=jsonld                                                         |
| http://localhost:8080/fcrepo/rest/59/0c/10/d5/590c10d5-fb76-4789-86d0-3e3776a2aaf0                     | http://localhost:8000/node/1?_format=jsonld                                                                  |
| http://localhost:8080/fcrepo/rest/80/0a/e1/1c/800ae11c-06ce-413d-87a9-4fe38c046abc                     | http://localhost:8000/media/2?_format=jsonld                                                                 |
| http://localhost:8080/fcrepo/rest/2018-07/Hyboid-presents-Sternrekorder-Mix-for-Valerie-Collective.jpg | http://localhost:8000/_flysystem/fedora/2018-07/Hyboid-presents-Sternrekorder-Mix-for-Valerie-Collective.jpg |
| http://localhost:8080/fcrepo/rest/cc/53/e4/01/cc53e401-350f-4ce4-9882-f21d243a8c35                     | http://localhost:8000/taxonomy/term/6?_format=jsonld                                                         |
| http://localhost:8080/fcrepo/rest/fe/47/a0/bc/fe47a0bc-29b3-4d6b-9bf5-7cb7d3b89604                     | http://localhost:8000/taxonomy/term/2?_format=jsonld                                                         |
+--------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
13 rows in set (0.01 sec)

Note that files in Fedora are mapped in Gemini, whilst the others are not. Conversely, for Media, if the source is not in Fedora it is mapped in Gemini. Media whose source are in Fedora are not mapped, but looked up using the describedby link header in Fedora responses.

You should be able to confirm that all the resources that should be are present in Fedora.

[dannylamb]

My gist was failing to deploy some openseadragon changes, so I updated it and the install instructions. This PR is good to go now.

[seth-shaw-unlv]

Works as advertised. Yay!

Note: on CentOS the last line of the gist didn't work (jar copying failed); but it wasn't any trouble to run that line's commands one at a time.

[whikloj]

Ok, took me a bit to understand but if I understand correctly the derivative "files" are not moved to Fedora but the derivative media objects are.

[dannylamb]

@whikloj Yes. That's it. Once Fedora 5 lands we'll actually wire those Media up as external content. The approach in this PR would remain (and probably be improved) to work with something like Trellis which won't support the feature.

[dannylamb]

@Islandora-Devops/committers @manez @MarcusBarnes @ysuarez @mjordan @rosiel @bryjbrown @DigitLib and whoever else^^

Ok, Travis is green everywhere except islandora-deprecated/islandora_image#24, which requires Islandora/documentation#92 to get merged first. islandora_image tests extend the base test in islandora.

I've responded to a lot of feedback already, but more is welcome. I've cycled my box using these instructions and everything works as I expect it to.

Some things I've done:

• Make a new Image and give it a Preservation Master
  • Make sure everything shows up in Fedora
  • Make sure everything shows up in the triplestore
• Configure one (or both) of the derivative actions to use Fedora instead of the public filesystem.
  • Ingest another object w/preservation master
  • Go through the same checks.
• Update the preservation master using the REST api
  • curl -i -X PUT -H "Authorization: Bearer blahblahblah" -H "Content-Type: image/jpeg" --data-binary @some_image.jpg http://localhost:8000/node/1/media/image/2
  • Make sure new derivatives are generated.
  • I tried various sized images to make sure there's no wonkiness with image resizing.

[dannylamb]

You should be able to test now without having to run that bash script. Pull in these changes and vagrant up should work.

[whikloj]

@dannylamb ansible-role-crayfish needs to move to 0.0.4

[dannylamb]

@whikloj Done. Thx. I manually set the crayfish version in my inventory to test before islandora-deprecated/ansible-role-crayfish#11 landed and didn't notice this.

I assume this means we'll need to run ansible-galaxy manually before vagrant uping to test?

[whikloj]

No I think when you run vagrant up it will try to overwrite it...I think. I'll give it a try and see what happens.

[whikloj]

Nope it doesn't overwrite, easiest is to remove the contents of your roles/external directory.

[whikloj]

Ubuntu works after removing the roles/external directory.
Testing Centos now.

[whikloj]

Also works on Centos.