Test symfony microservice

[whikloj]

GitHub Issue: Islandora/documentation#1224

Related PRs:

• Rewrite as symfony Islandora/Crayfish#76
• Change to symfony/flex Islandora/Crayfish-Commons#28
• Split silex and symfony configuration islandora-deprecated/ansible-role-crayfish#29

What does this Pull Request do?

Tests the above PRs

How should this be tested?

• Pull down this branch
• rm roles/external/*
• vagrant up
• Ensure the box comes up.
• Ensure image derivatives (Service File, Thumbnail File) still get created for your resources.

Interested parties

@Islandora-Devops/committers

[DonRichards]

"Ensure the box comes up and the microservices still work." is too vague for a new person to approach, in my opinion. I'd need the specified test steps to be a bit more concrete for me to feel confident I'm actually testing the right thing here.

1. Ingest a large image into a collection
2. look for x was created
3. Go to this and check for something
4. Here's an example illustration of the what you should see
5. Etc.

Or in the documentation a place that should have these steps that can be pointed to from a PR like this would be useful. I'm assuming it doesn't already exist. If it does, I'm an idiot. I should have looked for it first before writing this long response. Here's crossing my fingers I didn't just waist too much time with some guidance on making PRs more approachable. Anyways, thanks for the PR. :-D

[Natkeeran]

@DonRichards

• Verify syncing to Fedora and Triplestore
• Verifying that image and video/audio derivatives are created

The above should cover all the current microservices. For detail test cases, please see the following link: https://docs.google.com/spreadsheets/d/1K0OlrN4xE-jRKX7-zJCFhL980YjQBQZyKqqCGyScyZ8/edit#gid=0

[Natkeeran]

indexing image, and video microservices seem to be working based on sanity checks.

I do see the following issue when going to view the large image:

page not found
http://142.93.150.11:8000/http://142.93.150.11:8000/_flysystem/fedora/2019-08/UTSC.tiff

The following file does exist in fedora: http://142.93.150.11:8080/fcrepo/rest/2019-08/UTSC.tiff/fcr:metadata.

May not be a related issue!

[whikloj]

@Natkeeran correct.

That is because this (Islandora/openseadragon#13) has been merged but this claw-playbook doesn't have this (#139) change.

I could rebase it if you'd like.

[Natkeeran]

@whikloj
No need. I don't have enough symfony knowledge to do a proper review, but the sanity tests seem good. thank you.

[whikloj]

@Natkeeran you have as much symfony knowledge as I have and someone is going to have to say it works or not.

[whikloj]

@Islandora-Devops/committers Just bumping this, I know we're all busy but this does need to be done and it is only the first step towards complete de-Silex-ification.

[seth-shaw-unlv]

If no one gets to this first, I can probably take a look at it when I get back from IslandoraCon.

[seth-shaw-unlv]

@whikloj, I spun this up this morning. Image (tiff) and video (mp4) derivatives worked. My audio test (mp3) is crashing on GeminiLookup (so I don't think this is your issue). Digital document thumbnail derivatives for a PDF failed and produced the following entry in houdini's log:

[2019-10-14 10:54:20] app.DEBUG: Remember me skipped: it is not configured for the firewall. {"authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-10-14 10:54:20] app.DEBUG: > GET /houdini/convert [] []
[2019-10-14 10:54:20] app.INFO: Convert request. [] []
[2019-10-14 10:54:20] app.DEBUG: X-Islandora-Args: {"args":"-thumbnail 100x100"} []
[2019-10-14 10:54:20] app.DEBUG: Content Types: [] []
[2019-10-14 10:54:20] app.DEBUG: Content Type Chosen: {"type":"image/jpeg"} []
[2019-10-14 10:54:20] app.INFO: Imagemagick Command: {"cmd":"convert - -thumbnail 100x100 jpeg:-"} []
[2019-10-14 10:54:20] crayfish.cmd_execute_service.ERROR: Process exited with non-zero code. {"exit_code":1,"stderr":"convert: Postscript delegate failed `/tmp/magick-znAo436s': No such file or directory @ error/pdf.c/ReadPDFImage/678.\nconvert: no images defined `jpeg:-' @ error/convert.c/ConvertImageCommand/3046.\n"} []
[2019-10-14 10:54:20] app.ERROR: RuntimeException: {"exception":"[object] (RuntimeException(code: 500): convert: Postscript delegate failed `/tmp/magick-znAo436s': No such file or directory @ error/pdf.c/ReadPDFImage/678.\nconvert: no images defined `jpeg:-' @ error/convert.c/ConvertImageCommand/3046.\n at /var/www/html/Crayfish/Houdini/vendor/islandora/crayfish-commons/src/CmdExecuteService.php:120)"} []
[2019-10-14 10:54:20] app.DEBUG: < 500 [] []

I tested two different PDFs and both crashed with the same messages.

[whikloj]

Yeah this only affects Houdini, so only image derivatives. Any other successes or failures are unrelated. It looks like PDF is not compiled into imagemagick. Not sure when we added that to the playbook but I'll have a look. Thanks for giving this a test. 👍

[dannylamb]

islandora-deprecated/ansible-role-crayfish@0fefe97

It does a search for the imagemagick config directory. Maybe that's failing or there's some other subtle centos-ism at play.

[whikloj]

vagrant up with centos/7

[whikloj]

Ok, so the whole Digital document thing seems broken. I'm going to bring up a regular playbook to see if I can understand what it is supposed to do normally. I'm seeing errors about the hypercube executable path not being set.

[seth-shaw-unlv]

@whikloj, I'm turning my attention to some other tasks for now, but let me know when you need me to test something again.

[whikloj]

Problem was that when other PRs are merged then this type of alternate branch (used for testing) doesn't pull them in. I have rebased the ansible-role-crayfish PR to take in the changed configuration variables for hypercube.

I was able to use it for a basic & large image, audio, video and digital document in Centos.

If anyone else would please test this so it doesn't fall behind I'd appreciate it.

[seth-shaw-unlv]

Spinning up a new box right now.

[seth-shaw-unlv]

@whikloj, this branch doesn't build correctly as it doesn't account for the recent versioning updates. I spun up the box but attempting to index anything throws errors:

2019-12-18 17:34:50,062 | DEBUG | ing-fcrepo-file] | DefaultErrorHandler              | 57 - org.apache.camel.camel-core - 2.20.4 | Failed delivery for (MessageId: ID:claw-32771-1576689743881-3:3:-1:1:1 on ExchangeId: ID-claw-1576690145044-8-31). On delive
ry attempt: 3 caught: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "isNewVersion" (class ca.islandora.alpaca.indexing.fcrepo.event.AS2Object), not marked as ignorable (3 known properties: "type", "id", "url"])
 at [Source: java.io.ByteArrayInputStream@3131cb2f; line: 1, column: 511] (through reference chain: ca.islandora.alpaca.indexing.fcrepo.event.AS2Event["object"]->ca.islandora.alpaca.indexing.fcrepo.event.AS2Object["isNewVersion"])

[seth-shaw-unlv]

Should I just rebuild and deploy Alpaca? Or is there something in the playbook I should update and have it re-provision?

[whikloj]

Rebased

[whikloj]

Ugh this is screwed. Forget it. We'll just use Silex.

[whikloj]

Ok 😓 I think this should work. Just make sure to remove your roles/external/* before doing the vagrant up

[seth-shaw-unlv]

Ok. I did the build (CentOS) and ran my tests again. Everything looks good except those Digital Document thumbnails, which are spitting out the same error as before.

[whikloj]

Right and that seems to be a problem with convert being unable to create thumbnails for PDFs.

[vagrant@claw drupal]$ curl -o test.pdf http://127.0.0.1:8080/fcrepo/rest/2019-12/DRAFT-redraw--islandora-7x-architecture.pdf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  124k  100  124k    0     0  8117k      0 --:--:-- --:--:-- --:--:-- 8332k

[vagrant@claw drupal]$ convert test.pdf -thumbnail 100x100 jpeg:-
convert: Postscript delegate failed `test.pdf': No such file or directory @ error/pdf.c/ReadPDFImage/678.
convert: no images defined `jpeg:-' @ error/convert.c/ConvertImageCommand/3046.

I'll bring up a normal dev version and try the same thing.

[seth-shaw-unlv]

I can verify that this build works with Ubuntu 18, so there is a CentOS thing going on with this branch of the playbook.

[whikloj]

I'm going to bring up a normal dev centos/7 box and see if I can solve that problem separate from this PR.

[whikloj]

So the reason for this is super simple.

[vagrant@claw ~]$ gs -v
-bash: gs: command not found

[vagrant@claw ~]$ sudo yum update ghostscript
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: muug.ca
 * epel: muug.ca
 * extras: muug.ca
 * nux-dextop: mirror.li.nux.ro
 * remi-safe: mirror.team-cymru.com
 * updates: muug.ca
Package(s) ghostscript available, but not installed.
No packages marked for update

[vagrant@claw ~]$ sudo yum install ghostscript
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: muug.ca
 * epel: muug.ca
 * extras: muug.ca
 * nux-dextop: mirror.li.nux.ro
 * remi-safe: mirror.team-cymru.com
 * updates: muug.ca
Resolving Dependencies
--> Running transaction check
---> Package ghostscript.x86_64 0:9.25-2.el7_7.3 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================================================================
 Package                         Arch                       Version                             Repository                   Size
==================================================================================================================================
Installing:
 ghostscript                     x86_64                     9.25-2.el7_7.3                      updates                     112 k

Transaction Summary
==================================================================================================================================
Install  1 Package

Total download size: 112 k
Installed size: 150 k
Is this ok [y/d/N]: y
Downloading packages:
ghostscript-9.25-2.el7_7.3.x86_64.rpm                                                                      | 112 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : ghostscript-9.25-2.el7_7.3.x86_64                                                                              1/1 
  Verifying  : ghostscript-9.25-2.el7_7.3.x86_64                                                                              1/1 

Installed:
  ghostscript.x86_64 0:9.25-2.el7_7.3                                                                                             

Complete!

[vagrant@claw ~]$ convert test.pdf -thumbnail 100x100 jpeg:- > test.jpg

[vagrant@claw ~]$ identify test.jpg
test.jpg JPEG 100x56 100x56+0+0 8-bit DirectClass 6.58KB 0.000u 0:00.000

[whikloj]

New ticket for this tangential issue Islandora/documentation#1391

[seth-shaw-unlv]

Spinning up a fresh box with this PR combined with islandora-deprecated/ansible-role-crayfish#32.

[seth-shaw-unlv]

@whikloj everything is working. 🎊

Is there a particular order we should be merging these with?

[whikloj]

Crayfish first, then probably cut a new version of Crayfish. Then ansible-role-crayfish and cut a new version of ansible-role-crayfish.

Then I can update this PR for the new ansible-role-crayfish and make it ready for review.

The Crayfish-Commons will have to remain as a separate branch until all the other microservices are converted.

[seth-shaw-unlv]

👍
I've approved Crayfish but there are composer.lock conflicts that need to be resolved.

[seth-shaw-unlv]

@whikloj or @dannylamb, I presume the new Crayfish version should be 1.1.0? Or is this 1.0.1?

[whikloj]

Not really a bugfix, but no new features either. 🤷‍♂ Either is fine by me.

[seth-shaw-unlv]

I noticed while doing the release that we had several new features added since the last release, so 1.1.0 it is. Go ahead and update ansible-role-crayfish.

[whikloj]

Closing this as its purpose is now complete.