One app to rule them all #82

whikloj · 2021-10-19T01:06:40Z

GitHub Issue: Islandora/documentation#731

What does this Pull Request do?

Moves Alpaca from an OSGI bundle to a single runnable jar with all the various services inside.

What's new?

Removes OSGI bindings
Pulls SparqlUpdateProcessors in from fcrepo-camel code (it was causing problems with Jena libraries otherwise)
Create single runnable jar that takes configuration from a properties file.
Does this change require documentation to be updated? Docs are updated
Does this change add any new dependencies? No
Does this change require any other modifications to be made to the repository (i.e. Regeneration activity, etc.)? No
Could this change impact execution of existing code? Deployment will need to be altered.

How should this be tested?

Using a playbook or ISLE instance, build the jar and then copy it into the system (somewhere it can access Fedora, Blazegraph, ActiveMQ, the microservices).

Shutdown Karaf and copy the settings from all the various configuration files to a single properties file.

Start the jar with that properties file and it should perform all the various actions.

ie.

java -jar islandora-alpaca-app-2.0.0-all.jar -c config.properties

Additional Notes:

There are some stacktraces thrown that I left in place but are not actually a problem (for instance when you choose a media file, Drupal triggers us to send an event but it doesn't have all the necessary URLs yet. It can be ignored but could also be a sign of a problem.

Could leave the less verbose in the INFO level and move the stacktrace to the DEBUG logging level, but I don't want anyone to miss an actual problem. 🤷

Interested parties

@Islandora/8-x-committers (especially those you don't hate Java or at least will do Java, @dannylamb, @elizoller, @adam-vessey, etc)

codecov · 2021-10-19T19:46:05Z

Codecov Report

Merging #82 (2b5e23d) into 2.x (1bb1918) will decrease coverage by 11.72%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##                2.x      #82       +/-   ##
=============================================
- Coverage     89.80%   78.07%   -11.73%     
- Complexity       79      143       +64     
=============================================
  Files            11       30       +19     
  Lines           304      561      +257     
  Branches          1       27       +26     
=============================================
+ Hits            273      438      +165     
- Misses           30      101       +71     
- Partials          1       22       +21

Impacted Files	Coverage Δ
...lpaca/indexing/triplestore/TriplestoreIndexer.java	`96.22% <0.00%> (-0.33%)`	⬇️
...aca/http/client/StaticTokenRequestInterceptor.java	`73.33% <0.00%> (ø)`
...paca/connector/derivative/DerivativeConnector.java	`100.00% <0.00%> (ø)`
...java/ca/islandora/alpaca/support/event/AS2Url.java
.../islandora/alpaca/support/event/AS2Attachment.java
...a/ca/islandora/alpaca/support/event/AS2Object.java
...ora/alpaca/support/event/AS2AttachmentContent.java
...va/ca/islandora/alpaca/support/event/AS2Event.java
...va/ca/islandora/alpaca/support/event/AS2Actor.java
.../connector/derivative/RequestConfigConfigurer.java
... and 27 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1bb1918...2b5e23d. Read the comment docs.

elizoller · 2021-10-20T18:00:30Z

can you be more specific about

Start the jar with that properties file
?
like literally java -jar ... or do i need to run it some other way?

whikloj · 2021-10-20T18:03:22Z

Sorry @elizoller I didn't realize I hadn't put that in here.

I did update the README which (hopefully) has all the needed information. https://github.com/Islandora/Alpaca/blob/d92362fd418cd5ad2d306ef7901a536bd555907d/README.md

README.md

whikloj · 2021-10-22T16:26:39Z

Sorry folks I decided to pull in the changes from #66 as well. This should be ready for a full test.

elizoller · 2021-10-26T16:47:35Z

I feel like I can't really speak to the code part of this directly but I can provide feedback on what I did to test this PR.
I pulled in this PR in alpaca (in a playbook) and used the gradle command in the readme to build it. Then I started the jar with the commands provided in the readme with the provided examples.properties file. Then I went into the playbook and created some items and media. I verified the following functionality works: indexing items and media in fedora, deleting items removes them from fedora, updating an item creates a version in fedora, homarus/ffmpeg derivs generated, houdini/imagemagick derivatives generated, hypercube/tesseract/pdf full text extracted, crayfits/fits derivatives generated.

whikloj · 2021-10-27T14:09:09Z

No one can speak to Java code.

Like all code it is best glimpsed in one's peripheral vision. It is the fairy lights in the trees, the oasis on the horizon, it is real but subjective.

islandora-indexing-fcrepo/src/main/java/ca/islandora/alpaca/indexing/fcrepo/FcrepoIndexer.java

seth-shaw-unlv · 2021-10-27T15:49:56Z

Other than my question about connectionClose (see above), I don't see any obvious issues. I think we can probably trust your smoke tests on this, @elizoller.

seth-shaw-unlv · 2021-10-27T16:07:48Z

Okay, as far as I'm concerned, if @elizoller is happy with his tests and @adam-vessey has his questions adequately addressed, 👍.

whikloj · 2021-11-03T21:15:02Z

Sorry for the delay on this, I have reverted the URL changes and then I had to work out a way of testing as the old way did not seem to work.

adam-vessey

Noticed PMD flagging things... should we be introducing/adding in some Github actions that check for these?

...tor-derivative/src/main/java/ca/islandora/alpaca/connector/derivative/DerivativeOptions.java

islandora-indexing-fcrepo/src/main/java/ca/islandora/alpaca/indexing/fcrepo/FcrepoIndexer.java

...estore/src/main/java/ca/islandora/alpaca/indexing/triplestore/processors/ProcessorUtils.java

islandora-support/src/main/java/ca/islandora/alpaca/support/config/PropertyConfig.java

islandora-support/src/main/java/ca/islandora/alpaca/support/event/AS2Object.java

DonRichards · 2021-11-29T21:32:34Z

@whikloj Sorry, I'm about to show how little I understand Alpaca and this pull request but is it possible to have gradle do the "build the jar and then copy it into the system (somewhere it can access Fedora, Blazegraph, ActiveMQ, the microservices)."?

whikloj · 2021-11-30T16:23:56Z

Ok there are still 2 PMD rule issues we need to make a decision about, but otherwise they are cleaned up.

@DonRichards once Alpaca is finalized we can deploy a build jar to Maven and Github then people deploying it can just download the pre-built jar instead of having to compile it themselves.

* Change up instantiation/initialization slightly. * PostConstruct is allowed to apply to private methods... ... seems like a false-positive in PMD? Got example exception from: https://stackoverflow.com/a/48679770 * Changing to camel's 3.7.6 LTS.

adam-vessey · 2021-12-06T21:40:35Z

So, did realize something with the concurrent consumer stuff: It wouldn't really allow multiprocessing without the additional asyncConsumer parameter... throwing together another little PR to introduce it: whikloj#2

whikloj · 2021-12-06T22:04:02Z

So I think it depends on what you mean by multiprocessing. For instance in our Islandora Legacy I use camel to kick off tesseract, it will run concurrentConsumers equal to the number of cores - 1, but each message is processed completely before the a new one is started. I'm not against the PR for including asyncConsumer, I'm just not completely clear how it decides when to grab a new item to start processing. Perhaps this is where the difference between concurrentConsumers and maxConcurrentConsumers` comes in?

adam-vessey · 2021-12-07T14:41:23Z

Possibly... my interpretation of concurrentConsumers vs maxConcurrentConsumers is somewhat analogous to the -Xms and -Xmx values to the JVM... initial value when starting, and the max to which things can grow if called for.

Without asyncConsumer, it seems like only a single thing for each queue can be processed at a time (per instance of consumers listening, anyway... as in, running multiple instances of the jar could allow for multiple, in a round-about way), but that said... and it's entirely possible that I'm missing something, but given how it's described, I'm not sure of the value of being able to specify concurrentConsumers without also specifying asyncConsumer=true offers much... maybe a alleviating some amount of overhead?

whikloj · 2021-12-07T15:11:42Z

I think we are in somewhat agreement, the only difference (I think) is that I now believe that without asyncConsumers the concurrentConsumers would never rise from its initial value and maxConcurrentConsumers might be irrelevant.

I do know that you can set concurrentConsumers to a number greater than 1 and have multiple messages processed at the same time. Async is (I think) that once the message has been pulled from the JMS queue and handed off (maybe just to the route or maybe to a secondary route) then a new message is grabbed from the JMS queue. In normal processing it would wait until the first message completed the entire route.

I'll give your PR a build today and I have no issue adding it as a configurable option.

adam-vessey · 2021-12-07T21:10:55Z

Hmm... yeah, giving things a further poke, and seeing multiple convert instances at a time, from Houdini; however... I wonder... is it because of the streaming? Not really familiar with Camel's lifecycle WRT JMS messages... wondering if it's more, when the to() gets a response, that it would acknowledge the message (removing from the queue, allowing another thing to be handled), even though it's not quite done of the message?

whikloj · 2021-12-07T21:37:10Z

Yeah I think that is why they don't guarantee the order if you use asyncConsumers. If you have a route like

from("input")
.to("processing")
.to("output");

and queue with

A, B, C

with a single consumer but asyncConsumers=true then depending on how long each thing takes in processing the output endpoint could see A, C, B, C, A, B or C, B, A...

* Fix up warning about the unclosed app context. * Slap together async-consumer stuff. * Add separator for asyncConsumer parameter. * Update example as suggested.

adam-vessey · 2021-12-08T14:53:19Z

Since... I've now got code in this, I'm not really supposed to merge it, so... yeah... but I would if I could. :P

whikloj · 2021-12-08T15:18:39Z

Yeah we'll have to drag some other @Islandora/8-x-committers in to this

elizoller · 2021-12-08T15:25:43Z

@whikloj / @adam-vessey - do you want me to give it another round of smoke tests?

jordandukart · 2021-12-08T15:28:46Z

Great job @whikloj and @adam-vessey for working through this!

elizoller · 2021-12-08T15:52:21Z

sounds like @alxp already got to it here: Islandora-Devops/islandora-playbook#208 (comment)

whikloj · 2021-12-08T21:54:40Z

@elizoller if you're able to have a look, please do.

alxp

Just one question about the "solr" config comment.

alxp · 2021-12-09T15:21:40Z

example.properties

+connection.timeout=-1
+socket.timeout=-1
+
+# Solr indexer options


Should this be "Fedora indexer options"?

alxp · 2021-12-09T15:29:35Z

islandora-alpaca-app/src/main/java/ca/islandora/alpaca/driver/AlpacaDriver.java

+    public static void main(final String[] args) {
+        final AlpacaDriver driver = new AlpacaDriver();
+        final CommandLine cmd = new CommandLine(driver);
+        //cmd.setExecutionExceptionHandler(new AppExceptionHandler(driver));


Commented-out code should usually be removed.

seth-shaw-unlv · 2021-12-09T21:48:20Z

@alxp , do you want to merge it? Or are you blocked due to codecov?

whikloj · 2021-12-10T17:10:29Z

Thank you all for seeing this through. I'm going to try and release it, see how that goes.

whikloj added 7 commits October 14, 2021 19:35

Make a single jar with configurable services

521922d

Add example properties

f06de3f

Remove old property files, fix license on classes copied from Fedora

8c6f97a

More fixes to README

ba53a51

Fixes from tests not working

dbb615f

Logging for custom http client settings

864d34e

Alter README

c99968e

Alter Github Actions to use Java 11

d92362f

adam-vessey reviewed Oct 20, 2021

View reviewed changes

README.md Show resolved Hide resolved

whikloj added 2 commits October 21, 2021 09:42

Make concurrent consumers configurable per service

fc9ea36

Add disableStreamCache=true option to http calls

0c25018

seth-shaw-unlv reviewed Oct 27, 2021

View reviewed changes

islandora-indexing-fcrepo/src/main/java/ca/islandora/alpaca/indexing/fcrepo/FcrepoIndexer.java Show resolved Hide resolved

adam-vessey changed the base branch from 1.x to 2.x October 27, 2021 17:26

whikloj added 2 commits November 3, 2021 16:00

Switch back to using toD with full URL

a6069b7

Missed clean-up

61bc97b

whikloj mentioned this pull request Nov 13, 2021

Update for one-jar Alpaca Islandora-Devops/islandora-playbook#208

Merged

adam-vessey reviewed Nov 29, 2021

View reviewed changes

whikloj added 2 commits November 30, 2021 10:12

Fix PMD rule violations

5e5d11e

Fix GH actions for different branches

5f4ffcb

Ignore 2 PMD rules in specific cases

1508006

adam-vessey self-assigned this Dec 1, 2021

whikloj and others added 2 commits December 3, 2021 10:38

Correctly reference the exchange

abe052a

Move to inject the CamelContext (#1)

309089b

* Change up instantiation/initialization slightly. * PostConstruct is allowed to apply to private methods... ... seems like a false-positive in PMD? Got example exception from: https://stackoverflow.com/a/48679770 * Changing to camel's 3.7.6 LTS.

Async consumers for multiprocessing (#2)

a57a2dc

* Fix up warning about the unclosed app context. * Slap together async-consumer stuff. * Add separator for asyncConsumer parameter. * Update example as suggested.

adam-vessey approved these changes Dec 8, 2021

View reviewed changes

adam-vessey removed their assignment Dec 8, 2021

kstapelfeldt assigned adam-vessey and whikloj Dec 8, 2021

alxp requested changes Dec 9, 2021

View reviewed changes

Code review

2b5e23d

alxp approved these changes Dec 9, 2021

View reviewed changes

alxp merged commit 7bf983d into Islandora:2.x Dec 9, 2021

whikloj deleted the one-app branch December 9, 2021 22:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

One app to rule them all #82

One app to rule them all #82

whikloj commented Oct 19, 2021 •

edited

Loading

codecov bot commented Oct 19, 2021 •

edited

Loading

elizoller commented Oct 20, 2021

whikloj commented Oct 20, 2021

whikloj commented Oct 22, 2021

elizoller commented Oct 26, 2021

whikloj commented Oct 27, 2021

seth-shaw-unlv commented Oct 27, 2021

seth-shaw-unlv commented Oct 27, 2021

whikloj commented Nov 3, 2021

adam-vessey left a comment

DonRichards commented Nov 29, 2021

whikloj commented Nov 30, 2021

adam-vessey commented Dec 6, 2021

whikloj commented Dec 6, 2021

adam-vessey commented Dec 7, 2021

whikloj commented Dec 7, 2021

adam-vessey commented Dec 7, 2021

whikloj commented Dec 7, 2021

adam-vessey commented Dec 8, 2021

whikloj commented Dec 8, 2021

elizoller commented Dec 8, 2021

jordandukart commented Dec 8, 2021

elizoller commented Dec 8, 2021

whikloj commented Dec 8, 2021

alxp left a comment

alxp Dec 9, 2021

whikloj Dec 9, 2021

alxp Dec 9, 2021

seth-shaw-unlv commented Dec 9, 2021

whikloj commented Dec 10, 2021

One app to rule them all #82

One app to rule them all #82

Conversation

whikloj commented Oct 19, 2021 • edited Loading

What does this Pull Request do?

What's new?

How should this be tested?

Additional Notes:

Interested parties

codecov bot commented Oct 19, 2021 • edited Loading

Codecov Report

elizoller commented Oct 20, 2021

whikloj commented Oct 20, 2021

whikloj commented Oct 22, 2021

elizoller commented Oct 26, 2021

whikloj commented Oct 27, 2021

seth-shaw-unlv commented Oct 27, 2021

seth-shaw-unlv commented Oct 27, 2021

whikloj commented Nov 3, 2021

adam-vessey left a comment

Choose a reason for hiding this comment

DonRichards commented Nov 29, 2021

whikloj commented Nov 30, 2021

adam-vessey commented Dec 6, 2021

whikloj commented Dec 6, 2021

adam-vessey commented Dec 7, 2021

whikloj commented Dec 7, 2021

adam-vessey commented Dec 7, 2021

whikloj commented Dec 7, 2021

adam-vessey commented Dec 8, 2021

whikloj commented Dec 8, 2021

elizoller commented Dec 8, 2021

jordandukart commented Dec 8, 2021

elizoller commented Dec 8, 2021

whikloj commented Dec 8, 2021

alxp left a comment

Choose a reason for hiding this comment

alxp Dec 9, 2021

Choose a reason for hiding this comment

whikloj Dec 9, 2021

Choose a reason for hiding this comment

alxp Dec 9, 2021

Choose a reason for hiding this comment

seth-shaw-unlv commented Dec 9, 2021

whikloj commented Dec 10, 2021

whikloj commented Oct 19, 2021 •

edited

Loading

codecov bot commented Oct 19, 2021 •

edited

Loading