Create a camel 'Pipeline' #449
Comments
|
FYI, this is how we solve the "what derivatives go where" conundrum for everybody. Making a derivative would become a list of operations, and the user could configure the last one to ingest into Fedora, Drupal, or both. We're gonna get some insane flexibility from this. |
|
Couple confirmation questions:
|
|
This seems like it's entirely overlapping with |
|
@ajs6f I think this is far too simple to be part of |
|
I'm not asking about this exact piece of technology-- I'm asking about its uses. IOW, it's hard for me to imagine a use for this piece of work that wouldn't be better served as part of |
@ajs6f Yes, the end result here could be used to create and store derivatives, or any other sequence of well defined asynchronous operations. The long term goal for me (post MVP) is to provide config entities per asynchronous operation with nice descriptions and labels. That way we can work with them in a UI so repository admins can control their asynchronous workflows outside of karaf config files. If you think this core piece would be better suited for fcrepo-camel-toolbox, that's a conversation I'll gladly have. |
|
Yeah, that's where I was going, and to be clear, I'm not claiming that this is a sure thing for |
|
@ajs6f At the end of the day, this is nothing more than a question of where do we deposit the work. I'm not expecting the greater community to do that work for us. |
|
Sure, and where you deposit the work has a significant impact on how expensive it is and its future maintenance costs. This ticket might be developing new Camel gear, or it might be merely configuring Camel gear the maintenance of which is a shared expense. It's worth asking that question. I think we might easily overestimating the novelty of this need. |
|
@ajs6f Fair enough. |
|
And having shoved my neck out, I'll bring the axe down by taking this ticket through the exploratory stage that I just vociferously demanded. I'll take responsibility for talking with @acoburn and determining what, if any, we can get done as part of the larger |
|
I just don't want to basically come in here and say @whikloj, I demand that you work this ticket in the way that I think is best. That's not cool. |
|
Let me preface this by saying that I haven't thought this through in any sort of rigorous manner, but let me argue the other side for now, namely that such a pipeline should not go into fcrepo-camel-toolbox.
|
|
@acoburn, this all makes sense-- but it's exactly your third point/recommendation that is what I am suggesting here. I'm suggesting that either we find the generic pattern in Does that makes sense to you, @acoburn, or am I missing your point? Thanks for checking in with this discussion! |
|
@ajs6f if your suggestion here follows what I describe in the third point, then, yes, I would completely support that. |
|
@acoburn I think that it does. The crucial question is: can we extract the generic pattern in this ticket as a contribution to "Configure a chained series of simple (i.e. one Camel endpoint) steps, each feeding the next. This pipeline is fed by resources that change in the repository as they change, in the usual @dannylamb @whikloj Does that sound like it captures a generic form of what you are trying to do here? Now that I read it back, it sounds almost like a generic version of several of the |
|
To go through @acoburn's points:
@ajs6f You should also see https://github.com/Islandora-CLAW/Alpaca/tree/master/islandora-connector-broadcast and https://gitlab.amherst.edu/acdc/repository-extension-services/tree/master/acrepo-connector-broadcast, which are pretty similar and differ in the manner described above. |
|
@ajs6f To more directly answer the question you posed, that is pretty much exactly the long-term goal, except there's folks in the islandora audience who will want the derivative stored just in drupal, and not in fedora. That usually comes from people with large binaries like video. So that's a curveball from our end. But then again, if we can just layer in our step at the end, or swap something out, etc... |
|
Here's a suggestion for moving forward, which is pretty much what @bseeger and I do at Amherst:
That has been the process for much of what is currently in fcrepo-camel-toolbox. |
|
@acoburn Yes, that's the concrete MO. The reason I wanted to start this discussion earlier is to clarify whether there might be such a nugget. It sounds like there is, so whoever is doing this ticket should have that in mind. First things first, the functionality has to work for CLAW, but it would be good to keep the longer term in mind. |
|
I am going to un-assign myself from this. I think there are others that could benefit from this small well defined need for some Camel. @Natkeeran?? Also based on @acoburn's statement above, I think we go ahead and implement this for CLAW and if we discover some generalities we can open a discussion at that point. |
|
I am trying out your POC. While implementing it in Java DSL, I am getting the following exception. Any suggestion as to the causes: |
|
@Natkeeran Not sure if this is helpful at all, but here is a Java DSL camel route I |
|
@Natkeeran I'd try and set this up using blueprint within Alpaca and its build process. You may be down a sinkhole you wouldn't encounter in that context. If you 'd like, I can help you get that set up with a feature and all. |
|
And just after posting that I see the problem: |
|
@Natkeeran Offer still stands if you want help getting that set up all the way with a feature. Just tag me. |
|
@Natkeeran I've done some explorations after our last few rounds of discussion, and I think we can merge the broadcaster and pipeline code if we make the exchange pattern come from a header. Tag me or skype me if you want to discuss this. Basically just take the work you've already done and replace the broadcaster with it, hacking out the ExchangePattern (InOnly, InOut) from a header. |
|
Resolved with Islandora/Alpaca@558ef69 Additional changes to be made in a subsequent ticket. |
Similar to #448
Create a small camel route that sends a message through a dynamic list of recipients, with the output of each being passed as input to the next. This can be accomplished using the Routing Slip and Request / Reply patterns. The list should be read from a header, which you can assume comes with the message.
Check out this gist for a POC of this concept.
The text was updated successfully, but these errors were encountered: