Bringing back jar-dependencies to logstash, including offline support for it.

[purbon]

Hi,
in an effort of having a good citizenship in the gems world, embedding jars the way we do it nowadays in logstash, and has been done for long time in jruby projects is not optimal. To do this work there is a set of tools like https://github.com/mkristian/jar-dependencies mantained by Herr @mkristian.

In LS we removed the usage of jar-dependencies because our way of using it was not providing us with all the necessary features we aim for, we should be sure managed dependencies are:

• Downloaded and setup during a plugin installation, this could happen in many situations (for example with proxies, etc)
• Enable offline installation of plugins.
• Add test in logstash to be sure jars are packaged in the plugins when the gem is build/installed/etc..

[colinsurprenant]

I think at this point we have to evaluate the benefits vs the costs of trying to solve this. In particular for offline installation, I do not see how it will be possible to package plugins that depends on jars, via jar-dependencies (and Maven, ...), for self-contained offline installation. We also have to keep in perspective that only a handful of plugins depends on jars, so I am not sure if it is really worth investing much time on this at this point.

Also, other than the size issue of including a jar into a gem, which from a user-perspective has zero impact, our plugins systems allows up yo very easily update a jar version if needed and release a new plugin version.

Am I missing somethings? Thoughts?

[purbon]

@colinsurprenant I'm starting to think and work on this, and as I spoke to @suyograo this would be my five cents back to the jruby community. I will keep digging on ways to get this working for us.

I think the idea of having jars into gems is a necessary pain to suffer, so enabling the option to fetch and and work as expected is very nice feature to have in jruby projects.

[colinsurprenant]

@purbon sure but what about offline? There might be a solution but I am not sure I see a practical/pragmatic one, but that doesn't one does not exits :P

[purbon]

I don't know and I need to investigate before making my mind.

[mkristian]

I mentioned on an other issue how offline packaging could work with jars as packaging container. they can have the embedded gem and its jars. to use such a jar you just need to require it.

for example jruby-openssl does use jar-dependencies but comes with the jars packed inside the gem. here jar-dependencies just ensures that only one version of jar gets loaded into the JRubyClassLoader (via require_jar). this means the overall application still has control over the which version of jars used or excluded or replaced.

the trigger is that jar-dependencies is just development dependency of the gem then during gem installation it acts as any other gem - no maven, etc

of what I know so far from logstash and the problems here mentioned, this seems to be a nice middle path. use jar-dependencies but pack the jars.

[purbon]

thanks for bringing this back in this issue @mkristian , as we spoke, I'm going to be investigating this proposals. Thanks a lot for your help during the process :P expect many questions coming to you, xD!

[colinsurprenant]

@mkristian so basically what you are saying is that like with jruby-openssl, jar-dependencies is only used to fetch the dependant jars at dev time but when packaging the gem, the jars are actually packed/vendored in the gem. The benefit of doing this will be to control the dependant jars versions through the gemspec (and have a rake task to actually pack/vendor the jar in the gem for when releasing the gem), right?

[mkristian]

this is right. the benefit of tracking the jar can be easier explained the other way around: https://github.com/guyboertje/jrjackson comes with some jackson jars. no clue which version and no way to use others. the leafy gems https://github.com/lookout/leafy uses dropwizard which will pick some version of those jacksons jars. when you use both gems then you will have probably two different versions of jackson in your jruby-classloader.

[purbon]

Hi,
This is what I was testing yesterday afternoon with the log4j input [1],
for now it could be a solution.

More thoughts and research to come.

/purbon

[1] logstash-plugins/logstash-input-log4j#14

On Sat, 15 Aug 2015 09:55 Christian Meier notifications@github.com wrote:

this is right. the benefit of tracking the jar can be easier explained the
other way around: https://github.com/guyboertje/jrjackson comes with some
jackson jars. no clue which version and no way to use others. the leafy
gems https://github.com/lookout/leafy uses dropwizard which will pick some
version of those jacksons jars. when you use both gems then you will have
probably two different versions of jackson in your jruby-classloader.

—
Reply to this email directly or view it on GitHub
#3735 (comment).

[colinsurprenant]

@mkristian if two "conflicting" java packages are required by 2 separate gems that use jar-dependencies, what will happen? does jar-dependencies create separate class loaders for both java classes and allow the use of both versions side-by-side?

[mkristian]

@colinsurprenant there is no way to solve this on the classloader level. the first jar wins and the second produces a warning. similar to what is done when you want to activate a gem which is already activated with a different version. so the idea is to keep the JRubyClassLoader clean in the sense that you have only one version of the jar loaded. there are more issues with embedded JRuby where you can inherit all kind of jars via parent classloader of jruby which can "overwrite" the jars which you want to require from your ruby code (this is true with and without jar-dependencies).

there is way to pick the version you want to use via file called Jars.lock. this is what the jruby-gradle-plugins uses all over the place to tell jruby which jars the gradle dependency resolver decided to use. or jbundler works in a similar way also via jar-dependencies. this Jars.lock basically bypasses the require_jar. with this you can for example pick another bouncy-castle version for jruby-openssl other the embedded one.

[purbon]

Cool, all this provide lots of interesting information.

[colinsurprenant]

@mkristian ok, makes sense. thanks for the info! so I think the hybrid approach using jar-dependencies for dev time and packaging the jars in the gem at release seems a good compromise.

this also relates to this jar conflicts issue #3153

[colinsurprenant]

pulling @guyboertje to sync on jar-dependencies integration in JrJackson

[purbon]

FYI: my current idea, is to follow for now @mkristian recommendation and use jar-dependencies to pull the jars and get the gem with them. Need to work on having a clear way to check if actually the jars where installed when building the gem, or raise an error (proxy, firewall, network issue, etc)

This would be my line of work for now. What do you think?

[mkristian]

@guyboertje took the "freedom" to add some more hints to guyboertje/jrjackson#35 (comment) and please feel free to ask in case you have problems.

[purbon]

Two PR bringing back jar-dependencies has been proposed:

• Bring back the jar-dependencies  logstash-plugins/logstash-output-elasticsearch#238
• Bringing back jar dependencies as managing tool for jars in LS logstash-plugins/logstash-input-log4j#14

What this basically does is to pull the jars, and pack them in the gem at build time, so like this the repo has less size and is more manageable, plus the gem has the jars and offline installation is also working.

Review and comments there are welcome.

/cc @colinsurprenant @ph @jsvd @suyograo

[purbon]

As reported by @andrewvc and one of the problems we found earlier with jar-dependencies, there is a problem when fetching pre releases, like the beta from elasticsearch 2.0 for example. I guess a problem we should fix before merging jar-dependencies in as we might like to use this dependencies, even if just for testing purposes.

Relates to: mkristian/jar-dependencies#26

[ph]

I think we all agree here that we are using jar-dependencies as a way to declare the dependencies in the gemspec and as a way to create the gem with the bundled artifact.

We know have pretty good integration test to make sure the needed jars are included in the most important plugins.

I will close this since there is no more actionable items to do.