Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding the PropertyNamingStrategies from jackson for fixing hadoop ingestion #14671

Merged
merged 1 commit into from
Aug 1, 2023

Conversation

pranavbhole
Copy link
Contributor

…gestion

Fixes #XXXX.

Description

We found that aws jdk core is using the jackson version 2.12.7 and it is not compatible with older jackson esp inside hadoop container as it loads the classes from given extension. We found that s3 extension is loading S3 required jars and instantiating the PropertyNamingStrategies from newer jackson and failing.
Creating the PR to pull this class into druid repo itself to fix the hadoop ingestion and make it compatible with newer jackson.

Fixed the bug ...

Renamed the class ...

Added a forbidden-apis entry ...

Release note


Key changed/added classes in this PR
  • MyFoo
  • OurBar
  • TheirBaz

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@abhishekagarwal87
Copy link
Contributor

Just to add a bit more colour here.

Hadoop has support for classloader isolation that follows a child-first class loading mechanism. There are jackson classes present on Hadoop server and there are jackson classes present in the libs that druid uploads. The aws sdk that is part of druid lib, wants to load PropertyNamingStrategies but it is not available in the child classloader so that class gets loaded through the parent classloader. The parent classloader would also load the PropertyNamingStrategy referenced by the PropertyNamingStrategies. All is well so far. But then the child classloader needs to load PropertyNamingStrategy later which it is able to load. But because it has also been loaded through the parent classloader, we get a linkage error. Somewhat explained here - http://frankkieviet.blogspot.com/2009/03/javalanglinkageerror-loader-constraint.html

@abhishekagarwal87
Copy link
Contributor

@pranavbhole - code inspection jobs are failing.

@pranavbhole pranavbhole force-pushed the jacksonBackwardComp branch 4 times, most recently from 6ac3d23 to 922388a Compare August 1, 2023 01:20
@pranavbhole
Copy link
Contributor Author

@abhishekagarwal87 pushed the fix.

@abhishekagarwal87 abhishekagarwal87 merged commit 8a10b46 into apache:master Aug 1, 2023
74 checks passed
@LakshSingla LakshSingla added this to the 28.0 milestone Oct 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants