[Java] Remove Netty references from ArrowBuf and move Allocator out of vector package

[asfimport]

Arrow currently has a hard dependency on Netty and exposes this in public APIs. This shouldn't be the case. There could be many allocator implementations with Netty as one possible option. We should remove hard dependency between arrow-vector and Netty, instead creating a trivial allocator. ArrowBuf should probably expose an T unwrap(Class clazz) method instead to allow inner providers availability without a hard reference. This should also include drastically reducing the number of methods on ArrowBuf as right now it includes every method from ByteBuf but many of those are not very useful, appropriate.

This work should come after we do the simpler ARROW-3191

Reporter: Jacques Nadeau / @jacques-n
Assignee: Liya Fan / @liyafan82

Related issues:

• [Java] Add support for ArrowBuf to point to arbitrary memory. (is blocked by)
• [Java] Separate Netty Memory to its own module (is a parent of)
• [Java] Remove Netty dependency for ArrowBuf (is a parent of)
• [Java] Move ArrowBuf into the Arrow package (is a parent of)
• [Java] Move Netty memory manager into a separate module (is a parent of)
• [Java] Remove Netty dependency for BufferAllocator and ReferenceManager (is a parent of)
• [Java] Remove Netty dependency from arrow-vector (relates to)

_{Note: This issue was originally created as ARROW-4526. Please see the migration documentation for further details.}

[asfimport]

Jacques Nadeau / @jacques-n:
As part of this, ArrowBuf should move into the Arrow packages from the Netty packages

[asfimport]

Jacques Nadeau / @jacques-n:
[~fan_li_ya], can you do this as a series of separate PRs so it is more manageable to review?

Here is a suggested set of PRs:

1. Remove all references to Netty from ArrowBuf, BufferAllocator, ReferenceManager.

2. Move Netty memory manager into a separate module such that the basic allocator does not depend on it.

3. Move ArrowBuf into the Arrow package.

   Some other smaller steps may also make sense as you start working on it.

[asfimport]

Liya Fan / @liyafan82:
@jacques-n Sure. Thanks for the suggestion. The steps are reasonable.

[asfimport]

Liya Fan / @liyafan82:
For step 1, I think we can first create a issue that remove Netty references for ArrowBuf.
The task for BufferAllocator and ReferenceManager should be started after ARROW-7329 is finished.

[asfimport]

Ryan Murray / @rymurr:
Hey [~fan_li_ya] what is the status of this ticket now that ARROW-7329 is finished. I have some time to help if there is anything I can do.

[asfimport]

Liya Fan / @liyafan82:
@rymurr Thanks for your time.
Of the 3 work items listed above, we have finished item 1 in ARROW-7935 and ARROW-7935.
We are now working on item 3 (ARROW-8229, a PR will be submitted soon).

So if you are interested, maybe you can take item 2.
I have created an issue (ARROW-8230), and please feel free to assign it to yourself, if you are interested :)

[asfimport]

Ryan Murray / @rymurr:
Thanks! I will take a look :-)

[asfimport]

Bryan Cutler / @BryanCutler:
[~fan_li_ya] I added ARROW-9300 as a sub-task. After that what is the remaining work to be done, just remove the netty dependency from arrow-vector? Do you think this feasible to finish for 1.0.0?

[asfimport]

Liya Fan / @liyafan82:
@BryanCutler IMO, the major part of the work has been done. So I think it is feasible to finish before 1.0.0. @rymurr do you have any comments?

[asfimport]

Ryan Murray / @rymurr:
I agree [~fan_li_ya] the PR for ARROW-9300 should be the last piece and it should be green now. (its 95% moving files and updating poms so should be uncontroversial)

[asfimport]

Bryan Cutler / @BryanCutler:
Thanks [~fan_li_ya] and @rymurr , so is one of the goals of this Jira to remove netty as a required dependency of arrow-vector? It seems that ARROW-9300 still leaves netty-common as a dependency, but I'm not sure what references it.  Also, I'm wondering if the netty allocator is in a different module, and it's only a dependency to arrow-vector in test scope, is it no longer the default allocator?

[asfimport]

Liya Fan / @liyafan82:
@BryanCutler I agree with you that the ultimate goal is to remove netty as a required dependency.

For the test cases to work, we must reference a concrete allocation manager implementation (either netty or unsafe). Currently, we are still referencing the netty implementation, because it has always been the default option. I think we will switch to the unsafe allocation manager some time in the future, when the timing is right.

BTW, the reference is done implicitly through refection, to avoid the problem of cyclic dependency.

[asfimport]

Ryan Murray / @rymurr:
I think the important part (for me) is to remove netty from the public API for v1.0.0. There will be some cleanup in the background to completely remove netty if we choose to do so and this can happen post 1.0.0.

[asfimport]

Bryan Cutler / @BryanCutler:
I'll elevate ARROW-9300 to a blocker for now because this is something that would be good to have squared away before 1.0.0, and get it up on the radar. I also added an issue for the removal of Netty as a dependency, and I think we should provide some end-user documentation on the changes to the allocators. 

[asfimport]

Wes McKinney / @wesm:
I removed this from 1.0.0