Allow presto to connect to kerberized Hive clusters (v2)

[losipiuk]

No description provided.

[losipiuk]

I checked connectivity to Hive metastore with SASL over socks proxy and it works.

[electrum]

This should probably be categorized

[arhimondr]

IOException could be thrown only for tocken authentication from Credentials.readTokenStorageFile. As we are not using that type of authentication i don't think that it make sense to add special exception code for that particular issue.

[arhimondr]

@electrum rebased onto actual master + comments addressed.

[losipiuk]

When we benchmarked the solution we noticed great performance degradation when authenticating wrappers are in use. Simple queries on ORC table were executing 3x slower than without wrappers.

Solution 1

We have a working solution for that but it is somewhat hacky. It depends on patching UserGroupInformation in hadoop library.

These are PR for 3 versions of hadoop lib:

• Faster authentication in UserGroupInformation presto-hadoop-apache1#4
• Faster authentication in UserGroupInformation presto-hadoop-cdh4#12
• Faster authentication in UserGroupInformation presto-hadoop-apache2#10

The source of performance degradation is fact that UserGroupInformation.doAs(PrivilegedAction) is a delegate to Subject.doAs(PrivilegedAction) from java.security, which is slow.

The change we made exploits the fact that call to actual Subject.doAs is rarely needed, when compared to number of method calls we wrap into UGI.doAs(PrivilegedAction) in Presto.
We also depend on the fact, that when call to Subject.doAs(PrivilegedAction) is actually needed, th Hadoop lib rewraps the code which needs that again into UGI.doAs(PrivilegedAction).

The change we made is basically extending UserGroupInformation class with faster versions of doAs methods taking (SubjectPrivilegedAction and SubjectPrivilegedExceptionAction).
We also change the original doAs(PrivilegedAction) and doAs(PrivilegedExceptionAction) and getCurrentUser() to interact with logged-user context set by the faster method versions.

In Presto's authenticating wrappers we use faster versions of UGI.doAs methods.

See this commit for actual diff: prestodb/presto-hadoop-apache2@240a81d

This is not a clean solution, as it patches external library and depends on brittle code flow assumptions.
It is very effective in terms of performance though.

Solution 2

After implementing this we also got another idea. It does not involve patching Hadoop lib but have other drawbacks.

Let's focus on ConnectorPageSource wrapper, as this is culprit for read-flow. The solution has two parts:

1. Use UGI.doAs wrappers only in getNextPage() and close(). This assumes that other methods do not need authentication. This is probably true for most implementation but generally it does not have to be the case. We can extend the mechanism so ConnectorPageSource implementation can somehow declare (through annotation?) which methods need to use authentication.
2. the wrapper for getNextPage() has buffer of pages and uses UGI.doAs only when buffer of pages is empty. The buffer is filled with multiple pages (order of 50) within single UGI.doAs wrap block. This way we amortize the cost of single doAs over multiple calls to CPS.getNextPage().

Similar mechanism can be implemented for ConnectorPageSink. RecordCursor is problematic and probably we would have to skip special implementation of RecordPageSource with this approach.

The drawback of this approach is that we are using more memory (buffer of pages), and we process data in less streamlined manner.
Also in case when other methods than getNextPage actually need authorization we have a problem again.

@electrum, @dain. What do you think about implemented solution? We would really appreciate input here. Any idea how to solve the problem here in a cleaner way would be very beneficial.

cc: @arhimondr, @pnowojski, @ilfrin

[dain]

the wrapper for getNextPage() has buffer of pages and uses UGI.doAs only when buffer of pages is empty. The buffer is filled with multiple pages (order of 50) within single UGI.doAs wrap block. This way we amortize the cost of single doAs over multiple calls to CPS.getNextPage().

This won't work because we return lazy blocks. With ORC, we need to know which column values are needed before the next page is fetched, because when fetching a page we advance all streams.

In general, I think the solution to this is that we pass the UGI to the HivePageSourceFactory or HiveRecordCursorProvider during construction and require the implementation to handle the UGI. Except for a few obscure file formats, the file format code opens a file input stream during creation and never goes back to the file system to open new streams. We can then add special handling for these formats as we find them. (BTW this is the same approach we take with the thread context class loader). With this approach we would remove the UGI setting wrappers for page source and record cursor.

[dain]

The UGI should be explicitly passed in during the construction, so we should not need thread tricks here.

[dain]

IIRC, when we talked, we decided that we were going to do more work on this. Let me know when that is done and I'll take another look.

[arhimondr]

@dain Yes, that is right.

Superseeded by: #4867

@losipiuk Please close this PR.