HBASE-23347 Allowable custom authentication methods for RPCs

dev-support/design-docs/HBASE-23347-pluggable-authentication.md

@@ -0,0 +1,179 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Pluggable Authentication for HBase RPCs
+
+## Background
+
+As a distributed database, HBase must be able to authenticate users and HBase
+services across an untrusted network. Clients and HBase services are treated
+equivalently in terms of authentication (and this is the only time we will
+draw such a distinction).
+
+There are currently three modes of authentication which are supported by HBase
+today via the configuration property `hbase.security.authentication`
+
+1. `SIMPLE`
+2. `KERBEROS`
+3. `TOKEN`
+
+`SIMPLE` authentication is effectively no authentication; HBase assumes the user
+is who they claim to be. `KERBEROS` authenticates clients via the KerberosV5
+protocol using the GSSAPI mechanism of the Java Simple Authentication and Security
+Layer (SASL) protocol. `TOKEN` is a username-password based authentication protocol
+which uses short-lived passwords that can only be obtained via a `KERBEROS` authenticated
+request. `TOKEN` authentication is synonymous with Hadoop-style [Delegation Tokens](https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/hadoop_tokens.html#delegation-tokens). `TOKEN` authentication uses the `DIGEST-MD5`
+SASL mechanism.
+
+[SASL](https://docs.oracle.com/javase/8/docs/technotes/guides/security/sasl/sasl-refguide.html)
+is a library which specifies a network protocol that can authenticate a client
+and a server using an arbitrary mechanism. SASL ships with a [number of mechanisms](https://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml)
+out of the box and it is possible to implement custom mechanisms. SASL is effectively
+decoupling an RPC client-server model from the mechanism used to authenticate those
+requests (e.g. the RPC code is identical whether username-password, Kerberos, or any
+other method is used to authenticate the request).
+
+RFC's define what [SASL mechanisms exist](https://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xml),
+but what RFC's define are a superset of the mechanisms that are
+[implemented in Java](https://docs.oracle.com/javase/8/docs/technotes/guides/security/sasl/sasl-refguide.html#SUN).
+This document limits discussion to SASL mechanisms in the abstract, focusing on those which are well-defined and
+implemented in Java today by the JDK itself. However, it is completely possible that a developer can implement
+and register their own SASL mechanism. Writing a custom mechanism is outside of the scope of this document, but
+not outside of the realm of possibility.
+
+The `SIMPLE` implementation does not use SASL, but instead has its own RPC logic
+built into the HBase RPC protocol. `KERBEROS` and `TOKEN` both use SASL to authenticate,
+relying on the `Token` interface that is intertwined with the Hadoop `UserGroupInformation`
+class. SASL decouples an RPC from the mechanism used to authenticate that request.
+
+## Problem statement
+
+Despite HBase already shipping authentication implementations which leverage SASL,
+it is (effectively) impossible to add a new authentication implementation to HBase. The
+use of the `org.apache.hadoop.hbase.security.AuthMethod` enum makes it impossible
+to define a new method of authentication. Also, the RPC implementation is written
+to only use the methods that are expressly shipped in HBase. Adding a new authentication
+method would require copying and modifying the RpcClient implementation, in addition
+to modifying the RpcServer to invoke the correct authentication check.
+
+While it is possible to add a new authentication method to HBase, it cannot be done
+cleanly or sustainably. This is what is meant by "impossible".
+
+## Proposal
+
+HBase should expose interfaces which allow for pluggable authentication mechanisms
+such that HBase can authenticate against external systems. Because the RPC implementation
+can already support SASL, HBase can standardize on SASL, allowing any authentication method
+which is capable of using SASL to negotiate authentication. `KERBEROS` and `TOKEN` methods
+will naturally fit into these new interfaces, but `SIMPLE` authentication will not (see the following
+chapter for a tangent on SIMPLE authentication today)
+
+### Tangent: on SIMPLE authentication
+
+`SIMPLE` authentication in HBase today is treated as a special case. My impression is that
+this stems from HBase not originally shipping an RPC solution that had any authentication.
+
+Re-implementing `SIMPLE` authentication such that it also flows through SASL (e.g. via
+the `PLAIN` SASL mechanism) would simplify the HBase codebase such that all authentication
+occurs via SASL. This was not done for the initial implementation to reduce the scope
+of the changeset. Changing `SIMPLE` authentication to use SASL may result in some
+performance impact in setting up a new RPC. The same conditional logic to determine
+`if (sasl) ... else SIMPLE` logic is propagated in this implementation.
+
+## Implementation Overview
+
+HBASE-23347 includes a refactoring of HBase RPC authentication where all current methods
+are ported to a new set of interfaces, and all RPC implementations are updated to use
+the new interfaces. In the spirit of SASL, the expectation is that users can provide
+their own authentication methods at runtime, and HBase should be capable of negotiating
+a client who tries to authenticate via that custom authentication method. The implementation
+refers to this "bundle" of client and server logic as an "authentication provider".
+
+### Providers
+
+One authentication provider includes the following pieces:
+
+1. Client-side logic (providing a credential)
+2. Server-side logic (validating a credential from a client)
+3. Client selection logic to choose a provider (from many that may be available)
+
+A provider's client and server side logic are considered to be one-to-one. A `Foo` client-side provider
+should never be used to authenticate against a `Bar` server-side provider.
+
+We do expect that both clients and servers will have access to multiple providers. A server may
+be capable of authenticating via methods which a client is unaware of. A client may attempt to authenticate
+against a server which the server does not know how to process. In both cases, the RPC
+should fail when a client and server do not have matching providers. The server identifies
+client authentication mechanisms via a `byte authCode` (which is already sent today with HBase RPCs).
+
+A client may also have multiple providers available for it to use in authenticating against
+HBase. The client must have some logic to select which provider to use. Because we are
+allowing custom providers, we must also allow a custom selection logic such that the
+correct provider can be chosen. This is a formalization of the logic already present
+in `org.apache.hadoop.hbase.security.token.AuthenticationTokenSelector`.
+
+To enable the above, we have some new interfaces to support the user extensibility:
+
+1. `interface SaslAuthenticationProvider`
+2. `interface SaslClientAuthenticationProvider extends SaslAuthenticationProvider`
+3. `interface SaslServerAuthenticationProvider extends SaslAuthenticationProvider`
+4. `interface AuthenticationProviderSelector`
+
+The `SaslAuthenticationProvider` shares logic which is common to the client and the
+server (though, this is up to the developer to guarantee this). The client and server
+interfaces each have logic specific to the HBase RPC client and HBase RPC server
+codebase, as their name implies. As described above, an implementation
+of one `SaslClientAuthenticationProvider` must match exactly one implementation of
+`SaslServerAuthenticationProvider`. Each Authentication Provider implementation is
+a singleton and is intended to be shared across all RPCs. A provider selector is
+chosen per client based on that client's configuration.
+
+A client authentication provider is uniquely identified among other providers
+by the following characteristics:
+
+1. A name, e.g. "KERBEROS", "TOKEN"
+2. A byte (a value between 0 and 255)
+
+In addition to these attributes, a provider also must define the following attributes:
+
+3. The SASL mechanism being used.
+4. The Hadoop AuthenticationMethod, e.g. "TOKEN", "KERBEROS", "CERTIFICATE"
+5. The Token "kind", the name used to identify a TokenIdentifier, e.g. `HBASE_AUTH_TOKEN`
+
+It is allowed (even expected) that there may be multiple providers that use `TOKEN` authentication.
+
+N.b. Hadoop requires all `TokenIdentifier` implements to have a no-args constructor and a `ServiceLoader`
+entry in their packaging JAR file (e.g. `META-INF/services/org.apache.hadoop.security.token.TokenIdentifier`).
+Otherwise, parsing the `TokenIdentifier` on the server-side end of an RPC from a Hadoop `Token` will return
+`null` to the caller (often, in the `CallbackHandler` implementation).
+
+### Factories
+
+To ease development with these unknown set of providers, there are two classes which
+find, instantiate, and cache the provider singletons.
+
+1. Client side: `class SaslClientAuthenticationProviders`
+2. Server side: `class SaslServerAuthenticationProviders`
+
+These classes use [Java ServiceLoader](https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html)
+to find implementations available on the classpath. The provided HBase implementations
+for the three out-of-the-box implementations all register themselves via the `ServiceLoader`.
+
+Each class also enables providers to be added via explicit configuration in hbase-site.xml.
+This enables unit tests to define custom implementations that may be toy/naive/unsafe without
+any worry that these may be inadvertently deployed onto a production HBase cluster.

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java

@@ -41,8 +41,6 @@
 import java.net.SocketAddress;
 import java.net.UnknownHostException;
 import java.util.Collection;
-import java.util.HashMap;
-import java.util.Map;
 import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.ScheduledFuture;
@@ -58,17 +56,13 @@
 import org.apache.hadoop.hbase.client.MetricsConnection;
 import org.apache.hadoop.hbase.codec.Codec;
 import org.apache.hadoop.hbase.codec.KeyValueCodec;
-import org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos.TokenIdentifier.Kind;
 import org.apache.hadoop.hbase.security.User;
 import org.apache.hadoop.hbase.security.UserProvider;
-import org.apache.hadoop.hbase.security.token.AuthenticationTokenSelector;
 import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
 import org.apache.hadoop.hbase.util.PoolMap;
 import org.apache.hadoop.hbase.util.Threads;
 import org.apache.hadoop.io.compress.CompressionCodec;
 import org.apache.hadoop.ipc.RemoteException;
-import org.apache.hadoop.security.token.TokenIdentifier;
-import org.apache.hadoop.security.token.TokenSelector;
 
 import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
 
@@ -104,14 +98,6 @@ public abstract class AbstractRpcClient<T extends RpcConnection> implements RpcC
   private static final ScheduledExecutorService IDLE_CONN_SWEEPER = Executors
       .newScheduledThreadPool(1, Threads.newDaemonThreadFactory("Idle-Rpc-Conn-Sweeper"));
 
-  @edu.umd.cs.findbugs.annotations.SuppressWarnings(value="MS_MUTABLE_COLLECTION_PKGPROTECT",
-      justification="the rest of the system which live in the different package can use")
-  protected final static Map<Kind, TokenSelector<? extends TokenIdentifier>> TOKEN_HANDLERS = new HashMap<>();
-
-  static {
-    TOKEN_HANDLERS.put(Kind.HBASE_AUTH_TOKEN, new AuthenticationTokenSelector());
-  }
-
   protected boolean running = true; // if client runs
 
   protected final Configuration conf;

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java

@@ -54,6 +54,7 @@
 import org.apache.hadoop.hbase.security.HBaseSaslRpcClient;
 import org.apache.hadoop.hbase.security.SaslUtil;
 import org.apache.hadoop.hbase.security.SaslUtil.QualityOfProtection;
+import org.apache.hadoop.hbase.security.provider.SaslClientAuthenticationProvider;
 import org.apache.hadoop.hbase.trace.TraceUtil;
 import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
 import org.apache.hadoop.hbase.util.ExceptionUtil;
@@ -361,9 +362,10 @@ private void disposeSasl() {
 
   private boolean setupSaslConnection(final InputStream in2, final OutputStream out2)
       throws IOException {
-    saslRpcClient = new HBaseSaslRpcClient(authMethod, token, serverPrincipal,
-        this.rpcClient.fallbackAllowed, this.rpcClient.conf.get("hbase.rpc.protection",
-          QualityOfProtection.AUTHENTICATION.name().toLowerCase(Locale.ROOT)),
+    saslRpcClient = new HBaseSaslRpcClient(this.rpcClient.conf, provider, token,
+        serverAddress, securityInfo, this.rpcClient.fallbackAllowed,
+        this.rpcClient.conf.get("hbase.rpc.protection",
+            QualityOfProtection.AUTHENTICATION.name().toLowerCase(Locale.ROOT)),
         this.rpcClient.conf.getBoolean(CRYPTO_AES_ENABLED_KEY, CRYPTO_AES_ENABLED_DEFAULT));
     return saslRpcClient.saslConnect(in2, out2);
   }
@@ -375,11 +377,10 @@ private boolean setupSaslConnection(final InputStream in2, final OutputStream ou
    * connection again. The other problem is to do with ticket expiry. To handle that, a relogin is
    * attempted.
    * <p>
-   * The retry logic is governed by the {@link #shouldAuthenticateOverKrb} method. In case when the
-   * user doesn't have valid credentials, we don't need to retry (from cache or ticket). In such
-   * cases, it is prudent to throw a runtime exception when we receive a SaslException from the
-   * underlying authentication implementation, so there is no retry from other high level (for eg,
-   * HCM or HBaseAdmin).
+   * The retry logic is governed by the {@link SaslClientAuthenticationProvider#canRetry()}
+   * method. Some providers have the ability to obtain new credentials and then re-attempt to
+   * authenticate with HBase services. Other providers will continue to fail if they failed the
+   * first time -- for those, we want to fail-fast.
    * </p>
    */
   private void handleSaslConnectionFailure(final int currRetries, final int maxRetries,
@@ -389,40 +390,44 @@ private void handleSaslConnectionFailure(final int currRetries, final int maxRet
     user.doAs(new PrivilegedExceptionAction<Object>() {
       @Override
       public Object run() throws IOException, InterruptedException {
-        if (shouldAuthenticateOverKrb()) {
-          if (currRetries < maxRetries) {
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Exception encountered while connecting to " +
-                "the server : " + StringUtils.stringifyException(ex));
-            }
-            // try re-login
-            relogin();
-            disposeSasl();
-            // have granularity of milliseconds
-            // we are sleeping with the Connection lock held but since this
-            // connection instance is being used for connecting to the server
-            // in question, it is okay
-            Thread.sleep(ThreadLocalRandom.current().nextInt(reloginMaxBackoff) + 1);
-            return null;
-          } else {
-            String msg = "Couldn't setup connection for "
-                + UserGroupInformation.getLoginUser().getUserName() + " to " + serverPrincipal;
-            LOG.warn(msg, ex);
-            throw new IOException(msg, ex);
+        // A provider which failed authentication, but doesn't have the ability to relogin with
+        // some external system (e.g. username/password, the password either works or it doesn't)
+        if (!provider.canRetry()) {
+          LOG.warn("Exception encountered while connecting to the server : " + ex);
+          if (ex instanceof RemoteException) {
+            throw (RemoteException) ex;
           }
-        } else {
-          LOG.warn("Exception encountered while connecting to " + "the server : " + ex);
-        }
-        if (ex instanceof RemoteException) {
-          throw (RemoteException) ex;
+          if (ex instanceof SaslException) {
+            String msg = "SASL authentication failed."
+                + " The most likely cause is missing or invalid credentials.";
+            throw new RuntimeException(msg, ex);
+          }
+          throw new IOException(ex);
         }
-        if (ex instanceof SaslException) {
-          String msg = "SASL authentication failed."
-              + " The most likely cause is missing or invalid credentials." + " Consider 'kinit'.";
-          LOG.error(HBaseMarkers.FATAL, msg, ex);
-          throw new RuntimeException(msg, ex);
+
+        // Other providers, like kerberos, could request a new ticket from a keytab. Let
+        // them try again.
+        if (currRetries < maxRetries) {
+          LOG.debug("Exception encountered while connecting to the server", ex);
+
+          // Invoke the provider to perform the relogin
+          provider.relogin();
+
+          // Get rid of any old state on the SaslClient
+          disposeSasl();
+
+          // have granularity of milliseconds
+          // we are sleeping with the Connection lock held but since this
+          // connection instance is being used for connecting to the server
+          // in question, it is okay
+          Thread.sleep(ThreadLocalRandom.current().nextInt(reloginMaxBackoff) + 1);
+          return null;
+        } else {
+          String msg = "Failed to initiate connection for "
+              + UserGroupInformation.getLoginUser().getUserName() + " to "
+              + securityInfo.getServerPrincipal();
+          throw new IOException(msg, ex);
         }
-        throw new IOException(ex);
       }
     });
   }
@@ -459,7 +464,7 @@ private void setupIOstreams() throws IOException {
         if (useSasl) {
           final InputStream in2 = inStream;
           final OutputStream out2 = outStream;
-          UserGroupInformation ticket = getUGI();
+          UserGroupInformation ticket = provider.getRealUser(remoteId.ticket);
           boolean continueSasl;
           if (ticket == null) {
             throw new FatalConnectionException("ticket/user is null");

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcConnection.java

@@ -159,8 +159,8 @@ private void scheduleRelogin(Throwable error) {
         @Override
         public void run() {
           try {
-            if (shouldAuthenticateOverKrb()) {
-              relogin();
+            if (provider.canRetry()) {
+              provider.relogin();
             }
           } catch (IOException e) {
             LOG.warn("Relogin failed", e);
@@ -183,16 +183,16 @@ private void failInit(Channel ch, IOException e) {
   }
 
   private void saslNegotiate(final Channel ch) {
-    UserGroupInformation ticket = getUGI();
+    UserGroupInformation ticket = provider.getRealUser(remoteId.getTicket());
     if (ticket == null) {
       failInit(ch, new FatalConnectionException("ticket/user is null"));
       return;
     }
     Promise<Boolean> saslPromise = ch.eventLoop().newPromise();
     final NettyHBaseSaslRpcClientHandler saslHandler;
     try {
-      saslHandler = new NettyHBaseSaslRpcClientHandler(saslPromise, ticket, authMethod, token,
-          serverPrincipal, rpcClient.fallbackAllowed, this.rpcClient.conf);
+      saslHandler = new NettyHBaseSaslRpcClientHandler(saslPromise, ticket, provider, token,
+          serverAddress, securityInfo, rpcClient.fallbackAllowed, this.rpcClient.conf);
     } catch (IOException e) {
       failInit(ch, e);
       return;