Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Implements a server for refreshing HDFS tokens, a part of secure HDFS support. #453

Open
wants to merge 43 commits into
base: branch-2.2-kubernetes
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
7abb5ee
Add skeleton
kimoonkim Aug 19, 2017
88c0c03
Renew part
kimoonkim Aug 21, 2017
ca0b583
Compile succeeds
kimoonkim Aug 21, 2017
025e2ba
Login to kerberos
kimoonkim Aug 21, 2017
0a7a15d
Clean up constants
kimoonkim Aug 22, 2017
b3534f1
Refresh server works
kimoonkim Aug 22, 2017
cbe2777
Deployment config file
kimoonkim Aug 22, 2017
4f36793
Fix Dockerfile to match names
kimoonkim Aug 22, 2017
874b8e9
Add as independent project with own pom.xml
kimoonkim Sep 11, 2017
5dc49ca
Add working Dockerfile and deployment yaml file
kimoonkim Sep 12, 2017
388063a
Fix a bug by including hadoop conf dir in the classpath
kimoonkim Sep 13, 2017
1426523
Add token-refresh-server as extra build-only module
kimoonkim Sep 20, 2017
50c3a66
Use akka scheduler for renew tasks
kimoonkim Oct 3, 2017
c2ccaa9
Relogin to Kerberos periodically
kimoonkim Oct 3, 2017
a2aec2b
Renew at 90% mark of deadline
kimoonkim Oct 3, 2017
ec70b47
Get renew time from data item key
kimoonkim Oct 3, 2017
0b049fd
Fix compile error
kimoonkim Oct 4, 2017
d42c568
Obtain new tokens
kimoonkim Oct 4, 2017
5d96879
Fix bugs
kimoonkim Oct 4, 2017
c0e28d4
Write back tokens to K8s secret
kimoonkim Oct 4, 2017
57c847e
Handle recently added secrets
kimoonkim Oct 5, 2017
ce1bb7f
Use k8s client editable to update secret data
kimoonkim Oct 5, 2017
5162339
Add a comment
kimoonkim Oct 5, 2017
196cd8a
Keep only secret metadata in memory
kimoonkim Oct 5, 2017
56ef8e6
Fix a regex match bug
kimoonkim Oct 5, 2017
93b2acf
Tested
kimoonkim Oct 5, 2017
ba2e79a
Updated parent version
kimoonkim Oct 5, 2017
1d74579
Address review comments
kimoonkim Oct 6, 2017
95e68d3
Add TODO for token status rest endpoint
kimoonkim Oct 6, 2017
a006233
Address review comments
kimoonkim Oct 6, 2017
9dc8345
Address review comments
kimoonkim Oct 10, 2017
f4d5ee9
Support configuration
kimoonkim Oct 10, 2017
1462d2c
Fix a typo
kimoonkim Oct 11, 2017
193d0f9
Add some unit tests
kimoonkim Oct 12, 2017
eb10c4a
Add more tests
kimoonkim Oct 12, 2017
078ac2a
Clean up
kimoonkim Oct 12, 2017
aeb269a
Add more tests
kimoonkim Oct 12, 2017
87dedbc
Minor clean-up
kimoonkim Oct 13, 2017
4d1cb74
Add unit tests for renew tasks
kimoonkim Oct 13, 2017
8997f04
Verify test results more
kimoonkim Oct 13, 2017
2ed55af
Rename the new profile to kubernetes-hdfs-extra
kimoonkim Oct 18, 2017
01be03e
Fix style issues
kimoonkim Oct 18, 2017
0baaf0b
Fix Hadoop 2.7 dependency issue
kimoonkim Oct 19, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions assembly/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,9 @@
</profile>
<profile>
<id>kubernetes</id>
<!-- Not including extra Kubernetes artifacts such as token-refreh-server
because some of them are auxiliary services that should not be part
of the Spark distribution -->
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down
11 changes: 11 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -2619,6 +2619,17 @@
</modules>
</profile>

<profile>
<!-- Extra Kubernetes modules for HDFS support. They should be built
regularly, but may not need to be part of the Spark distribution. -->
<id>kubernetes-hdfs-extra</id>
<modules>
<!-- Auxiliary service for refreshing Hadoop tokens. This should not be
part of the Spark distribution -->
<module>resource-managers/kubernetes/token-refresh-server</module>
</modules>
</profile>

<profile>
<id>kubernetes-integration-tests</id>
<modules>
Expand Down
22 changes: 19 additions & 3 deletions resource-managers/kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,18 +28,34 @@ building Spark normally. For example, to build Spark against Hadoop 2.7 and Kube

dev/make-distribution.sh --tgz -Phadoop-2.7 -Pkubernetes

# Kubernetes Code Modules
# Kubernetes Modules

Below is a list of the submodules for this cluster manager and what they do.

* `core`: Implementation of the Kubernetes cluster manager support.
* `token-refresh-server`: Extra Kubernetes service that refreshes Hadoop
tokens for long-running Spark jobs that access secure data sources like
Kerberized HDFS.
* `integration-tests`: Integration tests for the project.
* `docker-minimal-bundle`: Base Dockerfiles for the driver and the executors. The Dockerfiles are used for integration
tests as well as being provided in packaged distributions of Spark.
* `integration-tests-spark-jobs`: Spark jobs that are only used in integration tests.
* `integration-tests-spark-jobs-helpers`: Dependencies for the spark jobs used in integration tests. These dependencies
are separated out to facilitate testing the shipping of jars to drivers running on Kubernetes clusters.

# Building Extra Submodules for Spark with Kubernetes

There are non-core extra submodules such as token-refresh-server. To build
those, use the `kubernetes-*-extra` profiles like `kubernetes-hdfs-extra`
when invoking Maven. For example, to build the token-refresh-server submodule:

build/mvn package -Pkubernetes-hdfs-extra \
-pl resource-managers/kubernetes/token-refresh-server -am

Some of these submodules are helper Kubernetes services. They need not be part
of the Spark distribution. The distribution build script will not include
artifacts from these submodules.

# Running the Kubernetes Integration Tests

Note that the integration test framework is currently being heavily revised and is subject to change.
Expand All @@ -64,7 +80,7 @@ build/mvn integration-test \
-pl resource-managers/kubernetes/integration-tests -am
```

# Running against an arbitrary cluster
## Running against an arbitrary cluster

In order to run against any cluster, use the following:
```sh
Expand All @@ -74,7 +90,7 @@ build/mvn integration-test \
-DextraScalaTestArgs="-Dspark.kubernetes.test.master=k8s://https://<master> -Dspark.docker.test.driverImage=<driver-image> -Dspark.docker.test.executorImage=<executor-image>"
```

# Preserve the Minikube VM
## Preserve the Minikube VM

The integration tests make use of [Minikube](https://github.com/kubernetes/minikube), which fires up a virtual machine
and setup a single-node kubernetes cluster within it. By default the vm is destroyed after the tests are finished.
Expand Down
68 changes: 68 additions & 0 deletions resource-managers/kubernetes/token-refresh-server/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
layout: global
title: Hadoop Token Refresh Server on Kubernetes
---

Spark on Kubernetes may use Kerberized Hadoop data sources such as secure HDFS or Kafka. If the job
runs for days or weeks, someone should extend the lifetime of Hadoop delegation tokens, which
expire every 24 hours. The Hadoop Token Refresh Server is a Kubernetes microservice that renews
token lifetime and puts the replacement tokens in place.

# Building the Refresh Server

To build the refresh server jar, simply run Maven. For example:

mvn clean package

The target directory will have a tarball that includes the project jar file as well as
3rd party dependency jars. The tarball name would end with `-assembly.tar.gz`. For example:

target/token-refresh-server-kubernetes_2.11-2.2.0-k8s-0.3.0-SNAPSHOT-assembly.tar.gz

# Running the Refresh Server

To run the server, follow the steps below.

1. Build and push the docker image:

docker build -t hadoop-token-refresh-server:latest \
-f src/main/docker/Dockerfile .
docker tag hadoop-token-refresh-server:latest <YOUR-REPO>:<YOUR-TAG>
docker push <YOUR-REPO>:<YOUR-TAG>

2. Edit the main application config file, src/main/conf/application.conf
and create a `configmap`:

kubectl create configmap hadoop-token-refresh-server-application-conf \
--from-file=src/main/conf/application.conf

3. Create another k8s `configmap` containing Hadoop config files. This should enable Kerberos and secure Hadoop.
It should also include the Hadoop servers that would issue delegation tokens such as the HDFS namenode
address:

kubectl create configmap hadoop-token-refresh-server-hadoop-config \
--from-file=/usr/local/hadoop/conf/core-site.xml

4. Create yet another k8s `configmap` containing Kerberos config files. This should include
the kerberos server address and the correct realm name for Kerberos principals:

kubectl create configmap hadoop-token-refresh-server-kerberos-config \
--from-file=/etc/krb5.conf

5. Create a k8s `secret` containing the Kerberos keytab file. The keytab file should include
the password for the system user Kerberos principal that the refresh server is using to
extend Hadoop delegation tokens. See
hadoop-token-refresh-server.kerberosPrincipal in the application.conf.

kubectl create secret generic hadoop-token-refresh-server-kerberos-keytab \
--from-file /mnt/secrets/krb5.keytab

6. Optionally, create a k8s `service account` and `clusterrolebinding` that
the service pod will use. The service account should have `edit` capability for
job `secret`s that contains the Hadoop delegation tokens.

7. Finally, edit the config file for k8s `deployment` and launch the service pod
using the deployment. The config file should include the right docker image tag
and the correct k8s `service account` name.

kubectl create -f src/main/conf/kubernetes-hadoop-token-refresh-server.yaml
100 changes: 100 additions & 0 deletions resource-managers/kubernetes/token-refresh-server/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.2.0-k8s-0.5.0-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>

<artifactId>token-refresh-server-kubernetes_2.11</artifactId>
<packaging>jar</packaging>
<name>Hadoop Token Refresh Server on Kubernetes</name>
<properties>
<akka.actor.version>2.5.4</akka.actor.version>
<commons-logging.version>1.2</commons-logging.version>
<kubernetes.client.version>2.2.13</kubernetes.client.version>
</properties>
<dependencies>
<dependency>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-actor_${scala.binary.version}</artifactId>
<version>${akka.actor.version}</version>
</dependency>
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-client</artifactId>
<version>${kubernetes.client.version}</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
</dependency>
<dependency>
<!-- Add codehaus jackson back that the root pom.xml excluded. It is
needed by Hadoop 2.7 Token.decodeIdentifier method. -->
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>${commons-logging.version}</version>
</dependency>
<dependency>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-testkit_${scala.binary.version}</artifactId>
<version>${akka.actor.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptors>
<descriptor>src/main/assembly/assembly.xml</descriptor>
</descriptors>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<assembly>
<id>assembly</id>
<formats>
<format>tar.gz</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<unpack>false</unpack>
<scope>compile</scope>
</dependencySet>
<dependencySet>
<unpack>false</unpack>
<scope>provided</scope>
</dependencySet>
</dependencySets>
</assembly>
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Main application config file for the Hadoop token refresh server. Override the values below
# as needed.
hadoop-token-refresh-server {

# Kerberos principal that the refresh server should use as its login user. This principal should
# match the keytab file used for the refresh server.
# For a token to be renewed for the next 24 hours by the refresh server, the token should
# designate this refresh server principal as the renewer. To allow a brand new token to be
# obtained by the refresh server, the HDFS namenode configuration should specify this refresh
# server principal as the special proxy for the job users. See
# https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Superusers.html#Configurations
# for details.
kerberosPrincipal = "MY-REFRESH-SERVER-KERBEROS-PRINCIPAL"

# Set this to true if the refresh server should scan secrets across all namespaces. Set it to
# false and specify namespaceToScan if the refresh server should scan secrets only from
# the specific namespace.
scanAllNamespaces = true

# Effective only if scanAllNamespaces is false. A specific namespace that the refresh server
# should scan secrets from.
namespaceToScan = "default"
}
Loading