Skip to content

[ZEPPELIN-1320] Run zeppelin interpreter process as web front end user #1554

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
prabhjyotsingh wants to merge 8 commits into from
Closed

[ZEPPELIN-1320] Run zeppelin interpreter process as web front end user #1554

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
aff1bf0
user should have option to run these interpreters as different user.
Oct 23, 2016
dd0731d
fix missing test cases
Oct 23, 2016
0ff80ec
Merge remote-tracking branch 'origin/master' into ZEPPELIN-1320-2
Oct 28, 2016
03b2f20
use user instead of ""
Oct 31, 2016
02c3084
Merge remote-tracking branch 'origin/master' into ZEPPELIN-1320-2
Nov 10, 2016
5a76839
show User Impersonate only when interpreter setting is "per user" and…
Nov 10, 2016
1b26cc0
add doc
Nov 14, 2016
dc69c9d
@Leemoonsoo review comment: making ZEPPELIN_SSH_COMMAND configurable
Nov 16, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions bin/interpreter.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ function usage() {
echo "usage) $0 -p <port> -d <interpreter dir to load> -l <local interpreter repo dir to load>"
}

while getopts "hp:d:l:v" o; do
while getopts "hp:d:l:v:u:" o; do
case ${o} in
h)
usage
Expand All @@ -42,6 +42,14 @@ while getopts "hp:d:l:v" o; do
. "${bin}/common.sh"
getZeppelinVersion
;;
u)
ZEPPELIN_IMPERSONATE_USER="${OPTARG}"
if [[ -z "$ZEPPELIN_IMPERSONATE_CMD" ]]; then
ZEPPELIN_IMPERSONATE_RUN_CMD=`echo "ssh ${ZEPPELIN_IMPERSONATE_USER}@localhost" `
else
ZEPPELIN_IMPERSONATE_RUN_CMD=$(eval "echo ${ZEPPELIN_IMPERSONATE_CMD} ")
fi
;;
esac
done

Expand Down Expand Up @@ -178,9 +186,9 @@ addJarInDirForIntp "${LOCAL_INTERPRETER_REPO}"
CLASSPATH+=":${ZEPPELIN_INTP_CLASSPATH}"

if [[ -n "${SPARK_SUBMIT}" ]]; then
${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
${ZEPPELIN_IMPERSONATE_RUN_CMD} `${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &`
else
${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER} ${PORT} &
${ZEPPELIN_IMPERSONATE_RUN_CMD} ${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER} ${PORT} &
fi

pid=$!
Expand Down
1 change: 1 addition & 0 deletions conf/zeppelin-env.sh.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,4 @@
# export ZEPPELINHUB_API_ADDRESS # Refers to the address of the ZeppelinHub service in use
# export ZEPPELINHUB_API_TOKEN # Refers to the Zeppelin instance token of the user
# export ZEPPELINHUB_USER_KEY # Optional, when using Zeppelin with authentication.
# export ZEPPELIN_IMPERSONATE_CMD # Optional, when user want to run interpreter as end web user. eg) 'sudo -u ${ZEPPELIN_IMPERSONATE_USER}'
1 change: 1 addition & 0 deletions docs/_includes/themes/zeppelin/_navigation.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
<li><a href="{{BASE_PATH}}/manual/interpreterinstallation.html">Interpreter Installation</a></li>
<!--<li><a href="{{BASE_PATH}}/manual/dynamicinterpreterload.html">Dynamic Interpreter Loading</a></li>-->
<li><a href="{{BASE_PATH}}/manual/dependencymanagement.html">Interpreter Dependency Management</a></li>
<li><a href="{{BASE_PATH}}/manual/userimpersonation.html">Interpreter User Impersonation</a></li>
<li role="separator" class="divider"></li>
<li class="title"><span><b>Available Interpreters</b><span></li>
<li><a href="{{BASE_PATH}}/interpreter/alluxio.html">Alluxio</a></li>
Expand Down
Binary file added docs/assets/themes/zeppelin/img/screenshots/user-impersonation.gif
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ Join to our [Mailing list](https://zeppelin.apache.org/community.html) and repor
* Usage
* [Interpreter Installation](./manual/interpreterinstallation.html): Install not only community managed interpreters but also 3rd party interpreters
* [Interpreter Dependency Management](./manual/dependencymanagement.html) when you include external libraries to interpreter
* [Interpreter User Impersonation](./manual/userimpersonation.html) when you want to run interpreter as end user
* Available Interpreters: currently, about 20 interpreters are available in Apache Zeppelin.

####Display System
Expand Down
66 changes: 66 additions & 0 deletions docs/manual/processenduser.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
layout: page
title: "Run zeppelin interpreter process as web front end user"
description: "Set up zeppelin interpreter process as web front end user."
group: manual
---
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
{% include JB/setup %}

## Run zeppelin interpreter process as web front end user

* Enable shiro auth in shiro.ini

```
[users]
user1 = password1, role1
user2 = password2, role2
```

* Enable password-less ssh for the user you want to impersonate (say user1).

```
adduser user1
#ssh-keygen (optional if you don't already have generated ssh-key.
ssh user1@localhost mkdir -p .ssh
cat ~/.ssh/id_rsa.pub | ssh user1@localhost 'cat >> .ssh/authorized_keys'
```

* Start zeppelin server.

<hr>
<div class="row">
<div class="col-md-12">
<b> Screenshot </b>
<br /><br />
</div>
<div class="col-md-12" >
<a data-lightbox="compiler" href="../assets/themes/zeppelin/img/screenshots/user-impersonation.gif">
<img class="img-responsive" src="../assets/themes/zeppelin/img/screenshots/user-impersonation.gif" />
</a>

</div>
</div>
<hr>

* Go to interpreter setting page, and enable "User Impersonate" in any of the interpreter (in my example its shell interpreter)

* Test with a simple paragraph

```
%sh
whoami
```

9 changes: 9 additions & 0 deletions zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterOption.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ public class InterpreterOption {
boolean isExistingProcess;
boolean setPermission;
List<String> users;
boolean isUserImpersonate;

public boolean isExistingProcess() {
return isExistingProcess;
Expand Down Expand Up @@ -66,6 +67,14 @@ public List<String> getUsers() {
return users;
}

public boolean isUserImpersonate() {
return isUserImpersonate;
}

public void setUserImpersonate(boolean userImpersonate) {
isUserImpersonate = userImpersonate;
}

public InterpreterOption() {
this(false);
}
Expand Down
22 changes: 18 additions & 4 deletions ...n-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ public class RemoteInterpreter extends Interpreter {
private int maxPoolSize;
private String host;
private int port;
private String userName;
private Boolean isUserImpersonate;

/**
* Remote interpreter and manage interpreter process
Expand All @@ -72,7 +74,9 @@ public RemoteInterpreter(Properties property,
int connectTimeout,
int maxPoolSize,
RemoteInterpreterProcessListener remoteInterpreterProcessListener,
ApplicationEventListener appListener) {
ApplicationEventListener appListener,
String userName,
Boolean isUserImpersonate) {
super(property);
this.noteId = noteId;
this.className = className;
Expand All @@ -85,6 +89,8 @@ public RemoteInterpreter(Properties property,
this.maxPoolSize = maxPoolSize;
this.remoteInterpreterProcessListener = remoteInterpreterProcessListener;
this.applicationEventListener = appListener;
this.userName = userName;
this.isUserImpersonate = isUserImpersonate;
}


Expand All @@ -100,7 +106,9 @@ public RemoteInterpreter(
int connectTimeout,
int maxPoolSize,
RemoteInterpreterProcessListener remoteInterpreterProcessListener,
ApplicationEventListener appListener) {
ApplicationEventListener appListener,
String userName,
Boolean isUserImpersonate) {
super(property);
this.noteId = noteId;
this.className = className;
Expand All @@ -111,6 +119,8 @@ public RemoteInterpreter(
this.maxPoolSize = maxPoolSize;
this.remoteInterpreterProcessListener = remoteInterpreterProcessListener;
this.applicationEventListener = appListener;
this.userName = userName;
this.isUserImpersonate = isUserImpersonate;
}


Expand All @@ -125,7 +135,9 @@ public RemoteInterpreter(
Map<String, String> env,
int connectTimeout,
RemoteInterpreterProcessListener remoteInterpreterProcessListener,
ApplicationEventListener appListener) {
ApplicationEventListener appListener,
String userName,
Boolean isUserImpersonate) {
super(property);
this.className = className;
this.noteId = noteId;
Expand All @@ -138,6 +150,8 @@ public RemoteInterpreter(
this.maxPoolSize = 10;
this.remoteInterpreterProcessListener = remoteInterpreterProcessListener;
this.applicationEventListener = appListener;
this.userName = userName;
this.isUserImpersonate = isUserImpersonate;
}

private Map<String, String> getEnvFromInterpreterProperty(Properties property) {
Expand Down Expand Up @@ -205,7 +219,7 @@ public synchronized void init() {
RemoteInterpreterProcess interpreterProcess = getInterpreterProcess();

final InterpreterGroup interpreterGroup = getInterpreterGroup();
interpreterProcess.reference(interpreterGroup);
interpreterProcess.reference(interpreterGroup, userName, isUserImpersonate);
interpreterProcess.setMaxPoolSize(
Math.max(this.maxPoolSize, interpreterProcess.getMaxPoolSize()));
String groupId = interpreterGroup.getId();
Expand Down
6 changes: 5 additions & 1 deletion ...src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterManagedProcess.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ public int getPort() {
}

@Override
public void start() {
public void start(String userName, Boolean isUserImpersonate) {
// start server process
try {
port = RemoteInterpreterUtils.findRandomAvailablePortOnAllLocalInterfaces();
Expand All @@ -101,6 +101,10 @@ public void start() {
cmdLine.addArgument(interpreterDir, false);
cmdLine.addArgument("-p", false);
cmdLine.addArgument(Integer.toString(port), false);
if (isUserImpersonate && !userName.equals("anonymous")) {
cmdLine.addArgument("-u", false);
cmdLine.addArgument(userName, false);
}
cmdLine.addArgument("-l", false);
cmdLine.addArgument(localRepoDir, false);

Expand Down
7 changes: 4 additions & 3 deletions ...preter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcess.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,18 +63,19 @@ public RemoteInterpreterProcess(

public abstract String getHost();
public abstract int getPort();
public abstract void start();
public abstract void start(String userName, Boolean isUserImpersonate);
public abstract void stop();
public abstract boolean isRunning();

public int getConnectTimeout() {
return connectTimeout;
}

public int reference(InterpreterGroup interpreterGroup) {
public int reference(InterpreterGroup interpreterGroup, String userName,
Boolean isUserImpersonate) {
synchronized (referenceCount) {
if (!isRunning()) {
start();
start(userName, isUserImpersonate);
}

if (clientPool == null) {
Expand Down
2 changes: 1 addition & 1 deletion ...src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterRunningProcess.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ public int getPort() {
}

@Override
public void start() {
public void start(String userName, Boolean isUserImpersonate) {
// assume process is externally managed. nothing to do
}

Expand Down
4 changes: 3 additions & 1 deletion ...rpreter/src/test/java/org/apache/zeppelin/interpreter/remote/RemoteAngularObjectTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,9 @@ public void setUp() throws Exception {
env,
10 * 1000,
null,
null
null,
"anonymous",
false
);

intpGroup.put("note", new LinkedList<Interpreter>());
Expand Down
4 changes: 3 additions & 1 deletion ...c/test/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterOutputTestStream.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,9 @@ private RemoteInterpreter createMockInterpreter() {
env,
10 * 1000,
this,
null);
null,
"anonymous",
false);

intpGroup.get("note").add(intp);
intp.setInterpreterGroup(intpGroup);
Expand Down
8 changes: 4 additions & 4 deletions ...er/src/test/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcessTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ public void testStartStop() {
10 * 1000, null, null);
assertFalse(rip.isRunning());
assertEquals(0, rip.referenceCount());
assertEquals(1, rip.reference(intpGroup));
assertEquals(2, rip.reference(intpGroup));
assertEquals(1, rip.reference(intpGroup, "anonymous", false));
assertEquals(2, rip.reference(intpGroup, "anonymous", false));
assertEquals(true, rip.isRunning());
assertEquals(1, rip.dereference());
assertEquals(true, rip.isRunning());
Expand All @@ -61,7 +61,7 @@ public void testClientFactory() throws Exception {
RemoteInterpreterManagedProcess rip = new RemoteInterpreterManagedProcess(
INTERPRETER_SCRIPT, "nonexists", "fakeRepo", new HashMap<String, String>(),
mock(RemoteInterpreterEventPoller.class), 10 * 1000);
rip.reference(intpGroup);
rip.reference(intpGroup, "anonymous", false);
assertEquals(0, rip.getNumActiveClient());
assertEquals(0, rip.getNumIdleClient());

Expand Down Expand Up @@ -106,7 +106,7 @@ public void testStartStopRemoteInterpreter() throws TException, InterruptedExcep
, 10 * 1000);
assertFalse(rip.isRunning());
assertEquals(0, rip.referenceCount());
assertEquals(1, rip.reference(intpGroup));
assertEquals(1, rip.reference(intpGroup, "anonymous", false));
assertEquals(true, rip.isRunning());
}
}
23 changes: 17 additions & 6 deletions ...terpreter/src/test/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,9 @@ private RemoteInterpreter createMockInterpreterA(Properties p, String noteId) {
env,
10 * 1000,
null,
null);
null,
"anonymous",
false);
}

private RemoteInterpreter createMockInterpreterB(Properties p) {
Expand All @@ -108,7 +110,9 @@ private RemoteInterpreter createMockInterpreterB(Properties p, String noteId) {
env,
10 * 1000,
null,
null);
null,
"anonymous",
false);
}

@Test
Expand Down Expand Up @@ -207,7 +211,9 @@ public void testRemoteSchedulerSharing() throws TTransportException, IOException
env,
10 * 1000,
null,
null);
null,
"anonymous",
false);


intpGroup.get("note").add(intpA);
Expand All @@ -223,7 +229,9 @@ public void testRemoteSchedulerSharing() throws TTransportException, IOException
env,
10 * 1000,
null,
null);
null,
"anonymous",
false);

intpGroup.get("note").add(intpB);
intpB.setInterpreterGroup(intpGroup);
Expand Down Expand Up @@ -687,7 +695,8 @@ public void should_push_local_angular_repo_to_remote() throws Exception {
//Given
final Client client = Mockito.mock(Client.class);
final RemoteInterpreter intr = new RemoteInterpreter(new Properties(), "noteId",
MockInterpreterA.class.getName(), "runner", "path","localRepo", env, 10 * 1000, null, null);
MockInterpreterA.class.getName(), "runner", "path", "localRepo", env, 10 * 1000, null,
null, "anonymous", false);
final AngularObjectRegistry registry = new AngularObjectRegistry("spark", null);
registry.add("name", "DuyHai DOAN", "nodeId", "paragraphId");
final InterpreterGroup interpreterGroup = new InterpreterGroup("groupId");
Expand Down Expand Up @@ -733,7 +742,9 @@ public void testEnvronmentAndPropertySet() {
env,
10 * 1000,
null,
null);
null,
"anonymous",
false);

intpGroup.put("note", new LinkedList<Interpreter>());
intpGroup.get("note").add(intp);
Expand Down
Loading