Added Shiro security #53

hayssams · 2015-04-27T09:00:14Z

Added shiro security. HTTP calls and websockets communications are both protected.

No need to proceed web resources here.

No cleanup is done, since the same user accross different devices share the same ticket The Map size is at most the number of different user names having access to a Zeppelin instance

…each note is now attached to a user

…e username.

…tial HTTP call to the SecurityRestApi. This proves that web socket access is done by the user who provided the credentials through Shiro on the HTTP channel.

Tests are updated accordingly by using the anonymous user.

hayssams · 2015-04-27T09:04:39Z

Support for authentication only.
Notes are not shared among users.
To share notes, you'll have to share the username/password among different users.

Leemoonsoo · 2015-04-29T04:46:21Z

Hi @hayssams, Thanks for contribution. Let me test this branch.

hayssams · 2015-05-04T16:08:18Z

Hi @Leemoonsoo, did you get a chance to test this PR ?

Leemoonsoo · 2015-05-05T02:01:42Z

Sorry for late responding and thanks again for contribution.

In my understanding, this branch trying to provides 1) security of rest/websocket api. 2) multi-tenant environment for notebook save/load.

Because of this PR brings wide range of changes, we have couple of things to discuss.

providing login window
displaying current user
how to merge changes https://issues.apache.org/jira/browse/ZEPPELIN-26
migration script for notebook directory structure change.
notebook multi tenant model (per user / shares all, due to any user actually can read any notebook from filesystem api on Spark/Pyspark interpreter)

I'm feeling like if it is possible to separate 1) security of rest/websocket 2) multi-tenancy of notebook into different issue, it'll be much faster to review and test.

What do you think?

hayssams · 2015-05-05T11:27:13Z

You're right, this PR addresses 1) & 2).

Regarding the points you raised

The login window is useful if we want to support form auth in addition to the basic auth this PR provides. This may be added in another PR easily with almost no impact to the current code.
Right, we need to display the current user. Do you want me to add it at the top of the note with a logout button ?
Zeppelin-26 : Did not look at it yet.
Migration script : that should be easy. to just move existing notes to the anonymous subdirectory with no impact on the existing code
Multitenant : Would starting the "interpreter.sh" script with a different identity solve the problem ?

Actually feature 2) involves a very small amount of code (load/save notes in user directory).

I am personally interested by feature 2). let me know if you want me to complete the other points you raised first or if you prefer to stick to feature 1). I'll respect your choice.

anthonycorbacho · 2015-05-05T14:33:45Z

Hi,

For me I am really confuse about the change on the rest api, why do we need principal and ticket? and why this is part of the rest api url?
Why dont we create cookies for that?
Why do you refer stormpath in the shiro ini file?
Why the login page should be handled on an other PR?
since in the ini file the urls are anonimus /** = anon how do you authenticate users?
how the sessions are handled?

hayssams · 2015-05-05T14:55:17Z

Hi Anthony

You're right using cookies for the HTTP API would be a much better choice.
Regarding storm path, The comments you find in the shiro.ini file come from the default shiro.ini setup; Wen can safely ignore/remove them
The login page is not required for shiro to work. We can use basic auth by uncommenting line 17 in the shiro.ini file (/** = authcBasic). The browser will popup a login dialog without any coding required.
By default auth is not activated. To activate it, simply comment line 16 and uncomment line 17 in shiro.ini. The file shiro.ini need to be accessible through the classpath.

hayssams · 2015-05-06T08:13:39Z

I missed your last question about session handling.
I do not use sessions since the credentials are passed with each call.

* master: #64 Force opacity 0 in .ace_text-input.ace_composition Fix process creation behavior ZEPPELIN-59 fix interpreter restart Fixes a typo in conf_dir check ZEPPELIN-26 Pluggable notebook persistence layer Conflicts: zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java

elbamos · 2015-12-28T05:42:06Z

I think we should have an organized plan to review and address this PR.

I propose that the PR be merged into a new branch of Zeppelin, where the community as a whole will have an easier time reviewing and making PRs on the branch.

Then I propose that we review the aspects of this separately. By reviewing/approving/voting-on each aspect in sequence, we will have a clear idea of where the PR is in process, and that will make it easier to move PR forward to final merge (if that's what we want to do). The aspects I see are:

Feature set -- is this the kind of security interface we want to put in Zeppelin?
Architecture -- Is this (shiro) how we want to implement security?
Does it work?
Do we have adequate tests to be able to support and maintain it?
Is it adequately documented that people will be able to use it?

The reason I propose a more formal process is that this PR has been sitting here for 8 months now. With that, and with some 82 (!!!) commits, a more structured approach seems like the best way to resolve this PR (one way or the other).

(In full disclosure -- My only view about this is that since its security-related, it needs to have mounds of tests to make sure the security doesn't get compromised by later commits.)

…into shira-security # Conflicts: # zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java # zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java # zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinRestApiTest.java # zeppelin-web/src/components/navbar/navbar.controller.js # zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java # zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java # zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/NotebookRepoSync.java

jongyoul · 2015-12-28T09:27:06Z

In addition to @elbamos's comments, I hope that the community would understand that it's too hard to review a kind of this PR which has changed widely because committers may not know features well. I, however, also know some features take extensive changes. In this case, I propose that a contributor makes a feature Jira issue that includes several sub-tasks, and makes a - relatively - small PR for every sub-task. It would help review, test and merge the PR easily.

rconline · 2015-12-29T14:10:54Z

@jongyoul @elbamos @hayssams this change is an important one, and we should try to get this in. My two cents on how we could divide the whole pull request into:

inclusion of libraries and dependencies
Addition of security filter, changes in Angular Object, configurable - turned off by default. With a basic set of test-cases, which ensure the changes are non-breaking.
Front-end changes, which include -display of username, accepting header parameters of principal, ticket.
Subsequently, addition of more test cases, before we call it done. Till such time we keep the feature as default off.
(Each of the above steps will have user guides/documentation).

If the committers can provide some bandwidth for review, I'm willing to make the above changes + documentation.

As side notes:

We will need to include the notion of proxy-users at sometime, which enable command execution of secure spark/hdfs clusters
Subsequently, there will have to be a mechanism of how to provide default access to the cluster users, either by way of import of AD/LDAP users, through some auth_to_local kind of rules.

I'm of the opinion that Shiro is a good strategy, larger projects such as https://knox.apache.org/ use Shiro, for HDFS Api gateway security.

…ed user are returned.

hayssams · 2015-12-29T15:22:44Z

@elbamos @jongyoul @rconline
Let me write down the whole set of changes I've made and how they may be split in my view in smaller commits.
There are also different strategies in using shiro. I would like to submit them and take into account the one you advise to use.

…into shira-security

jongyoul · 2015-12-29T15:37:21Z

@hayssams Sure! You can split this into several commits because this PR is your contribution and we will thank you so much of that. If you want to do this, would you mind creating Jira issue or linking a related issue that is already created?

jeffsteinmetz · 2015-12-29T17:39:23Z

Is there an example of how shiro would be pluggable with other forms of
authentication and authorization?
Such as Java Web tokens (JWT), which require storing a secret. And the JWT
contains authorization roles.
On Dec 29, 2015 6:11 AM, "rconline" notifications@github.com wrote:

@jongyoul https://github.com/jongyoul @elbamos
https://github.com/elbamos @hayssams https://github.com/hayssams this
change is an important one, and we should try to get this in. My two cents
on how we could divide the whole pull request into:

inclusion of libraries and dependencies

Addition of security filter, changes in Angular Object, configurable

turned off by default. With a basic set of test-cases, which ensure the
changes are non-breaking.

Front-end changes, which include -display of username, accepting
header parameters of principal, ticket.

Subsequently, addition of more test cases, before we call it done.
Till such time we keep the feature as default off. (Each of the above steps
will have user guides/documentation).

If the committers can provide some bandwidth for review, I'm willing to
make the above changes + documentation.

As side notes:

We will need to include the notion of proxy-users at sometime, which
enable command execution of secure spark/hdfs clusters

Subsequently, there will have to be a mechanism of how to provide
default access to the cluster users, either by way of import of AD/LDAP
users, through some auth_to_local kind of rules.

I'm of the opinion that Shiro is a good strategy, larger projects such as
https://knox.apache.org/ use Shiro, for HDFS Api gateway security.

—
Reply to this email directly or view it on GitHub
#53 (comment)
.

rconline · 2015-12-29T20:11:37Z

@hayssams please do that, it will be great. I can help with Junits, Docs and separation, should you need it, will commit into your branches, which can then be submitted as PRs.

…into shira-security

hayssams · 2015-12-30T14:00:25Z

@jongyoul @elbamos @rconline
I have just added a short security readme that may allow us to choose the best strategy in adding Shiro. Basically the two questions are :

Is the way we handle web socket security a good pattern
Should we rely on the ThreadLocal pattern or explicitly pass the username in function calls or should we use a different approach ?

Once we decide how to move on, I think that we will be able to focus on the implementation details and how this PR should be split.

Waiting to hear from you.

elbamos · 2015-12-31T04:16:21Z

@hayssams Can you provide a very step-by-step intro readme? Default username/pass, what the authentication backend is, how users/groups are added, how to make notebooks belong to one user or another... That very basic stuff?

hayssams · 2015-12-31T18:06:36Z

@elbamos @jongyoul @rconline
I created a new PR (#586) which implements HTTP et Websocket security but does not implement note ownership. This means that a user must authenticate to access Zeppelin but once authenticated, he will have access to all notes. That's what @Leemoonsoo wanted to do first.

For easier review, this new PR (#586) contains only two commits. The first one implements HTTP Auth and the second one Websocket authentication .

Once PR # 586 is merged / validated, I suggest that we move forward with note ownership and permissions.

jeffsteinmetz · 2015-12-31T18:27:43Z

One comment: why is Stormpath (a commercial auth service unrelated to Zeppelin) mentioned in the comments for this PR?
See:
https://github.com/apache/incubator-zeppelin/pull/586/files#diff-9b7b90bbe0069ba81385d11637dcd4e4R27

From: Hayssam Saleh
Reply-To: apache/incubator-zeppelin
Date: Thursday, December 31, 2015 at 10:06 AM
To: apache/incubator-zeppelin
Cc: Jeff Steinmetz
Subject: Re: [incubator-zeppelin] Added Shiro security (#53)

@elbamos @jongyoul @rconline
I created a new PR (#586) which implements HTTP et Websocket security but does not implement note ownership. This means that a user must authenticate to access Zeppelin but once authenticated, he will have access to all notes. That's what @Leemoonsoo wanted to do first.

For easier review, this new PR (#586) contains only two commits. The first one implements HTTP Auth and the second one Websocket authentication .

Once PR # 586 is merged / validated, I suggest that we move forward with note ownership and permissions.

—
Reply to this email directly or view it on GitHub.

hayssams · 2015-12-31T18:47:29Z

You are right that should be removed.
I took it from the shiro sample.
I am removing the whole section right away.

On 31 Dec 2015, at 20:27, Jeff Steinmetz notifications@github.com wrote:

One comment: why is Stormpath (a commercial auth service unrelated to Zeppelin) mentioned in the comments for this PR?
See:
https://github.com/apache/incubator-zeppelin/pull/586/files#diff-9b7b90bbe0069ba81385d11637dcd4e4R27

From: Hayssam Saleh
Reply-To: apache/incubator-zeppelin
Date: Thursday, December 31, 2015 at 10:06 AM
To: apache/incubator-zeppelin
Cc: Jeff Steinmetz
Subject: Re: [incubator-zeppelin] Added Shiro security (#53)

@elbamos @jongyoul @rconline
I created a new PR (#586) which implements HTTP et Websocket security but does not implement note ownership. This means that a user must authenticate to access Zeppelin but once authenticated, he will have access to all notes. That's what @Leemoonsoo wanted to do first.

For easier review, this new PR (#586) contains only two commits. The first one implements HTTP Auth and the second one Websocket authentication .

Once PR # 586 is merged / validated, I suggest that we move forward with note ownership and permissions.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub #53 (comment).

EBIZNEXT http://www.ebiznext.com Java / .NET */ DevOps / BigData / UX
/ Agile*
23 rue Vernet 75008 Paris
https://maps.google.com/?q=23%20rue%20Vernet%2075008%20Paris

hayssams · 2015-12-31T19:01:17Z

@jongyoul
I just created 2 JIRA issues.
ZEPPELIN-548 for Zeppelin Authentication
ZEPPELIN-549 for Zeppelin Authorisation

hayssams · 2015-12-31T19:02:37Z

I am closing this PR and suggest that we move the discussion to #586

Added Authentication. Once authenticated, a user has access to all notes. HTTP & Websocket channels are secured and require auth. This PR is based on #53 which also implements user ownership on notes. Author: Hayssam Saleh <hayssam.saleh@ebiznext.com> Closes #586 from hayssams/shiro-security-v2 and squashes the following commits: 47421b8 [Hayssam Saleh] Rollback classpath change since zeppelin conf dir already in classpath 5485dcd [Hayssam Saleh] Updates licences for shiro-core and shiro-web introduced in this PR 7200e77 [Hayssam Saleh] Default ticket / principal to anonymous in websocket message 30736a0 [Hayssam Saleh] Add support for cross site requests with credentials 1372231 [Hayssam Saleh] Test mode requires to user baseUrlSrv to connect to the REST API 96ec240 [Hayssam Saleh] use standard HTML tags for SECURITY-README.md 01ba543 [Hayssam Saleh] get ticket before Angular is bootstrapped 2a9e275 [Hayssam Saleh] Add implementation notes 96d1fac [Hayssam Saleh] correct comment in SECURITY-README and keep anonymous policy by default in zeppelin-site.xml.template 6fd9982 [Hayssam Saleh] Add minimal shiro.ini file for test phase 8eee51d [Hayssam Saleh] Remove cache optimization in shiro since it references stormpath and comes from there. 2017925 [Hayssam Saleh] exclude SECURITY-README from rat check f9b1952 [Hayssam Saleh] The Websocket channel is now as secure as the HTTP channel. e2affca [Hayssam Saleh] Securing the HTTP channel only. Websocket security is done in the next commit

guptabandhu25 · 2016-06-02T04:27:57Z

Hi Hayssams , the authentication letter is applied but in terms of exception as i am not getting any window for putting my username and password . I am in the customer network so it is not asking for any ausername and password .

Showing the message

ERROR [2016-06-02 14:27:22,265]({qtp513169028-34} NotebookServer.java[onMessage]:179) - Can't handle message
java.lang.Exception: Anonymous access not allowed
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:110)
at org.apache.zeppelin.socket.NotebookSocket.onMessage(NotebookSocket.java:56)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameHandler.onFrame(WebSocketConnectionRFC6455.java:835)
at org.eclipse.jetty.websocket.WebSocketParserRFC6455.parseNext(WebSocketParserRFC6455.java:349)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.handle(WebSocketConnectionRFC6455.java:225)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

### What is this PR for? [SECURITY-README.md](https://github.com/apache/zeppelin/blob/master/SECURITY-README.md) was added by #53 when Shiro auth was implemented for the first time. But I think we need to keep "Security Setup" information in one source; [Official docs website](https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/security/shiroauthentication.html#security-setup) and guide ppl to see this official docs page so that `SECURITY-README.md` can only contain dev related contents (not step by step setup guide). ### What type of PR is it? Documentation ### What is the Jira issue? N/A ### How should this be tested? Just clicking "View" would be enough I guess :) ### Questions: * Does the licenses files need update? N/A * Is there breaking changes for older versions? N/A * Does this needs documentation? N/A Author: AhyoungRyu <fbdkdud93@hanmail.net> Closes #1829 from AhyoungRyu/remove/duplicatedSection and squashes the following commits: d4b80b8 [AhyoungRyu] Minor update 8ce33be [AhyoungRyu] Remove note sentence 3e867ac [AhyoungRyu] Fix wired numbering d908da9 [AhyoungRyu] Remove outdated 'Security Setup' section in SECURITY-README

hayssams added 16 commits April 27, 2015 10:14

Added shirt dependencies

4295168

default shiro configuration file

e52f4fb

The connected or anonymous user should get a ticket first from this API

6e07f10

Shiro filter to protect access to the API.

8178c2c

No need to proceed web resources here.

Very simple ticket container

774ada9

No cleanup is done, since the same user accross different devices share the same ticket The Map size is at most the number of different user names having access to a Zeppelin instance

User name & ticket are now required to access the notebook api since …

c2b3cf3

…each note is now attached to a user

Notes are now loaded from and stored in a subdirectory named after th…

16f0ace

…e username.

Web socket handlers require the username & ticket got through the ini…

a8eb2cc

…tial HTTP call to the SecurityRestApi. This proves that web socket access is done by the user who provided the credentials through Shiro on the HTTP channel.

Notebooks are now user dependent. We still load all note in memory.

f81aaa5

Tests are updated accordingly by using the anonymous user.

Get ticket on startup

3f96384

Submit username & ticket on web socket requests

baf311c

add user & ticket to API calls

4c9167e

Upgrade tests

f9244c3

Message hold the username & ticket now

ba1da14

Angular objects are attached specific to a user

7ab1b23

For an anonymous user, set ticket to anonymous to allow testing

284e2ea

hayssams added 3 commits April 27, 2015 12:44

Add missing import

cd05b62

Add Apache Licence

5960221

anonymous/anonymous is always a valid username/ticket combination

059b0fd

hayssams added 3 commits May 7, 2015 11:02

Use HTTP Header for Auth

aa4199e

Remove reference to stormpath

9c5316c

hayssams added 4 commits December 28, 2015 08:44

fix security params

83deb10

fix indentation

1402ef1

More SecurityUtils

47b8f0b

Added support for Lucene. On search only the notebooks of the connect…

bb309f3

…ed user are returned.

Merge branch 'master' of https://github.com/apache/incubator-zeppelin …

29e0a52

…into shira-security

hayssams added 3 commits December 30, 2015 15:42

Security readme

eb00798

Merge branch 'master' of https://github.com/apache/incubator-zeppelin …

a39210e

…into shira-security

markdown fix

3b27605

minor update to import

81488ea

hayssams closed this Dec 31, 2015

hayssams mentioned this pull request Dec 31, 2015

Shiro security v2 #586

Closed

AhyoungRyu mentioned this pull request Jan 1, 2017

[DOCS] Remove outdated 'Security Setup' section in SECURITY-README #1829

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Shiro security #53

Added Shiro security #53

hayssams commented Apr 27, 2015

hayssams commented Apr 27, 2015

Leemoonsoo commented Apr 29, 2015

hayssams commented May 4, 2015

Leemoonsoo commented May 5, 2015

hayssams commented May 5, 2015

anthonycorbacho commented May 5, 2015

hayssams commented May 5, 2015

hayssams commented May 6, 2015

elbamos commented Dec 28, 2015

jongyoul commented Dec 28, 2015

rconline commented Dec 29, 2015

hayssams commented Dec 29, 2015

jongyoul commented Dec 29, 2015

jeffsteinmetz commented Dec 29, 2015

rconline commented Dec 29, 2015

hayssams commented Dec 30, 2015

elbamos commented Dec 31, 2015

hayssams commented Dec 31, 2015

jeffsteinmetz commented Dec 31, 2015

hayssams commented Dec 31, 2015

hayssams commented Dec 31, 2015

hayssams commented Dec 31, 2015

guptabandhu25 commented Jun 2, 2016 •

edited

Loading

Added Shiro security #53

Added Shiro security #53

Conversation

hayssams commented Apr 27, 2015

hayssams commented Apr 27, 2015

Leemoonsoo commented Apr 29, 2015

hayssams commented May 4, 2015

Leemoonsoo commented May 5, 2015

hayssams commented May 5, 2015

anthonycorbacho commented May 5, 2015

hayssams commented May 5, 2015

hayssams commented May 6, 2015

elbamos commented Dec 28, 2015

jongyoul commented Dec 28, 2015

rconline commented Dec 29, 2015

hayssams commented Dec 29, 2015

jongyoul commented Dec 29, 2015

jeffsteinmetz commented Dec 29, 2015

rconline commented Dec 29, 2015

hayssams commented Dec 30, 2015

elbamos commented Dec 31, 2015

hayssams commented Dec 31, 2015

jeffsteinmetz commented Dec 31, 2015

hayssams commented Dec 31, 2015

hayssams commented Dec 31, 2015

hayssams commented Dec 31, 2015

guptabandhu25 commented Jun 2, 2016 • edited Loading

guptabandhu25 commented Jun 2, 2016 •

edited

Loading