[ZEPPELIN-1500] pyspark completion doesn't work.

[astroshim]

What is this PR for?

@fanlu reported bug on #530.
This PR fixes bug of checking condition in PySparkInterpreter.

What type of PR is it?

Bug Fix

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1500

How should this be tested?

Outline the steps to test the PR here.

Questions:

• Does the licenses files need update? no
• Is there breaking changes for older versions? no
• Does this needs documentation? no

[felixcheung]

why is pythonScriptInitialized check removed?

[astroshim]

@felixcheung Thank you for quick review.
It's because we should get result message even if pythonScriptInitialized is true and I thought pythonScriptInitialized on the completion() increases the complexity of code.

[felixcheung]

do we have tests for completion?

[fanlu]

I use git apply this patch
but the problem is still there
ERROR [2016-09-28 23:23:53,892]({qtp659748578-108} NotebookServer.java[onMessage]:261) - Can't handle message
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.TApplicationException: completion failed: unknown result
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:453)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:122)
at org.apache.zeppelin.notebook.Paragraph.completion(Paragraph.java:242)
at org.apache.zeppelin.notebook.Note.completion(Note.java:520)
at org.apache.zeppelin.socket.NotebookServer.completion(NotebookServer.java:797)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:217)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.TApplicationException: completion failed: unknown result
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_completion(RemoteInterpreterService.java:362)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.completion(RemoteInterpreterService.java:342)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:449)
... 21 more

[fanlu]

@astroshim Another problem has emerged
ERROR [2016-09-28 23:29:20,018]({pool-1-thread-2} TThreadPoolServer.java[run]:296) - Error occurred during processing of message.
java.lang.NullPointerException
at org.apache.zeppelin.spark.PySparkInterpreter.completion(PySparkInterpreter.java:453)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:122)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.completion(RemoteInterpreterServer.java:469)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1742)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1727)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

[astroshim]

@fanlu That error seems like occurs before launching pyspark interpreter.
Could you please try completion after launching pyspark interpreter if works or not like following?

[fanlu]

@astroshim It worked in pyspark, thanks
How about python autocomplete
ERROR [2016-09-29 11:35:43,314]({qtp659748578-207} NotebookServer.java[onMessage]:261) - Can't handle message
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.TApplicationException: completion failed: unknown result
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:453)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:122)
at org.apache.zeppelin.notebook.Paragraph.completion(Paragraph.java:242)
at org.apache.zeppelin.notebook.Note.completion(Note.java:520)
at org.apache.zeppelin.socket.NotebookServer.completion(NotebookServer.java:797)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:217)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.TApplicationException: completion failed: unknown result
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_completion(RemoteInterpreterService.java:362)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.completion(RemoteInterpreterService.java:342)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:449)
... 21 more

[astroshim]

rebuild CI

[astroshim]

re-trigger CI

[astroshim]

please review this PR.

[felixcheung]

LGTM, tests are green

[Leemoonsoo]

LGTM and there're no further discussions