Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow start of Julia kernel. #3

Closed
BoPeng opened this issue Nov 29, 2017 · 19 comments
Closed

Slow start of Julia kernel. #3

BoPeng opened this issue Nov 29, 2017 · 19 comments
Assignees

Comments

@BoPeng
Copy link
Contributor

BoPeng commented Nov 29, 2017

Loading the three perhaps rarely used modules take 4 seconds whenever the Julia kernel is started. There should be some way for us to use them wherever they are actually used.

$ time julia -e 'using Feather;using DataFrames;using NamedArrays;'
WARNING: Method definition ==(Base.Nullable{S}, Base.Nullable{T}) in module Base at nullable.jl:238 overwritten in module NullableArrays at /Users/bpeng1/.julia/v0.6/NullableArrays/src/operators.jl:99.

real	0m4.514s
user	0m4.373s
sys	0m0.295s
$ time julia -e ''

real	0m0.327s
user	0m0.217s
sys	0m0.143s
@BoPeng
Copy link
Contributor Author

BoPeng commented Nov 29, 2017

There is a walk around to use eval to resolve this.

@HenryLeongStat
Copy link
Member

if !isdefined(:DataFrames)
    eval(Expr(:using,:DataFrames))

Works.

However, the followings:

function __julia_py_repr_matrix(obj)
  if !isdefined(:Feather)
    eval(Expr(:using,:Feather))
  end
  if !isdefined(:DataFrames)
    eval(Expr(:using,:DataFrames))
  end
  tf = joinpath(tempname())
  p = convert(DataFrame, obj)
  Feather.write(tf, p)
  return "numpy.asmatrix(read_dataframe(r'" * tf * "'))"
end
a=diagm(1:6)
__julia_py_repr_matrix(a)

Error would appear. Run __julia_py_repr_matrix(a) again after the error, it works. Trying to figure out the problem here.

@BoPeng
Copy link
Contributor Author

BoPeng commented Nov 29, 2017

Not sure but that post appears to mention the use of DataFrames.DataFrame or something of that nature.

@HenryLeongStat
Copy link
Member

Trying something like the followings:

function mat2df()
  if !isdefined(:DataFrames)
    eval(Expr(:using,:DataFrames))
  end
  if !isdefined(:DataFrames)
    print("Pakcage wasn't loaded \n \n")
  elseif isdefined(:DataFrames)
    print("Pakcage was loaded \n \n")
    try
      print("Since the package is loaded, we can create a dataframe \n \n")
      print(DataFrame(A = 1:4, B = 4.0:-1.0:1.0))
    catch
      print("package is loaded, why can't a dataframe be defined? \n \n")
      print(DataFrames.DataFrame(A = [1, 2], B = [4, 5]))
    end
  end
end

mat2df()

It does load the package, but it will raise an error:

manchongleong@Mande-MacBook-Pro:~$ julia testJ1.ji
Pakcage was loaded

Since the package is loaded, we can create a dataframe

package is loaded, why can't a dataframe be defined?

ERROR: LoadError: MethodError: no method matching DataFrames.DataFrame(; A=[1, 2], B=[4, 5])
Closest candidates are:
  DataFrames.DataFrame(; kwargs...) at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:98
  DataFrames.DataFrame(!Matched::Array{DataType,1}, !Matched::Array{Symbol,1}, !Matched::Array{Bool,1}, !Matched::Integer) at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:132 got unsupported keyword arguments "A", "B"
  DataFrames.DataFrame(!Matched::Array{D<:Associative,1}) where D<:Associative at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:157 got unsupported keyword arguments "A", "B"
  ...
Stacktrace:
 [1] mat2df() at /Users/manchongleong/testJ1.ji:26
 [2] include_from_node1(::String) at ./loading.jl:569
 [3] include(::String) at ./sysimg.jl:14
 [4] process_options(::Base.JLOptions) at ./client.jl:305
 [5] _start() at ./client.jl:371
while loading /Users/manchongleong/testJ1.ji, in expression starting on line 34

Seems like the problem is the world age. In my concepts, when the function runs, it is in the "old" age, and the package is loaded, it is in the "new" age, but when the syntax based on the package runs, it is still in the "old" age. That is just my guess after I read this discussion..

@HenryLeongStat
Copy link
Member

Btw, it does load the package, just fails to call it...

@HenryLeongStat
Copy link
Member

Still fail after adding Base.invokelatest:

function mat2df()
  #check_defined_eval(:Feather)
  #check_defined_eval(:DataFrames)
  #if !isdefined(:Feather)
  #  eval(Expr(:using,:Feather))
  #  #backend()
  #end
  if !isdefined(:DataFrames)
    eval(Expr(:using,:DataFrames))
  #  #backend()
  end
  #tf = joinpath(tempname())
  if !isdefined(:DataFrames)
    print("Pakcage wasn't loaded \n \n")
  #  #backend()
  elseif isdefined(:DataFrames)
    print("Pakcage was loaded \n \n")
    try
      print("Since the package is loaded, we can create a dataframe \n \n")
      #return convert(DataFrames.DataFrame, rand(2,2))
      print(Base.invokelatest(DataFrame(A = 1:4, B = 4.0:-1.0:1.0)))
    catch
      print("package is loaded, why can't a dataframe be defined? \n \n")
      #eval(Expr(:import,:DataFrames))
      #error(return convert(DataFrames.DataFrame, rand(2,2)))
      print(DataFrames.DataFrame(A = [1, 2], B = [4, 5]))
    end
  end
  #return convert(DataFrames.DataFrame, obj)
  #Feather.Feather.write(tf, p)
  #return "numpy.asmatrix(read_dataframe(r'" * tf * "'))"
end

mat2df()

ERROR: LoadError: MethodError: no method matching DataFrames.DataFrame(; A=[1, 2], B=[4, 5])
Closest candidates are:
  DataFrames.DataFrame(; kwargs...) at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:98
  DataFrames.DataFrame(!Matched::Array{DataType,1}, !Matched::Array{Symbol,1}, !Matched::Array{Bool,1}, !Matched::Integer) at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:132 got unsupported keyword arguments "A", "B"
  DataFrames.DataFrame(!Matched::Array{D<:Associative,1}) where D<:Associative at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:157 got unsupported keyword arguments "A", "B"
  ...
Stacktrace:
 [1] mat2df() at /Users/manchongleong/testJ1.ji:26
 [2] include_from_node1(::String) at ./loading.jl:569
 [3] include(::String) at ./sysimg.jl:14
 [4] process_options(::Base.JLOptions) at ./client.jl:305
 [5] _start() at ./client.jl:371
while loading /Users/manchongleong/testJ1.ji, in expression starting on line 34

@HenryLeongStat
Copy link
Member

HenryLeongStat commented Nov 30, 2017

I think the method mentioned in this discussion might work.

It says that we can wrap the function and the package in a new package. For example,

module __Read_DF
  export __read_DF
  using DataFrames
  function __read_DF(...)
        return ...
  end 
end
using __Read_DF

@BoPeng
Copy link
Contributor Author

BoPeng commented Nov 30, 2017

The invokelatest problem is fixed here but perhaps not in Julia yet.

@HenryLeongStat
Copy link
Member

I see, they haven't released the new version?

@BoPeng
Copy link
Contributor Author

BoPeng commented Nov 30, 2017

I do not understand the new proposal. Would not DataFrames be loaded when __Read_DF is loaded?

@HenryLeongStat
Copy link
Member

Seems not working, because when the module is defined, the syntaxes inside would be executed. I thought it would be like a function, and the syntaxes wouldn't be executed when it defines.

@BoPeng
Copy link
Contributor Author

BoPeng commented Nov 30, 2017

I just marked this ticket as help wanted. Let us ignore it and see if some Julia expert can help.

@HenryLeongStat
Copy link
Member

OK. I think a lot of julia users have the same problem as well. 😄

BoPeng pushed a commit that referenced this issue Nov 30, 2017

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
@HenryLeongStat
Copy link
Member

Now need to fix Python to Julia

@BoPeng
Copy link
Contributor Author

BoPeng commented Nov 30, 2017

What python to Julia? You can close this ticket if the specific issue is resolved.

HenryLeongStat added a commit that referenced this issue Nov 30, 2017
@HenryLeongStat
Copy link
Member

Test fails:

    def testGetPythonMatrixFromJulia(self):
        # Python -> Julia
        with sos_kernel() as kc:
            iopub = kc.iopub_channel
            # create a matrix 
            execute(kc=kc, code='''
import numpy as np
mat_var = np.matrix([[1,2],[3,4]])
''')
            clear_channels(iopub)
            execute(kc=kc, code="%use Julia")
            wait_for_idle(kc)
            execute(kc=kc, code="%get mat_var")
            wait_for_idle(kc)
            execute(kc=kc, code="size(mat_var)")
            res11 = get_display_data(iopub)
            self.assertEqual(res11, '(2, 2)')
            execute(kc=kc, code="%use sos")
            wait_for_idle(kc)
======================================================================
FAIL: testGetPythonMatrixFromJulia (__main__.TestJuliaKernel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_Julia_kernel1.py", line 58, in testGetPythonMatrixFromJulia
    ''')
  File "/Users/manchongleong/anaconda/lib/python3.6/site-packages/ipykernel/tests/utils.py", line 69, in execute
    validate_message(reply, 'execute_reply', msg_id)
  File "/Users/manchongleong/anaconda/lib/python3.6/site-packages/ipykernel/tests/test_message_spec.py", line 262, in validate_message
    nt.assert_equal(msg['parent_header']['msg_id'], parent)
AssertionError: 'f9d9614b-fde7fb337606b970be9a776b' != 'e2d25b4d-88c7fd586a61ea1a836fd754'
- f9d9614b-fde7fb337606b970be9a776b
+ e2d25b4d-88c7fd586a61ea1a836fd754

Does it mean it takes too long to execute the codes?

@BoPeng
Copy link
Contributor Author

BoPeng commented Nov 30, 2017

Does everything works interactively? If so, you can commit and I will have a look at the test. The test does fail from time to time due to differences in environment and it can be hard to make it work correctly.

@HenryLeongStat
Copy link
Member

Yes, it works fine interactively. I have already pushed it. Trying to fix the test.

HenryLeongStat added a commit that referenced this issue Dec 1, 2017
@HenryLeongStat
Copy link
Member

Still pretty slow... Especially for running the testing script test/test_Julia_kernel.py

@BoPeng BoPeng closed this as completed Apr 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants