Support for multi-dimension array

[HenryLeongStat]

As title...
References to https://github.com/vatlab/sos-r/issues/1

[mathieuboudreau]

Hi @BoPeng @HenryLeongStat ,

Any progress on this? We're having issues with SoS Julia right now where we can't even move a 2D array from Julia to Python, we can only move single-valued numbers. We can provide an example notebook if that is helpful.

[mathieuboudreau]

Even a 1D array is hanging for us (e.g. (1000,) )

[BoPeng]

A sample notebook would certainly help. My problem is that I have zero knowledge on Julia so I can fix it if it is a bug, but would have to ask for help, perhaps from @mathieuboudreau if more Julia side programming is needed.

[mathieuboudreau]

I'm not experience in Julia either, mostly MATLAB. haha. But we are trying to do a simple thing with Julia+ SoS, basically processing someone elses script in Julia and then plotting it in Python using Plotly. I'll get back to you with an example notebook - Julia has a few quirks.

[agahkarakuzu]

Quick note:

When casted as Array{int64,2} (column ordered array i.e. [1 2 3 4 5]), data transfer between Julia-->Python hangs, no matter how long the array is.

When casted as Array{int64,1} (row ordered array i,e, [1,2,3,4,5]) it works.

[mathieuboudreau]

^Quirks that I was talking about haha.

[mathieuboudreau]

Here is a notebook with a few examples, running on MyBinder: https://mybinder.org/v2/gh/mathieuboudreau/PhaseUnwrapping_book/julia_debug?filepath=jupyter-sos-bugs-example.ipynb

Looks like SoS can deal with Julia 1D arrays if they are columns but not rows (which might also explain why 2D arrays don't work either.)

[BoPeng]

OK, It was bug so I was able to fix it and added more tests. Since I have learned a bit more on Julia (by watching a youtube video), let me see if I can add more tests for more data types.

[BoPeng]

I have released sos-julia 0.18.3 with the fix. I did find some types that cannot be transferred from Julia to SoS, for example, mpg got from a pyarrow dataframe

%get mtcars --from R
mpg = mtcars["mgs"]

is of type 32-element Arrow.Primitive{Float64}, which I have no idea how to detect and send.

Anyway, let us fix sos-julia step by step and worry about these later. Please feel free to submit new tickets for types that do not work.

[agahkarakuzu]

Thank you so much @BoPeng, @zelenkastiot, can you give it a try with sos-julia version 0.18.3 and see how it works for you? Then we can discuss in the next meeting.

[mathieuboudreau]

Hi @BoPeng,

Unfortunately, your recent version has not resolved our issue yet. You can test it here: https://mybinder.org/v2/gh/mathieuboudreau/PhaseUnwrapping_book/julia_debug?filepath=jupyter-sos-bugs-example.ipynb

• If you comment out the Julia and Sos cell for the example with variable "d", you will see that the kernel still freezes.

I dug into your code a bit, and found some issues

• The reason it freezes for this [1 2] example (as opposed to e.g. [1, 2]) is because this case needs use DataFrames.
• Because of this, it encounters an infinite loop in your kernel here, which can be resolved (temporarily?) by simply removing the useless(?) while True condition. I think the reason why it loops infinitely is because DataFrames isn't loaded succesfully.
• If you do that quickfix, we'll still encounter an issue, because in the part of your code where you add packages like DataFrames, you actually don't import Pkg before calling Pkg.add(), so this needs to be added to each of the catches.
  • Note that this bug will always be missed in your CI tests, since you manually install them in the travis config file here. Because of that, I'd recommend that you remove these from the travis file so you catch the bug mentionned above in your tests, to ensure you handle that edge case well (which will happen to every user that don't add those packages beforehand, like us).

Even with all the fixes mentioned above, I still get an error, and that's where I'm currently stuck. I just get an error saying Failed to evaluate '"SOS_JULIA_REQUIRE:dataframes"': invalid syntax (<string>, line 1) Unrecognized return value of type type for action %put which I don't know how to debug.

Hope some of this info helps.