Plans for fully functional MacOS and Windows implementation for compatibility

[vsivsi]

Hi @dwhitena, I'm putting the finishing touches on my review of your JOSS submission, and everything is looking pretty good.

I do have a couple more general questions regarding your forward-looking plans for the project.

I didn't have any trouble installing or using the software through Jupyter Notebook or Nteract, which is great. But as a developer/scientist who uses MacOS as my desktop OS and uses Golang for systems level work, but not yet too much for data analysis, visualization, etc, I was excited to find this project a few years ago and watch it develop over that time.

It appears that you have done great work here in making it work as well as it does, but I'm left with the sense that this is still pretty unfinished from the perspective of someone who's used Jupyter/Nteract with Python, R, Julia and/or Node.js kernels. There are two main areas of concern:

1. Thanks for your recent response to my ping on the old issue regarding embedded image outputs for visualizations. For someone coming from one of the other languages above, using Jupyter with Golang is going to seem pretty impoverished if you can't easily and interactively plot outputs of the analyses you are attempting. It's a huge part of the Jupyter Notebook experience that is just completely lacking here.

2. The limitations imposed by gomacro, most specifically the restrictions for MacOS and Windows precluding the use of "non-core" third party libraries. I'm sure that I don't need to tell you that mixing in "non-standard" libraries from a variety of sources is a huge part of coding in Jupyter Notebooks. Without this capability, the MacOS and Windows support feels more like a "toy" environment rather than a way that users can get real work done. Getting around this by hosting the Linux kernel using Docker feels like a band-aid solution at best. I'm also concerned about the lack of proper support for Interfaces in gomacro as that is such a core feature of the Go language.

I think the reality is that much of your target audience for Gophernotes does not use a Linux Desktop, and the current limitations are going to seem highly restrictive to the two main constituencies of potential users:

1. Existing Jupyter users wanting to try out Golang are going to be disappointed by the lack of visualizations and support for 3rd party libraries (in their likely desktop OS)

2. Existing Golang programmers are also going to be disappointed by the lack of support for 3rd party libraries, as well as some of the other language limitations imposed by gomacro, and will probably find Jupyter less exciting without image outputs as well.

So I guess my question is, do you have plans or a roadmap for addressing these issues?

I'm not saying that this project isn't useful without fixing these things, I'm just concerned that it is way less useful than it might be (and in comparison with Python, Julia, R, and Node.js in the same environment.) Given that, I think it's important not to oversell the capabilities of this package until these concerns have been addressed, for fear of turning people off to using Golang in this way. I think we often get one shot to sell busy/influential people on something like this, and if they try it and it doesn't seem very useful, they're likely to (perhaps unfairly) write it off in the future.

Anyway, I'd like to hear your thoughts about this, because I really would like to see Golang succeed in this area, because in so many ways it is clearly superior to the other languages I've mentioned above.

[dwhitena]

Hi @vsivsi, and sorry for the delay here as I was a bit sick with the flu. Great summary of the current state of things here, and I totally agree with most of your conclusions. Regarding your specific questions:

1. The plan around visualizations is to bring in some of the work of the neugram and go-interpreter team who have already got plotting to work in there experimental notebooks. We should be able to take some of their messaging logic and get that to work. Just need a free weekend or two hopefully.

2. The plan around 3rd party library support for Mac etc. is to test the kernel with 1.10, which was just releases. This should solve the problem actually because it extends the plugins package to Mac etc.

I also agree that these things have held back Go usage in Jupyter for a while. I think we have a better foundation right now, but I also respect your opinion that maybe further marketing around Go in Jupyter should wait for these points to catch up. Hopefully that will be soon.

[cosmos72]

Hi all,

I am gomacro author, and I would like to add some considerations:

1. 3rd party library support for Mac OS X: in theory go-1.10 extends the plugin package to Mac OS X. In practice, it still seems buggy - I could not load even an "hello world" plugin with no dependencies - see Mac OS X (darwin): support import of 3rd party libraries at runtime using 'plugin' cosmos72/gomacro#12. Go team is aware it has issues, and further work is planned for go-1.11 - see plugin: darwin support is still fragile golang/go#23369
2. 3rd party library support on Windows: I am not aware of Go team plans to extend the plugin package to Windows. Without plugins, 3rd party library support would be really hard to implement.
3. interfaces: in January, gophernotes switched to the "fast" gomacro interpreter. It is more complete and partially supports interfaces. Quoting https://github.com/cosmos72/gomacro#current-status:
   "interpreted interfaces. They can be declared, but nothing more: there is no way to implement them or call their methods. Note: interfaces imported from compiled code are fully functional, and interpreted types can implement them."
   I know how to fully implement interpreted interfaces, and I will work on it - it's tricky and has an ugly corner case, but it can be done.

[cosmos72]

Hi again,
one more consideration if you don't mind:

since Go is a purely compiled language and does not have a native REPL, nor an official or de-facto standard interpreter or JIT, the task that gophernotes tries to accomplish is significantly more difficult than for other languages with a native REPL or an official interpreter, JIT or similar:
it has both to create a Go REPL and to fully integrate it into Jupyter Notebook or Nteract.

I am honored that gophernotes authors chose my gomacro as the Go interpreter to use, yet I must point out that gomacro is a one-person project, and is not fully complete.

The reason is simple: while creating a toy Go REPL is easy - just spawn a "go run" each time - creating a real Go REPL with 100% language support is hard, and some features are very hard to implement, especially if you want interpreted code to freely interact with compiled code.
Go interfaces, to name one.
That's also the reason why Github is full of half-finished Go interpreters.

The only project I know that claims a Go REPL with almost 100% language coverage is llgoi: https://groups.google.com/forum/#!msg/golang-nuts/tBzpDtzNXq8/nUZDKx7J3fIJ
It's an LLVM-based Go compiler that emits compiled code in RAM then executes it.
It's only for x86-64 Linux.
After digging a little, I found its new, official home: http://llvm.org/svn/llvm-project/llgo/trunk/docs/llgoi.rst

[glycerine]

I have great respect for what @cosmos72 has achieved with gomacro. Like many, sadly the licensing was a showstopper for my use. I wish gomacro was BSD or MIT licensed.

I recently created a Go interpreter, gijit; https://github.com/gijit/gi. gijit provides a Go REPL with OSX, Linux, and Windows support. It is 3-clause BSD licensed, like Go itself.

gijit reuses the Go front end parser and go standard libs to parse the full language. Import support --and goroutine support -- is immature, but the rest, including interfaces, defer, recover, pointers, embedding, etc. is solid. (Credit note: this is due in no small part to deriving from GopherJS, a very complete Go-source-to-Javascript-source compiler, and to targeting a mature backend, LuaJIT.)

[cosmos72]

As I wrote privately to @glycerine, sorry for not mentioning gijit. It is certainly a worthy project and its speed is genuinely impressive 🥇

While waiting for fixes to plugin.Open() on Mac OS X, I added a workaround that, while cumbersome, allows import to work on all platforms. See https://github.com/cosmos72/gomacro#importing-packages

If you think it might be useful for gophernote users, consider pulling the most recent gomacro.

Note: gophernotes will need to change at runtime the global variable gomacro/base.GomacroDir from its default value "github.com/cosmos72/gomacro" to "github.com/gopherdata/gophernotes/vendor/github.com/cosmos72/gomacro"

[cosmos72]

Update:

1. support for visualization is complete and working in a separate branch https://github.com/cosmos72/gophernotes/tree/display_image
   I am discussing the final "aesthetic" touches with @SpencerPark , see display images, HTML... and MIME rendering #105
2. 3rd party library support for Mac and Windows is still blocked by the current approach based on plugin package from Go standard library: it is very bugged on Mac, and not supported on Windows. The workaround described above is probably too cumbersome for general use: it requires recompiling gophernotes after importing a 3rd party library

[cosmos72]

Update: 3rd party library support for Mac seems to work with Go 1.10.3
Please test and send feedback :)

[cosmos72]

Update: 3rd party library support for Mac seems still unreliable (sometimes it works, sometimes crashes) with Go 1.10.2 and 1.10.3

Please test, possibly with both Go 1.10.2 and 1.10.3, and send feedback :)

[cosmos72]

Mistery solved: on Mac OS X, plugin requires Go 1.10.2+ and to never run strip gophernotes.
Under these conditions, import of third party packages at runtime works :)

[cosmos72]

Update:

• displaying images, HTML, LaTeX etc. is implemented and merged in the master branch, see https://github.com/gopherdata/gophernotes/blob/master/examples/Display.ipynb . I also recently added auto-rendering interfaces (described at the same link)
• support for interpreted interfaces is almost complete. There are a pair of unsupported case (seldom used in practice in a notebook): type switch and type assertion from interpreted interface to interpreted interface. It's possible, although definitely complicated, to implement these cases.
• loading 3rd party packages on Mac OS X is stable on Go 1.10.2+ - just remember to never run strip gophernotes
• loading 3rd party packages on Windows is still very cumbersome, requires recompiling gophernotes as described above.