WIP: Support Pyodide in Jyve #12

mdboom · 2018-04-18T16:22:33Z

This is just a proof of concept at this point, but I think that by sharing it I might spur some interest and collaboration.

This adds a new kernel to use Pyodide, a port of CPython and Numpy (and soon Pandas) to WebAssembly. See blog post for more info.

Thanks for Jyve, by the way. It made doing this kind of thing surprisingly easy. Most of this PR is copy-and-paste-and-search-and-replace on the brython extension.

bollwyvl · 2018-04-19T01:07:59Z

Fantastic stuff and welcome to Jyve! Oh, and thanks for all the open source!

I've got this PR running locally in FF with only a minor tweak. That Other Browser is complaining about RangeError: WebAssembly.Compile is disallowed on the main thread, but it also occurs over on the iodide demo, so it might be A Thing.

Though the integration won't be as magical as the iodide experience, no doubt, this is a really slick view of the technology.

I'm glad/sad the fractal rat's nest of loading stuff in the iframe from the brython hack was useful as a template, and have been trying a few other more involved things like fengari-web to see how far this whole thing can be pushed.

Anyhow, I'll pick some wee nits in the diffs, but really look forward to getting this in!

bollwyvl · 2018-04-19T01:08:53Z

README.md

@@ -63,6 +65,7 @@ jupyter labextension install \
  @deathbeds/jyve-lyb-phosphor \
  @deathbeds/jyve-p5-unsafe-extension \
  @deathbeds/jyve-typescript-unsafe-extension \
+  @deathbeds/jyve-pyodide-unsafe-extension \


sort (or meh, do later)

bollwyvl · 2018-04-19T01:09:10Z

index.ipynb

@@ -55,7 +55,8 @@
    "- [CoffeeScript](./notebooks/CoffeeScript.ipynb)\n",
    "- [TypeScript](./notebooks/TypeScript.ipynb)\n",
    "- [Brython](./notebooks/Brython.ipynb)\n",
-    "- [P5](./nodebooks/P5.ipynb)"
+    "- [P5](./notebooks/P5.ipynb)",


add trailing \n

(in the quote)

bollwyvl · 2018-04-19T01:10:04Z

notebooks/Pyodide.ipynb

+   "outputs": [
+    {
+     "data": {
+      "text/plain": [


I freaked out a little when i read this on my phone

bollwyvl · 2018-04-19T01:13:02Z

package.json

@@ -34,7 +34,7 @@
  },
  "scripts": {
    "bootstrap": "jlpm && jlpm lerna bootstrap && jlpm clean && jlpm build && jlpm lab:link",
-    "build": "jlpm build:core && jlpm build:js && jlpm build:coffee && jlpm build:typescript && jlpm build:brython && jlpm build:p5 && jlpm build:ext && jlpm build:lyb",
+    "build": "jlpm build:core && jlpm build:js && jlpm build:coffee && jlpm build:typescript && jlpm build:brython && jlpm build:p5 && jlpm build:pyodide && jlpm build:ext && jlpm build:lyb",


yeah, sorry, through no fault of yours this is getting bad... i think A Great Renaming must come, when we can use lerna scopes a bit to make this bad

bollwyvl · 2018-04-19T01:15:22Z

packages/jyve-pyodide-unsafe-extension/package.json

+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "python3",
+      "version": "ES2015"


hangover (not your fault)... really want to have these reflect the version of the implementation a la #10... and not duplicate them

So this version is supposed to be the version of Pyodide, or of the pyodide-unsafe-extension here...? Haven't made pyodide releases yet, but this will likely be a combination of the underlying CPython version and the pyodide-specific bits version.

I guess it would be the python version, e.g. what IPython puts out?

"metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } },

bollwyvl · 2018-04-19T01:31:34Z

packages/jyve-pyodide-unsafe/src/kernel.ts

+
+function bootstrap_pyodide(window: any) {
+  let baseURL = "http://iodide-project.github.io/pyodide-demo/";
+  let wasmURL = `${baseURL}pyodide.asm.wasm?x=${Date.now()}`;


In the near term, I'd love to see this hoisted as high as it can be (maybe even to package.json) and made a permalink backed by CDN, e.g.
https://cdn.rawgit.com/iodide-project/pyodide-demo/a39e3f3e80a438a07e6a8028fa4745768c16cbc2/pyodide.asm.wasm

Longer term, of course, i think we'd have to bite the bullet and figure out a way to actually

ship that and its big brother .data in an npm package

not have that get bundled into the main jupyterlab webpack bundle

One of the really nice things when demoing jyve is that going to https://deathbeds.github.io/jyve has pretty much always Just Worked... and since the Jyve can make the browser a touch... ahem... unstable, I end up having to reload a lot. The every-reload cache busting is a bummer in that way.

I think at some point, we (as in, the jupyterlab ecosystem) will need to seriously look at serviceworker/appcache approaches for some of these things.

Oh, and I guess I'd like it to work in chromium-based stuff. 😊

Interesting. Didn't know about cdn.rawgit.com. That's certainly better than the reload-forcing trick. That seems like an easy improvement to do now.

I haven't spent any time thinking about distribution and deployment beyond that. It's definitely suboptimal now.

bollwyvl · 2018-04-19T01:37:07Z

packages/jyve-pyodide-unsafe/src/kernel.ts

+  wasmXHR.open('GET', wasmURL, true);
+  wasmXHR.responseType = 'arraybuffer';
+  wasmXHR.onload = function() {
+    let Module: any = {};


So, we'll likely see this on more kernels. I'm interested in the pattern that makes this possible reliable and non-hangy. Excited to watch what develops!

Yes -- it definitely can / does hang if anything goes wrong. I think as easy win is just capping a timeout on things so at least we get control back if things fall off a cliff. Other than that, it's not clear. I know emscripten is growing some more use of Promises that might help clean things up on this side, but it's not quite ready as far as I can tell.

mdboom · 2018-04-19T17:25:12Z

The breakage on Chrome is known. I made an issue here so we can track that.

Thanks for the kudos. Glad to help out. I was expecting to have to do all this work to integrate with Jupyter and I was really excited that all this hard work had already been done, and it was basically low dozens of lines of code to get something working here.

bollwyvl · 2018-04-22T15:49:41Z

packages/jyve-pyodide-unsafe/src/kernel.ts

+
+    if (count <= 0) {
+      // TODO: Is there a way to popup an error modal here?
+      alert("Pyodide failed to load.");


This is fine for now... cracking open a dialog is pretty easy.

However, I am wondering if it wouldn't be more appropriate to show an in-notebook error, but I don't think there are any examples of that yet in jyve. I'd have to look at what a real kernel would do...

bollwyvl · 2018-04-22T16:10:45Z

I was expecting to have to do all this work to integrate with Jupyter and I was really excited that all this hard work had already been done

Thanks! Jyve was a thing that we had been kicking around for a while, and jupyterlab made it possible. One could have probably written this in notebook classic, but the fact that even my monkeypatches to Kernel, Session, etc. are mostly typed means this might actually be maintainable.

While I appreciate the boundaries we're starting to push here, I'd say pyodide has so much potential as to warrant a more serious look at all of those terrible hacks I had to do to core pieces. For example, we're trying to rethink how the jupyter.org site would work, and how to be ready to hype jupyterlab 1.0. A pyodide demo would have comparable bandwidth cost to the end user as, say, a video, and more predictable than mybinder in the face of hitting the top of key outlets.

Your dependency point is very apt about the ability to reproducibly address the dependency problem in a manner that doesn't require runtime access to npm, which all of the jyve kernels have... i've got the lyb wrappers that handle things already in the jupyterlab build, but there are so many other exciting things out there. Indeed, this is even a problem for jupyterlab itself. Especially with pyodide/brython/etc. the goal is to not make everyone become a web developer, but it's going to take a lot of work.

ANYHOOO... keep me posted as to when you'd like to land this. I'm very excited to get this into the demo site as son as it's merged, though I wouldn't think we'd want to drop an actual release of the jyve-pyiodide- packages until the chrome issue is ironed out.

bollwyvl · 2018-05-06T20:00:38Z

@mdboom I've been doing a bunch of automation-y things that might get in the way of continuing your mouth-watering pyodide integration work... I'll gin up a PR-PR and try to take care of much as i can before you have to deal with our autocratic CI masters...

bollwyvl · 2018-08-10T02:51:10Z

@mdboom I had some success (under very particular circumstances) getting the 🔥 new pyodide-demo working under chrome inside jupyterlab/jyve:

I'm still running into some issues with some of the promise timing, i think, as clearly i'm getting to run code, somewhere, but then it kinda gets lost. FF seems to never get past the second script tag being added. We shall see! Hopefully I can get it into a PR-able state, but i'm time crunched in a few directions right now!

Between serving wasm mime properly and exposing an offline jyve download UI, I've decided I'll need a serverextension (OH THE IRONY), which makes it a lot easier to consider the offline distribution story (through pypi/conda) for pyodide packages. It's starting to get a little meta, I guess, but there you go. Looking forward to making this happen for real, soon!

bollwyvl · 2018-08-11T19:47:20Z

Well, I got it working once by removing the package-loading machinery.

SimonBiggs · 2018-10-03T11:38:53Z

I just came across this. This looks exceptionally brilliant. How much further might this go?

Edit: nvm just saw the following #27

bollwyvl · 2018-10-03T12:27:56Z

Thanks for the interest! Still tinkering! Once I can get a build that passes the tests on Firefox and chrome, it will definitely be released! So far I've gotten it working on one or the other, but never both at the same time. When Jupyter lab 0.35 comes out (this week-ish), I plan to take another pass at it, as there have been many changes to both sides. Because of the sheer size of the pyodide files, getting this out will also require a server extension... This is a bit unfortunate, but lab's webpack MUST NEVER KNOW about these huge things. The upside is that I will be forced to finally ship the nbconvert profile that generates the offline demos, which is what this is all about. Also, an author will just have to pay the download price once, highly compressed, rather than every cache miss. Where else could this go? Well, once the boring packaging stuff is squared, it's really up to users. Pyodide makes it much less of a toy, but here are some ideas that will work with any of the kernels... Or just lab extensions: - in addition to a binder, labextensions and themes can add a jyve demo to their GitHub pages - A built Jyve lab will be a great match for distributed tech like ipfs and dat: indeed, the contents manager can be entirely replaced with these, meaning a jyve build can update at chat speed - papers, posters, and books can include a qr/url that readers can use to get to an interactive state as quickly as their network can resolve it, without the author having to pay for a k8s cluster - other js/html apps can embed a little bit of lab, and interact with it - another publishing mode could hide most of the chrome, and just give your users the kernel iframe, making games and experiments easily deployable Stay tuned!

SimonBiggs · 2018-10-03T12:43:32Z

Awesome! So, I am seeing if I can get my head around this and potentially port the ScriptedForms JupyterLab extension over to be primarily based on Jyve + Pyodide. Many end users in my industry (medical) have locked down machines and installing Python is exceptionally difficult for them (read essentially impossible). I'm trying to make an experience that is zero installation + easy to share.

...My thought was to combine ScriptedForms + Jyve + Pyodide + GitLab API to their own private repo (for file and results storage). And then create a website where people log in with a GitLab account and they can then begin creating, interacting, and sharing forms.

Does this sound achievable?

bollwyvl · 2018-10-03T17:06:05Z

Adding anything that can write to a trusted API from Jyve gives me the screaming Willies, but yes, all of that would work.

On Wed, Oct 3, 2018, 08:43 Simon Biggs ***@***.***> wrote: ...My thought was to combine ScriptedForms + Jyve + Pyodide + GitLab API to their own private repo (for file and results storage).

For that to work, there would need to be a gitlab idrive implementation... I know work had started, but it's not as robust as the GitHub one. Then, there would be a few ways to do this, either via a download/upload workflow, an extra server process, or by an in browser token. I don't think the GitHub one supports upload, not sure about git lab. So long as the browser token is very limited, this would work... I wouldn't want a bad actor deleting all your repos. Because, by default, you have no python process, you could add a some proxy that would handle the upload by mimicking how a server extension would respond.

And then create a website where people log in with a GitLab account and they can then begin creating, interacting, and sharing forms.

I'm not going to build a dedicated server for this purpose, but if you can hack the static output and your own labextensions to talk to your apis, you should be good.

…

Does this sound achievable? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACxRGwZ3pAMFG0UJrANaDkUK4I-epH0ks5uhLD1gaJpZM4TaX_B> .

SimonBiggs · 2018-10-03T21:13:51Z

Thanks @bollwyvl :).

I was hoping to be able to have the whole thing be static. As in have the whole thing running off of a GitHub/gitlab pages. Is that actually possible? It seems like it might be from what I can see on your demo.

Ideally the permissions for gitlab would just be for a single repo, so the two bad things I can think of that could happen would be a private repo used for this purpose having its data leaked, and the other be having it deleted. I'll have a look into how much limitation an API can have placed on it. Might be nice if repo deleting was explicitly not approved from the gitlab API key creation side.

I also do have another option, lose the benefits that come from being integrated with JupyterLab and just directly build in pyodide by itself into ScriptedForms. Would that give you less screaming willies? Some big downsides are the work I do becomes far less transferable to others. And ScriptedForms built ontop of JupyterLab if I do it well can tap into JupyterLab's extension ecosystem.

bollwyvl · 2018-10-04T00:07:31Z

Yeah, static-first all the way, glad it matches your expectation. I think a lab-forward approach is the right play, as you'll get a lot for "free," but i think my reservations are still well founded:

Looks like you can deploy a single-page-app on GitLab pages...
https://docs.gitlab.com/ce/user/project/pages/

...which would include some code to open a popup which to accept a redirect back to the app with a token...
https://docs.gitlab.com/ce/api/oauth2.html#implicit-grant

...which can then be used to upload a file (e.g. a notebook, or a json blob of response data, or whatever)...
https://docs.gitlab.com/ce/api/repository_files.html#create-new-file-in-repository

...but unfortunately, to be able to upload a file, that token can do anything that the user can do against the API, like delete all their repos. So one bad actor notebook sent out to a mass email list and 💥 no repos. There's nothing JupyterLab, the browsers, or the framework-of-the-week can do about it,

It looks like there are requests in the pipeline to allow for limiting the repo scope of a personal access token, but i couldn't find anything about the same for the implicit grant flow.

Having built some of these monsters before, i can say that it's no fun, but hopefully they'll figure it out.

Anyhow, short of writing the code, I'm happy to help in any way i can...

SimonBiggs · 2018-10-04T19:12:04Z

One option, which is a bit convoluted, is to request that users create a second GitLab account which only has develeper access, not maintainer access, to the repository where data will be stored. And then log in with that account only.

A warning like the following could be included on the login screen.

Code running within a ScriptedForm can do anything your GitLab user can do. Including deleting every repository you have delete rights to. If you ever plan to use a ScriptedForm which you have not written yourself you should create a second GitLab account which only ever is given developer rights to a repo. Instructions on how to do this are here. Note this does not stop a ScriptedForm that is downloaded from the internet stealing the contents of your private GitLab repositories. This would include all your ScriptedForms, and all the results from these. You should always trust the source of ScriptedForms that you open.

Glad you recommend the lab-forward approach. That was definitely my preferred route. And yeah, I figured I would need to use implicit grant flow given it was a static only site which GitLab (but not GitHub) supports.

Thank you for your help and directions. I think I shall prep this ship to begin sailing to its destination. I'll keep you posted.

SimonBiggs · 2018-10-05T10:40:03Z

So, I've got Jyve on my machine, I've set up the tests. Most of them are passing... the static ones are at least, so that's good enough for me. Now I have just entered the rabbit hole of https://github.com/deathbeds/jyve/blob/master/jyve/exporter.py ... that thing is magical.

bollwyvl · 2018-10-05T12:09:32Z

Hah. There's a reason the python side isn't released yet, sorry for the dust. Feel free to suggest places where it needs to be more modular and it configurable. As suggested, the settings will become complex enough that it will need ui, config schema, server extension. Another kinda crazy idea was to move it into the browser. This would allow a static site to modify "itself" (an in memory version), but would probably require cutting many corners. Also it would be much harder to track down extra assets and resources. I'd also like to replace wget with the tornado http client.... it did too much for free to get started, but is an unreasonable dependency, despite conda.

…

On Fri, Oct 5, 2018, 06:40 Simon Biggs ***@***.***> wrote: So, I've got Jyve on my machine, I've set up the tests. Most of them are passing... the static ones are at least, so that's good enough for me. Now I have just entered rabbit hole of https://github.com/deathbeds/jyve/blob/master/jyve/exporter.py ... that thing is magical. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACxREFTwmTZGsiGEI9lWPr18n_9ENaKks5uhzcDgaJpZM4TaX_B> .

SimonBiggs · 2018-10-05T12:12:32Z

@bollwyvl most of this is many many levels above my head. I'm sorta just diving in and seeing if I can tread water. So I likely won't be sending many suggestions your way.

WIP: Support Pyodide in Jyve

78fee81

bollwyvl reviewed Apr 19, 2018

View reviewed changes

notebooks/Pyodide.ipynb

"outputs": [

{

"data": {

"text/plain": [

Copy link

Contributor

bollwyvl Apr 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I freaked out a little when i read this on my phone

bollwyvl reviewed Apr 19, 2018

View reviewed changes

mdboom added 3 commits April 19, 2018 13:37

Simple text fixes mentioned in PR

fc02aad

Use cdn.rawgit.com

517c5b3

Cap loading timeout

ee4b5f3

bollwyvl reviewed Apr 22, 2018

View reviewed changes

bollwyvl mentioned this pull request Aug 9, 2018

More Documentation of File Size #26

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Support Pyodide in Jyve #12

WIP: Support Pyodide in Jyve #12

mdboom commented Apr 18, 2018

bollwyvl commented Apr 19, 2018

bollwyvl Apr 19, 2018

bollwyvl Apr 19, 2018

bollwyvl Apr 19, 2018

bollwyvl Apr 19, 2018

bollwyvl Apr 19, 2018

bollwyvl Apr 19, 2018

mdboom Apr 19, 2018

bollwyvl Apr 22, 2018

bollwyvl Apr 19, 2018

mdboom Apr 19, 2018

bollwyvl Apr 19, 2018

mdboom Apr 19, 2018

mdboom commented Apr 19, 2018

bollwyvl Apr 22, 2018

bollwyvl commented Apr 22, 2018

bollwyvl commented May 6, 2018

bollwyvl commented Aug 10, 2018

bollwyvl commented Aug 11, 2018

SimonBiggs commented Oct 3, 2018 •

edited

Loading

bollwyvl commented Oct 3, 2018 via email

SimonBiggs commented Oct 3, 2018 •

edited

Loading

bollwyvl commented Oct 3, 2018 via email

SimonBiggs commented Oct 3, 2018

bollwyvl commented Oct 4, 2018

SimonBiggs commented Oct 4, 2018 •

edited

Loading

SimonBiggs commented Oct 5, 2018 •

edited

Loading

bollwyvl commented Oct 5, 2018 via email

SimonBiggs commented Oct 5, 2018

WIP: Support Pyodide in Jyve #12

Are you sure you want to change the base?

WIP: Support Pyodide in Jyve #12

Conversation

mdboom commented Apr 18, 2018

bollwyvl commented Apr 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdboom commented Apr 19, 2018

Choose a reason for hiding this comment

bollwyvl commented Apr 22, 2018

bollwyvl commented May 6, 2018

bollwyvl commented Aug 10, 2018

bollwyvl commented Aug 11, 2018

SimonBiggs commented Oct 3, 2018 • edited Loading

bollwyvl commented Oct 3, 2018 via email

SimonBiggs commented Oct 3, 2018 • edited Loading

bollwyvl commented Oct 3, 2018 via email

SimonBiggs commented Oct 3, 2018

bollwyvl commented Oct 4, 2018

SimonBiggs commented Oct 4, 2018 • edited Loading

SimonBiggs commented Oct 5, 2018 • edited Loading

bollwyvl commented Oct 5, 2018 via email

SimonBiggs commented Oct 5, 2018

SimonBiggs commented Oct 3, 2018 •

edited

Loading

SimonBiggs commented Oct 3, 2018 •

edited

Loading

SimonBiggs commented Oct 4, 2018 •

edited

Loading

SimonBiggs commented Oct 5, 2018 •

edited

Loading