WebIDE Integration #1
Comments
|
@FWeinb - Great start. I'm glad it worked for you too.
Yes, I expected this would be something we need to work on. It it is a little tricky. Let me explain. The upload of the mod is separate from installation of the mod. The install can only happen on a clean boot of a virtual machine (because the mod's symbols need to be remapped). That's why the virtual machine always restarts after installing a mod. Restarting reboot to the device, which drops all network connections as Wi-Fi is restarted too. The mod is then run after restart. In theory, we could avoid dropping the network connection by not restarting the microcontroller and only restarting the virtual machine. But, that is complicated and unlikely to be stable (all running services need to be cleanly exited, all memory freed), so I don't particularly want to explore that route. Instead, I propose we rearrange the flow. Currently What do you think?
I'm not sure how to solve this. Chrome's move to treating http like a pariah on the web has very unfortunate consequences for IoT. We are bumping into one here. As I understand it, we cannot use public certificates here because ".local" addresses are not owned by a single entity. We could use a private certificate (with a great deal of work on the device side...) but that requires the user to install the private certificate into the browser (or machine running the browser) which works against the goal of having the WebIDE "just work" with arbitrary devices. Ideas? |
|
@phoddie The mixed-content policy is very hard to work around. It is very strange that there isn't a way to ask the user for permission to establish an unsecured connection. For now it would be possible to host the WebIDE on plain |
|
I'll make the changes to I am sympathetic to the challenges of browser security policies. I'm optimistic that we'll figure something out. That said, web security is security for the web, not the internet. While I am glad the browser vendors are working to improve the security of the web, IoT should not be obliged to support every decision made for the web. That would lead to a significant increase in complexity and cost for IoT. |
|
@phoddie To implement the stepping debugger I need to refactor the WebIDE a lot. The file handling isn't optimal and I need to think about how to switch between the debugging and editor view. I don't have that much time to work on it though. @lprader |
|
@FWeinb - Nice! Instrumentation is one of my favorite features. It is essential for embedded development. Happy to see it arrive so early here. ;) I've committed changes to support running the mod when the debugger connects, as we discussed above. Let me know if it works for you. The docs are up-to-date with the details. Regarding the workshop, our Mozilla friends invited us to help out with two workshops they are holding at JSConf EU. We would introduce the Moddable SDK as it relates to the Mozilla Web of Things initiative. That is in early June. It makes an interesting (if ambitious) goal, and perhaps in your part of the world? |
|
@FWeinb yes, definitely still interested in the potential for this in workshops! The most time consuming part of every workshop is getting all the necessary drivers and libraries installed and Moddable SDK tools built on everyone's laptop, so the WebIDE is really great since it cuts that part out of the process. The ability to load gists is perfect for that setting too. There are a few features I can think of off the top of my head that would be useful:
Like Peter said, there's JSConf EU in June. I'll also be running a workshop in late May, which would be an even more ambitious goal than that :) |
|
@phoddie Great will have a look at it as soon as possible. JS Conf EU sounds exciting, I am living in Germany so definitely in my part of the world but I am currently still a student (last semester) and do not know if I could find the time and money to go there. @lprader Building the stepping debugger is a huge task with a lot of ui design involved and would definitely take up the majority of my time. Late May early June is definitely ambitious but I think if we cut back on the debugger and focus on the features you mentioned we could have a useable WebIDE for the workshops. |
|
@phoddie main.js@@ -97,20 +97,20 @@ class ModServer extends Server {
case 6: // request body received
if (undefined !== this.position) {
trace("installed mod\n");
- this.server.restart();
}
break;
-
case 8:
return {headers: ['Access-Control-Allow-Origin', '*', 'Access-Control-Allow-Methods', 'GET, PUT'], body: "done\n"};
+ case 10:
+ if (undefined !== this.position) {
+ this.server.restart();
+ }
+ break;
}
}
restart() {
- Timer.set(() => {
- trace("restarting\n");
- this.close();
- restart();
- }, 5000);
+ trace("restarting\n");
+ restart();
}
}It is working but the ESP is crashing instead of restarting cleanly: |
|
Yikes. I'll take a look over the weekend. FWIW - usually some time is needed before restarting to allow the HTTP request to finish. Without that, the installer (IDE) cannot reliably receive the HTTP response code that the upload succeeded. |
|
@phoddie |
|
Thanks. I was thinking along the same lines on my drive to the office. I will try closing the listeners immediately but waiting to restart until the HTTP install connection is complete. |
|
I did a rework of the ui and added the toolbar from vscode to separat the editing and debugging. I toke the moddable logo and did a little modification and integrated it, if that is something I should not have done I will remove it:
The debugging button in the toolbar will reflect the connection state by displaying a little coloured dot in the corner. Green means connected, blue connecting and red a connection error. The instruments data is still just dumped into the UI, I will create something similar to what is found in xbug:
The next step will be to update the file explorer to support basic file operations. That will take a little more time because I have to improvement the internal state structure for that. |
|
@FWeinb - Nice progress. The logo is fun, but we probably should not break the Moddable logo into component parts. I have a bunch of changes nearly ready for you to try (I'll post again when they are live). I believe they give the behavior needed for the WebIDE -- the WebSocket receives a close message when restarting and no new incoming connections are accepted once an install, or any other operation that requires a restart, finishes. Further, the restart only waits 5 seconds when the network is slow. It usually is able to cleanly restart almost immediately (once any active HTTP requests are complete). In addition, the changes fix crashes on ESP8266 and ESP32 (very different lwip in the host platforms requires subtle changes in the Moddable client code...). Still, a deploy & debugging flow is a little tricky over the network, all the more so to work on a microcontroller. I imagine we will refine it further in the future. |
|
@phoddie I am very excited to work on the WebIDE more. |
|
Changes are pushed:
I hope the crashes have been eliminated, though they were always sporadic for me so it is difficult to be sure. |
|
@phoddie Great work, got it working in the WebIDE. Also got some improvements into the debugging, very crude UI but it is getting there: On thing I have noticed it that if the debugging is stopped for to long the ESP8266 is crashing. Also added some very basic file adding/removing: These icons are only shown on hover, to not distract too much from the editor. |
|
Great to see the pieces working together. I'll look into the crash. In my test, I never stop long at a breakpoint. It may just be the watchdog timer. |
|
I can't seem to get a crash when stopped at a breakpoint. I modified xsbug.js to wait two minutes at a break and continue. That works: How can I reproduce the problem? Note that while stopped in xsbug, the http server is blocked since that is hosted by the virtual machine being debugged. Consequently, while stopped at a breakpoint none of the commands to install, restart, etc, will respond. |
|
Thanks. Indeed, closing the browser while at a break caused problems. Fixes forthcoming - will let you know when everything is in place to try. |
|
@phoddie
I don't now if this is necessary but folder support would be nice. A way to download the code in a zip with all the files to compile it locally would be nice to. |
|
The upload works. While it seems to build the uploaded file, I couldn't seem to get it to execute -- the changes I made to mod.js to invoke it didn't seem to get deployed. I'll have to experiment with that further. |
|
There are some changes to help with the debugging connection reliability you reported. You'll need to update the Moddable SDK and then apply the latest xsPlatform.c/h patches from the
|
|
@phoddie I need to completely rewrite the file handling because it is currently very messy, that part was written as a first draft.
|
|
Very happy to hear the debugger connection behavior is trending in the right direction. I committed a small change to the I was able to drag & drop a file and have it work as expected. Edits redeployed too. Very nice. I did run into one hiccup worth noting. The file I uploaded first was named |
|
@phoddie The latest changes to The compiler is working with the full file tree but I think that I also modified the gist loading to just import the files into the current workspace (warning before overwriting files). |
That's troubling. I assume you are referring to the changes of April 10. Those are relatively monitor. I recall you said the changes of April 9 were working well for you. I'll take a look.
I'm not sure I understand. If you would provide instructions for how to see the problem, I can take a look. |
|
@phoddie I am currently importing „ping“ as if it where in the root directory but in the file tree you can see it is located in the „src“ directory. If you change it to „src/ping“ runmod is complaining that it can’t find the module. |
|
|
|
Regarding instabilities, I don't see any crashes. I do see the WebIDE waiting. I was able to resolve that by being less aggressive about restarting the microcontroller to allow the TCP socket to close cleanly. Here's a quick hack to try to see if that also addresses the problems you observe. Change line 122 of main.js from to |
|
Excellent. I'll try to have it committed before the weekend. |
|
Changes are committed. The docs have been updated as well. The patches changed, so be sure to reapply those. It is a lot of changes, so it wouldn't be surprising if there are problems. Let me know how it goes. |
|
(Now with ESP32 support) |
|
@phoddie
|
|
@FWeinb That's great! I'm glad it wasn't too tough to make the changes. I will try it out a bit later. Apologies for my oversight on the WebSocket constants. Those will be in our next Moddable SDK push. FWIW -- we have some progress on the wasm tools work. That should be available soon. |
|
@FWeinb - Just a quick note here that we've pushed an update to the Moddable SDK with a substantial revision to the wasm tools support. I've opened a separate issue for discussion of that. It includes a link to documentation as well. |
|
@phoddie that looks great! I will look into it and try to update the WebIDE with it. Might be a good idea to make that manifest file editable from within the WebIDE too. Really great to see your efforts in pushing these WASM tools. |
|
@phoddie As always, I made a little video showing it in action (click on the image). Creating breakpoints from within the editor and sending them to the microcontroler will be my next objective. |
|
Superb! And it is fast too. |
|
While implementing this I noticed
As you said. Everything did turn out pretty fast in the way the debugging connection is handling all the traffic while toggling the debug frame information. Good idea to eliminate the http server and only have the web socket connection! |
|
All clear points. I will look into those and get back to you.
|
In my testing, this works. I tested with the If you still see the problem after updating, please let me know the steps to reproduce it.
The microcontroller was not handling the close request from the browser. The result was that the microcontrollers believed it was still connected and the browser was waiting for a reply to its close request. WebSockets close is now implemented. I was able to connect and disconnect several times in row without restarting.
This one is complicated. I agree that a delay isn't a great approach. Let's try to eliminate that. After you send a restart, the WebSocket connection should cleanly close. If you wait until the web socket close is received, the device's web socket listener should be gone too. Terminating the listener on receiving the restart command is tough, because it is owned by the application script which is many layers away from the debugger bridge. I'll think about that some more. |
|
Thanks for looking into this Regarding that example: As I am thinking about it my initial idea was to remove the need for doing a doStep there to conceptually split the concerns of the installation process from the debug process. I might not be explaining this good enough. Just from a conceptual standpoint it is strange to have to do a doStep before you can install a new module onto the device, by mixing two concerns debugging and installing. The last command
Great! In the next update of the WebIDE I will close the connection to the device. |
The communication protocol does not require you to stop in the debugger in order to install. I gave that example as I understood your previous comment as indicating install did not work when stopped. That said, I think you should. Depending on the host, you may be overwriting the executing byte code for the currently running mod. That is effectively corrupting the running code and will surely lead to problems. On the ESP8266 it is currently safe because the upload goes to a separate section prior to being installed, but on the ESP32 the upload goes directly to the location where the code executes. They are different because of architectural differences in the hardware.
Install does not perform a restart. The install over http did restart, so perhaps you are recalling that. I changed it in the WebSocket version to give the IDE more control over the process (e.g. perhaps after install it wants to do something else, like update a preference, before restarting). |
Okay that makes sense. Let’s keep it as is.
Yes I did mix that up. |
My feeling is that a debugging interface is inherently dangerous because it needs broad access to do its job. There are lots of ways to crash or otherwise cause havoc. I'm not sure it is possible to make it safe in all case. I fear it could be a lot of work to try. At the moment, my focus is making the debug inteface functional, reliable, and fast. That's already challenging. Any tool using the debug interface (e.g. WebIDE) needs to be aware of what is safe and implement against that. And we are still discovering what is safe. ;) |
|
@phoddie |
|
Very nice. It works just the way it should. ;) |
|
After some testing today I ran into an issue that I can't solve. The Moddable Zero is always crashing after uploading the I tried to not run the mod on boot but still after trying to flash it via the WebIDE the ESP is crashing. |
|
The first stack trace is the installation of a new mod after rebooting ( The second trace is less obvious. It looks like it is trying to load and run the mod ( If you haven't already, try resetting your flash and reinstalling. It shouldn't matter but... And just in case... I posted a fix a few days ago to a bug that corrupted mod installs. Please make sure you are running with that. |
|
@phoddie |
|
@phoddie
|
|
Nice. First feature debugging feature not also in xsbug. |
|
@FWeinb - Both of us have occasionally encountered situations where a mod is corrupt. This seems to happen when the transfer is interrupted -- for example by unplugging the device in mid-transfer. The transfer is usually so fast that there isn't much opportunity to happen, although over Wi-Fi stalls do happen so it is more likely. Yesterday, I found a bug in the SPI read/write functions that caused occasional crashes while uploading a mod. I fixed that and will push in the next day or two. That got me thinking about how to ensure an interrupted mod upload doesn't leave a partially installed (and consequently corrupt) mod in place. The solution I found can be implemented in the WebIDE without a change to the device code. It relies on writing the device header only at the end of the transfer, rather than at the start.
|
|
@phoddie |
|
@phoddie I guess I have circled in on my problem regarding malformed xml data. If you set a breakpoint (e.g. by adding one using the WebIDE) and re-upload the same mod, in 99% of all cases I will get malformed XML from the device. No idea if this is related just wanted to let you know. |
A (more or less) reproducible case is great. I assume this is only over serial/USB, not WebSockets. Once you upload the mod, you then restart, I assume. What are the commands that get sent after that? |
|
I ran into a situation this weekend where during the mod upload I was also configuring the debugger (setting breakpoints, etc). That was too much over serial for the ESP8266 and bad things happened. I fixed it by making sure that during an install nothing else was happening over serial. Since after installing the next step is rebooting, that isn't a problem. Perhaps related to your serial corruption problem. |
|
@FWeinb - I just pushed some changes.
If you use the load module command, we can simplify runmod/main.js to this: |
As discussed in Moddable-OpenSource/moddable#132 I am opening an issue here.
I just had a minute to test the integration of the new WebSocket-Debugger into the WebIDE. Looking well so far I started by just piping in the device log into the log area:
What would be great is a way to detect when the mod was uploaded and run. I am currently querying
/mod/installbut it returns as soon as the upload was completed.Another problem is the mixed-content problem regarding https/http on resources hosted on the device itself. Chrome is blocking these requests and you need to manual allow loading the execution. This is problematic for uploading and connecting to the unsecured websocket connection.
The text was updated successfully, but these errors were encountered: