[Proposal] Parsing from Flash/Disk/Transaction

[sergioamr]

With my current development i am facing a problem in which i cannot allocate enough memory to load from the file system a program that should fit on JerryScript.

Arduino 101 has 80kb of RAM (divided in two MCUs, so only 55kb are for Zephyr and JS), 256kb of ROM and 1MB of slow FLASH memory.

I store several javascript programs of different sizes on that flash which i load and run one after the other.

The current approach to load a javascript program is to copy it to memory and then call the parser.

jerry_parse (const jerry_char_t *source_p, /**< script source */
  size_t source_size, /**< script source size */
  bool is_strict) /**< strict mode */

That means that the whole code has to be loaded in memory in one go, so i have to allocate it.
That memory will be freed instantly after the parser has finished.

I am not sure how feasible would be to have a simple API to pass a function that will do the reading for jerryscript and implement the fread functionality on the port, abstracting the reading from the memory to a callback would allow us to load from disk bigger programs and save memory usage.

[zherczeg]

What would be the API for this? I think the challenge of this task would depend on it. Perhaps a readbyte / peekbyte / seek probably would be the smallest one, but also could be a considerable performance overhead.

[zherczeg]

I have an interesting idea. Most JS source files can be split into smaller files, which can be parsed / executed independently. E.g. an a.js could be split into a1.js a2.js, a3.js ... Would such splitting solve your problem?

[zherczeg]

Come to think of it, snapshot could also be a solution to this problem. Or is there a blocker for not using it?

[weilianglin]

Yes, the bytecode in snapshot can be interpretered in Flash. To futher reduce the memory, all strings can be converted into external magic strings. So that we don't need to load strings into the memory.

[zherczeg]

So does it solve your problem?

[sergioamr]

@zherczeg Snapshot doesn't really solve my problem. I don't have the snapshot functionality due lack of space, since i really need the eval and JavaScript parser, since you can evaluate commands from the web editor and retrieve values from the different pins and execute code stored in the browser to access different sensors and examples.

I could push all of that in snapshot format but the user can evaluate and run javascript programs in the command line, so i still need the parsing and evaluation.

My current development involves uploading the code into the file system from a browser and running it from disk. I also retrieve the code back from the device to the web editor so it is only stored currently in the flash file system.

I also have a command line from where you can edit the code and save before running from within the device.

I am currently trying to free some memory, or I will have to allocate less heap for Jerryscript. I am not sure what is the minimum Jerryscript can have to be functional.

Dividing in different JS will not solve my problem, this is an education tool which the target group would not understand how to create a modular program yet, since they learn it basically as a Sketch from Arduino but in JS. As far as I know, we don't also have a system call to run the next program anyway.

So i guess, it doesn't work for me 💃

[zherczeg]

Would it help if you would create a jerry string from the source code? This way you would not need extra memory besides jerry heap.

[sergioamr]

That would be a good first step. Is the parser going up and down the program while parsing?

I actually should have a look to the code to see how unreasonable this question is.

[DemiMarie]

Lua implements this using lua_Reader.

[rerobika]

Static snapshot was introduced in #2239. The "core" of the code can be pre-complied and loaded/executed to/from the ROM so no need to load it into the RAM which saves a lot of memory. However, you cannot bypass the cost of the parsing of the incoming sources from the web editor but, you can use the "saved memory" for this purpose. IMO we can finally close this issue.