[FEATURE REQUEST]ESP3DLib 3.0

[luc-github]

Here features that will be covered by ESP3DLib 3.0 :

• Webui 2.1 3.0 support
• Move to SSDP asyncudp library
• Move ESP32 core to 2.0.2 / 2.0.3
• Move from SPIFFS to LittleFS
• AP mode setup reviewed
• New embedded page
• Add Telnet server support
• Add Notifications support
• Add MKS Cura pluggin support for file upload
• Move SD support from Marlin SD Lib to SDFat 2.0 for better performance (TBC),
  Note: Marlin itself will still use Marlin SD Lib as only ESP3DLib will be affected.

Testing boards:
1 - MRR_ESPA V1.3 without display
2 - FYSETC V1.0 + MKS TFT32_L V4.0 (With BTT FW)
3 - MKS TinyBee V1.0 + MKS Mini12864 V3.0
4 - Panda Zhu V1.2

Marlin Code base should be 2.0.9.3 bugfix 2.0.9.4? (TBD)

[luc-github]

Some update:
1 - Use ESP3D 3.0 as code base is almost done - what is not yet implemented is the SD upload
2 - Use of SSDP asyncudp library is implemented but I got some random (refering to esp_timer) crash and also some Watchdog reset on ESP3DLib task , even without the SSDP library so I need to dig in
using disableCore0WDT(); in Esp3DLib seems mitigating the issue for the moment but need to investigate
3 - Moving Marlin to latest ESP32 core git version is done (2.0.9.3+) - I did some change to make it support it in Marlin sources
4 - Moving from SPIFFS to LittleFS is also implemented
5 - Telnet is implemented
6 - Notifications are implemented as part of ESP3D 3.0 code but not yet tested

I have currently only tested with MRR_ESPA V1.3 without display - I still need to adjust code with display and with serial display

WebUI is finally the 3.0 as even it not yet finished it will be better to focus on latest versions
since use of latest Marlin git seems supporting long name I will also spend time to add it

#define LONG_FILENAME_HOST_SUPPORT    // Get the long filename of a file/folder with 'M33 <dosname>' and list long filenames with 'M20 L'
#define LONG_FILENAME_WRITE_SUPPORT   // Create / delete files with long filenames via M28, M30, and Binary Transfer Protocol

That said, I need to see how to handle Marlin with Long name support and Marlin without long name support in WebUI 3.0 refactoring

So it should save time to stay on WebUI 3.0 and ESP3DLib may directly jump to 3.0 as final version

Need to spend some time on stability issues before move to SD support then WebUI Update

[luc-github]

Here the base for the display message output

Some screenshots

[luc-github]

I spent all week on random crash and I think I worked around it, I still need to test AP mode in long run but I do not have anymore issue in STA mode so far (TBC)

For some reason having webserver running on task on Core0 is randomly triggering WDT - I have disabled the watchdog for the ESP3DLib task but it still randomly trigger WDT on core 0, one solution seems to disabling watchdog when handling webserver

disableCore0WDT();
_webserver->handleClient();
enableCore0WDT();

So I will use it as workaround from now, and continue porting
next step is Display output : direct and serial, then SD support, once done I will go back to WebUI rewrite to implement missing panels.
so far so good ^_^

[luc-github]

Ok final workaround is now validated instead of using task wdt feeder I use global wdt feeder and it seems ok after one week of long run test
Display output is now properly managed
Now working on SD Support - I wish I can use the SDFat instead of Marlin SDLib which is very slow for upload - it seems Marlin support now long name in command but no more subdirectory files for print command: luc-github/ESP3D-WEBUI#221 (comment)

[luc-github]

Seems use current ESP3D V3 code using SDFat V2 fit with existing Marlin SDLib

>[ESP740]
Directory on SD : /
<DIR> 	overlays 	
<DIR> 	System Volume Information 	
<DIR> 	Archive 	
<DIR> 	New folder 	
      	bcm2708-rpi-b-plus.dtb 	13.94 KB 	
      	COPYING.linux 	18.25 KB 	
      	LICENCE.broadcom 	1.46 KB 	
      	bcm2709-rpi-2-b.dtb 	15.00 KB 	
      	bcm2708-rpi-b.dtb 	13.68 KB 	
      	bcm2708-rpi-cm.dtb 	13.64 KB 	
      	issue.txt 	145 B 	
      	CUBE.GCO 	205.31 KB 	
      	CUBE01.GCO 	1.99 MB 	
      	SUPPORT2.GCO 	4.40 MB 	
      	bootcode.bin 	17.51 KB 	
      	cmdline.txt 	136 B 	
      	config.txt 	1.62 KB 	
      	fixup.dat 	6.47 KB 	
      	fixup_cd.dat 	2.48 KB 	
      	fixup_db.dat 	9.52 KB 	
      	kernel.img 	3.94 MB 	
      	kernel7.img 	4.04 MB 	
      	fixup_x.dat 	9.52 KB 	
      	start_cd.elf 	619.66 KB 	
      	start_x.elf 	3.73 MB 	
      	LICENSE.oracle 	18.53 KB 	
      	bcm2710-rpi-3-b.dtb 	15.62 KB 	
      	bcm2710-rpi-cm3.dtb 	14.99 KB 	
24 file(s), 4 dir(s)
Total 62.95 MB, Used 24.63 MB, Available: 38.31 MB
ok
>M20
Begin file list
CUBE.GCO 210240
CUBE01.GCO 2089832
SUPPORT2.GCO 4613256
ARCHIVE/CUBE-C~1.GCO 210240
NEWFOL~1/SUPPORT2.GCO 4613256
End file list
ok

So now need to see how to avoid conflict in usage and allow only to access SD at once - but so far so good

EDIT: current test use SdFat 2.0.6 and now I would like to bump to SdFat 2.1.2 =>need to check upgrade first

[luc-github]

Upgrade to SDFat 2.1.2 is now done and tested UTF file name also:
when it is good in ESP3DLib - even with latest bugfix and enabling

 #define UTF_FILENAME_SUPPORT

the output seems incorrect

>[ESP740]
Directory on SD : /
<DIR> 	overlays 	
<DIR> 	System Volume Information 	
<DIR> 	Archive 	
<DIR> 	New folder 	
      	bcm2708-rpi-b-plus.dtb 	13.94 KB 	
      	COPYING.linux 	18.25 KB 	
      	LICENCE.broadcom 	1.46 KB 	
      	bcm2709-rpi-2-b.dtb 	15.00 KB 	
      	bcm2708-rpi-b.dtb 	13.68 KB 	
      	bcm2708-rpi-cm.dtb 	13.64 KB 	
      	issue.txt 	145 B 	
      	CUBE.GCO 	205.31 KB 	
      	CUBE01.GCO 	1.99 MB 	
      	SUPPORT2.GCO 	4.40 MB 	
      	LEBOSSE.pdf 	0 B 	
      	bootcode.bin 	17.51 KB 	
      	cmdline.txt 	136 B 	
      	config.txt 	1.62 KB 	
      	fixup.dat 	6.47 KB 	
      	fixup_cd.dat 	2.48 KB 	
      	fixup_db.dat 	9.52 KB 	
      	kernel.img 	3.94 MB 	
      	kernel7.img 	4.04 MB 	
      	fixup_x.dat 	9.52 KB 	
      	VGMIL-6001.jpg 	1.69 MB 	
      	start_cd.elf 	619.66 KB 	
      	start_x.elf 	3.73 MB 	
      	LICENSE.oracle 	18.53 KB 	
      	bcm2710-rpi-3-b.dtb 	15.62 KB 	
      	Transfert.pdf 	57.88 KB 	
      	bcm2710-rpi-cm3.dtb 	14.99 KB 	
      	610203146-ddt.pdf 	7.30 MB 	
      	盧克test.GCO 	4.40 MB 	
      	6102031462-ddt.pdf 	7.30 MB 	
30 file(s), 4 dir(s)
Total 62.95 MB, Used 49.46 MB, Available: 13.48 MB
ok
>M20 L
Begin file list
CUBE.GCO 210240 CUBE.GCO
CUBE01.GCO 2089832 CUBE01.GCO
SUPPORT2.GCO 4613256 SUPPORT2.GCO
ARCHIVE/CUBE-C~1.GCO 210240 Archive/CUBE - Copy.G
NEWFOL~1/SUPPORT2.GCO 4613256 New folder/SUPPORT2.GCO
TEST~1.GCO 4613256 __test.GCO
End file list
ok

still need to test print command but seems not able to print sub folder file... TBC

WebUI is working as expected :

still need to handle possible sd access conflict in ESP3DLib
I am planning to do a SD release when ESP3DLib access SD and SD Mount when operation is done -> it will mimic the share SD behaviour => To be tested

[luc-github]

The sharing SD code have been updated as well as the new format of ESPXXX output as explained here : luc-github/ESP3D-WEBUI#157 (comment)
Still have FTP/WebDav SD access is not updated - I want to do the webui update first with updated output support and see how to handle the UTF / long name support in webUI as it seems it is still incomplete in Marlin : luc-github/ESP3D-WEBUI#221 (comment)

[luc-github]

Ok here we are - Webui V3 is now usable for Marlin-embedded but will still be updated : https://github.com/luc-github/ESP3D-WEBUI/blob/3.0-rewrite/dist/Printer3D/Marlin-embedded/index.html.gz?raw=true

to use ESP3DLib V2 with latest Marlin bug-fix and latest ESP32 core (2.0.3rc) you need currently to use this fork / branch : https://github.com/luc-github/Marlin/tree/ESP3DLibV2

here the board and status that I will updated during my tests

Board	Status	notes	notes 2
MRR_ESPA		No display	No IO expander
MRR_ESPE		No board on hand but compilation Ok	With IO expander
FYSECT_E4		with MKS TFT32_L V4.0 serial TFT	No IO expander
PANDA Zhu V1.2		No need UART patch	With IO expander
TinyBee V1		with MKS Mini12864 V3	With IO expander

I have already starting to test and found some cases that I could not test on simulator 😉 so will update WebUI / ESP3DLib in parallele

I do not have time to write doc for the configuration / feature list at this moment if you want to help on it, ping me, so you can do it on Wiki