Apps with lots of files are sloooow to install

[vgmoose]

For example, downloading Pynx's zip from the server works, but when it tries to install it takes about 3-5 minutes on the switch to extract all of the files to the SD card.

I don't think that this is a limitation of the switch, something in the zip extraction / manifest parsing code may be at fault. Pynx is also the only app that seems to have such a dramatic slowdown (it has 1417 files to extract to different folders).

This issue is twofold:

1. The extraction/installation time of creating the folders and copying those files should be cut down, if possible
2. A progress bar should be shown specifically for installing, rather than just downloading

For (2), the number of lines in the manifest file can be counted to get the total number of files, and then the progress bar can be updated using the callback in a similar fashion to how downloading does it.

EDIT: HCL-switch also has this issue, it has 658 files

[vgmoose]

Also worth noting that while there is a small lag on PC for copying these files, it's about 5 seconds rather than 5 minutes

[HelaFaye]

Well, zlib optimizations for ARMv8 CRC32 (in this branch: https://github.com/Adenilson/zlib/tree/NEON03 or https://github.com/Adenilson/zlib/tree/inffast02) seem to have practically no effect.

[NightlyFox]

I can take a look at this if you guys want! I love optimization 😍 something about seeing runtimes improve dramatically after a code change really tickles my fancy ;) lol... joking aside. Ill take a look at it later today. I have experience packaging/unpackaging zips and transfering them between servers over sftp.

[vgmoose]

That would be great! The issue might be in the libget library itself (Here is the current unzipper that's used.) I haven't tried to benchmark it on the PC, but on PC it's fast enough and I don't see any obvious issues that would might point to it being slower on the Switch.

[NightlyFox]

well, I have tried my usual tactics... which are passing by reference and getting rid of malloc entirely (unless needed). calling malloc over an over can slow it down, but it wasn't called that much.... so after like all of my modifications and me upping the buffer to 1MB to try and lower the number of writes to the file (which can slow the process down). I got a cool, like 3-5 seconds faster on the switch...lol 💃
I will be continuing to look at this but I honestly thought that the loop that wrote the buffer each time in the extract function was the bottleneck and I guess it wasn't. I know that creating the files/Directories is not an issue because at one point I coded in a bug that did not fully write each file (but only like 1/10th) of the size of each one... and it completed extremely fast and wrote each file and directory, so that gave me somewhat of a clue. That it may simply be the shear filesize of each file... and having to write them all to the switch.... i dont like that answer though lol.

[NightlyFox]

Quick question though... so I was googling faster compression libraries and I was able to find the "miniz.c" library... even tinier than minizip lol all in one file... supposedly made for performance. We could try using this to see if we improve the performance, should I? That is if it this library is compatible with the switch, it is written purely in C.... so that is good lol, and also uses some of zlibs code, so this library just may be compatible :3 hoping for the best. Also, if you want to look into it yourself too here is the link to where i got it...
https://code.google.com/archive/p/miniz/

[HelaFaye]

Try to compile it with the toolchain and see. That's what I did to test the inffast02 branch of zlib which has optimizations that have been merged into Chromium but, not zlib Master for some reason (5 second difference; 1m5-6s vs. 1m).

[NightlyFox]

Aye, if we merge our changes we might get a entire 8-10 second performance boost without changing libraries! Lol...

[NightlyFox]

So, the miniz.c file compiles and the nro is created, :) but i have to comment out all the other minizip code because it conflicts... i will rewrite the libget with the miniz.c file this weekend and see if i can get a performance increase

[NightlyFox]

so i have a little less than half of libgit written with the miniz library right now, you can take a look at the latest commit and run some tests on it if you want... for some reason it will not work on the switch but passes all extraction tests on my mac... not sure why :/

https://github.com/NightlyFox/libget

i also pushed up my makefile changes to the hb-appstore if you want to look at those, see if maybe i am doing something stupid lol

Edit: i also just found this.....
https://stackoverflow.com/questions/1360021/why-the-fwrite-libc-function-is-faster-than-the-syscall-write-function

i already tested replacing the write with fwrite and i do not think there were any performance improvements...

[vgmoose]

Errors like that are some of the worst... In the past, I've been able to debug similar issues (works on PC, crashes on Switch) by adding the flag -fstack-protector-all to the makefile flags, which would make it crash on the PC version as well.

In this case though, it doesn't help-- I added it to the libget makefile and the computer version is still working fine. Was it faster to extract (on computer) in your experience using miniz?

[NightlyFox]

Because the files are so small, i cant tell really..

Im going to use chrono to do some performance testing on my mac

so after performing 500 package installs in a row and taking the average time... i saw no significant performance increase..... a measely 0.23ms average vs a 0.25ms avg for the minizip library :3

how this will scale on the switch? no idea.... and considering i am still stumped as to why the miniz library just is not working.... i honestly do not think that small improvement is worth it :/

[vgmoose]

It's worth noting that it appears the libnx Retroarch port has the same issue when extracting "cheats.zip", the files themselves aren't big at all, but there are a lot of files/directories and it takes time to extract them all.

[vgmoose]

Adding the extraction indicator in 2.2 has mitigated this problem somewhat in my eyes. I have to wonder how much of the problem is just SD access time and beyond our control.

I’m closing this issue as we’re not pursuing it, but would still love to hear other ideas if this analysis isn’t correct.