Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #195 and #125 #197

Merged
merged 3 commits into from
Apr 17, 2022
Merged

Fixes #195 and #125 #197

merged 3 commits into from
Apr 17, 2022

Conversation

CalebQ42
Copy link
Contributor

@probonopd probonopd merged commit 9231e4d into probonopd:master Apr 17, 2022
@probonopd
Copy link
Owner

Thanks @CalebQ42. Highly appreciated. Happy Easter!

@probonopd
Copy link
Owner

Now that we can read squashfs, it would be interesting to see if we can get rid of all unsquashfs calls and how that would impact performance.

@CalebQ42
Copy link
Contributor Author

As it is right now, there shouldn't be any calls to unsquashfs (and there hasn't been since #115). goappimage has the capabilities to fallback to unsquashfs if it has issues, but by default is using my library. I don't believe the check for unsquashfs was ever removed though.

As for performance, my library is 2x~2.5x slower then unsquashfs when completely extracting an image.

@probonopd
Copy link
Owner

As it is right now, there shouldn't be any calls to unsquashfs

Sorry, my bad. I just saw it is still mentioned in the code.

As for performance, my library is 2x~2.5x slower then unsquashfs when completely extracting an image.

Do you know why that is?

@CalebQ42
Copy link
Contributor Author

As for performance, my library is 2x~2.5x slower then unsquashfs when completely extracting an image.

Do you know why that is?

I know at least part of it is that unsquashfs is written in C and seems to be very optimized C at that. I did, at one point, do my best to optimize it, and got it down to the ~2x slower point, before that I believe it was closer to ~4x slower. As it is, I would need to do research on how to optimize Go as it's already fully threaded and definitely uses ALL of the CPU when it's doing a full extract (just like unsquashfs does).

The test I've been doing is extracting the Firefox AppImage and the times are about .8 seconds for unsquashfs and 1.8 seconds for my library.

@probonopd
Copy link
Owner

Are the speed differences the same order of magnitude for zstd? Because that's probably where AppImage will go sooner or later.

@CalebQ42
Copy link
Contributor Author

So doing some tests with zstd, my library comes to be about 3x~4x slower. My library is a bit slower but unsquashfs runs a bit faster (.5 seconds vs 1.9 seconds).

@probonopd
Copy link
Owner

I see. So maybe using Go for this purpose is not suitable, or the C code is super optimized?

@CalebQ42
Copy link
Contributor Author

I'd have to do a bit more research into it to be sure. I am pretty sure that unsquashfs and zstd are both highly optimized C, which would give an inherent speed advantage vs Go, but I'd also like to see straight Go gzip vs Go zstd to see if it's just my library being a bottleneck.

@CalebQ42
Copy link
Contributor Author

And again I haven't ever done any research optimization. There's probably a decent amount of things that I could do to speed it up.

@CalebQ42
Copy link
Contributor Author

I wrote a quick test and it seems like the Go libraries for gzip and zstd have about equivalent performance to using command-line tools. In fact it seems the zstd library might be marginally faster (at least at the default compression level). So it definitely seems that my library is the main bottleneck.

@probonopd
Copy link
Owner

Thanks for the tests @CalebQ42. That sounds encouraging.

@probonopd
Copy link
Owner

Hi @CalebQ42 there seems to be another new squashfs implementation:

https://github.com/lawl/ayy/tree/master/squashfs

I read about it here:

So why an AppImage now? Because I got upset enough with packaging on linux that I wrote my own package manager for appimages, it only took minimal yak-shaving such as also writing my own squashfs driver.

Maybe you'd like to give this squashfs implementation a performance test?

@lawl
Copy link

lawl commented Apr 22, 2022

Maybe you'd like to give this squashfs implementation a performance test?

It's not optimized for speed. ayy is "pretty fast" because it (proudly) does very little.

ayy is optimized for this: https://github.com/lawl/ayy/blob/master/go.mod

the absolute minimal amount of dependencies possible (which is one dependency: openpgp).

@probonopd
Copy link
Owner

@lawl thanks for chiming in here. We are considering to use a Go implementation of squashfs (with zstd) for a potential future AppImage runtime entirely written in Go. But so far we are not sure whether we can match the current runtime's squashfs performance levels.

@lawl
Copy link

lawl commented Apr 22, 2022

I don't think ayy's squashfs library is currently a good fit for that, nor will it likely be in the near future. ayy tries to tackle other problems than writing a new runtime.

@probonopd
Copy link
Owner

OK, I see. anyhow, interesting project. Feel free to look around in this repo as well. Most likely you have better implementations for many things as this was my first contact to Go when I wrote this.

@CalebQ42 CalebQ42 deleted the master branch October 8, 2022 05:08
@CalebQ42 CalebQ42 restored the master branch October 8, 2022 05:48
@CalebQ42 CalebQ42 deleted the master branch July 31, 2023 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AppimageD Requires Sudo?
3 participants