slow opening of singlefile addon saved pages.

[rezad1393]

I use singlefile addon to save webpage(after webscrap addon no longer working).
and I have a lot of addon installed.

but I see when I open a HTML page saved with this addon, firefox almost hangs (the html file saved from this addon are one single file with images embedded in this html text)
opening from bookmarked local file address of the saved webpage cause firefox to open a new tab but not load the address for 2 sec and the load the file and then very slow firefox behavior until it completely loads(it is a local file so it is not network connection issues)

I tried disabling addons to see if it helps.
I got to these 4 that enabling any of them caused the issue that I described.
" https://addons.mozilla.org/en-US/firefox/addon/requestcontrol/
https://addons.mozilla.org/en-US/firefox/addon/privacy-pass/
https://addons.mozilla.org/en-US/firefox/addon/ipvfoo-pmarks/
https://addons.mozilla.org/en-US/firefox/addon/smartproxy/"

I can disable other 3 but I need your addon.
can you check this out and see if you can optimize it so that it works better?
this is not smartproxy fault(at least not all of it) but still it is a issue that made me wonder what other pages are slows by this addon (which I still love and use).

BTW this doesn't happen with only smartproxy installed so I know it is now all the addon fault but with my profile disabling the addon fixes the issue.
(all other three addons are also like that so maybe that helps find the issue).

I think it has to do with domains because the singlefile HTML file has data: type blob and maybe smartproxy and ip6voo and others are trying to work with domain names and get confused.

with this addon (https://addons.mozilla.org/en-US/firefox/addon/website-information/)
it show this domains (though may not be completely reliable)

even if I put smartproxy in no proxy mode I still have the issue.

[salarcode]

It could be a race condition between these addons.
It will take sometime for me to test this.

[rezad1393]

race condition between any one of those addons (including yours) and my other addons?

btw,I have another firefox profile with less addons installed and I saw that umatrix also introduced this issue with the pages

. I will try to see how many addons I can keep with your addon and still not get the issue.

UPDATE:
ok I tried with a webpage saved from this page https://www.formuler.tv/category-4k-android-hybrid
and I tried with a new empty profile and only smartproxy installed with no rules and with direct proxy set.

try it yourself.
open the saved page and while it is loading try to switch tabs if firefox. you will notice the lag.

[salarcode]

What tool you use to save pages?

[rezad1393]

oh I forgot to give the link
it is singlefile addon in firefox.
https://github.com/gildas-lormeau/SingleFile
https://addons.mozilla.org/en-US/firefox/addon/single-file/

[salarcode]

Initial check shows that the saved page generates about 258 requests in less than a second. double it for responses! All of these are "fake-request" and are generated by Firefox.
The more addon you have that controls the web requests the slower browser will get. Multiple that number several times.

I'm investigating this to improve performance in certain areas but the gain won't be much.

[rezad1393]

I understand this. I even suggested as much with the picture I uploaded.
I am saying that your addon has this issue with it, but I dont mean it is your fault.
I am trying to see if it can be fixed with the singlefile addon author here:
gildas-lormeau/SingleFile#225

what I am asking from you is to maybe optimize your addon too so that number of connections dont affect it this severely.
but ultimately it is your addon not mine and the road-map of development is for you to decide.
other addon that I referenced also create the same issue and I also said at the start that it is not your addon fault only.
I use your addon daily and I am trying to make it so that I am sure my browser slowness (whenever it happens) is not a but just slow iranian network :)

btw how did you track the bogus request?
they dont show up in browser console.

[salarcode]

Yes that's what i'm doing, investigating and improving.
You can improve the performance by disabling Error Detection for now until the fix comes in.

btw how did you track the bogus request?

Commenting out Debug.disable(); :)

[gildas-lormeau]

@salarcode Hi, I'm the author of SingleFile. On my end, I determined that browser.webRequest.onBeforeRequest.addListener was responsible for this issue. The problem is that this API does not seem to work at all with file URLs at least in Firefox. So, If you would like to circumvent this issue I think that you would have to replace { urls: ["<all_urls>"] } with { urls: ['https://*/*', 'http://*/*'] } when calling this API.

[salarcode]

@gildas-lormeau Thanks, that's a good point, the problem is not just onBeforeRequest, there are other listeners as well which I'll look after.
Also the rule *://*/* would be enough.

[gildas-lormeau]

@salarcode Indeed I did not test all the now infamous webRequest APIs. FYI, I'm going to open an issue in the bug tracker of Firefox, cf. gildas-lormeau/SingleFile#225 (comment).

[rezad1393]

did I just start a war? or maybe a new romance? If you guys got married you have to invite me to the wedding. :)

I think the http and https are enough for now (ftp is not really alive anymore) and using :///* may (will) interfere with other connections. like ipfs:// and so on. So I think using from minimal and later adding more rules is better that going with :///* .

btw should I tell all the other addons that I referenced here about this too? so that they fix it or report to firefox?

[rezad1393]

Yes that's what i'm doing, investigating and improving.
You can improve the performance by disabling Error Detection for now until the fix comes in.

btw how did you track the bogus request?

Commenting out Debug.disable(); :)

where is this ?
in the addon ?
so I cant just use firefox and see these request ?(without any other things installed)

[gildas-lormeau]

FYI, I created this issue in the bug tracker of Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1557300

[salarcode]

Thank you @rezad1393 and @gildas-lormeau the issue is fixed now.
@rezad1393 this was good collaboration
Debug.disable(); is in the code and in order to use it yo have to build the code after doing the changes

SmartProxy/src/core/Core.ts

Line 41 in 2189794

Debug.disable();

@gildas-lormeau Thanks, I've also updated the bugzila ticket with more info.

[rezad1393]

Can you use http-https instead of "*://*/*"
so that it doesn't affect other protocols? like ipfs:// and dat:// and so on?
this way you don't get this bug back in later.

[salarcode]

No it won't, match patterns are limited to http, https, ws, wss, ftp, ftps, data or file.
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Match_patterns

[rezad1393]

it says scheme can take two form
1 *
2 all that you named.

I think using http and https specifically is better so that if later mozilla decides to change the coverage your addon doesnt get this bug again but with different schemes.

you already added ws wss and so on. so removing * and adding http-https (that is the second form in mozilla docs) is not a stretch.

this is just a suggestion.
you are the author and it is for you to decide.

btw thanks for the fast hotfix. I have tested with a small saved file and it no longer has the slowness.
also do you think I should report this bug to the other addon developers too or wait for firefox fix maybe?

[salarcode]

Well actually Smartproxy should cover as much as it is possible, apart from the local files that are now excluded, the rest should be covered, so any changes from Firefox is welcome.

Also ws and wss are mentioned in the list because in Chrome star (*) doesn't include websocket protocols and has be explicitly specified.

I see you have already reported to the authors :) good to mention the bugzilla ticket as well.

[rezad1393]

you mean your addon should cover ipfs:// protocol?
I think that it shouldnt.
it isnt even http protocol and would be wasteful to cover it.
I am talking about these :
"Support for Decentralization Protocols" section of this page
https://blog.mozilla.org/addons/2018/01/26/extensions-firefox-59/

these are
Dat Project (dat://)
IPFS (dweb:// ipfs:// ipns://)
Secure Scuttlebutt (ssb://)

I think the best wat is still covering as little as possible but still covering the needed parts.
http/s and ws/s are http protocol still.
so that's is proxy territory but other are not http so covering them would at best be wasteful and at worst introduce bugs

[salarcode]

you mean your addon should cover ipfs:// protocol?

I mean the proxy should cover whatever Firefox parses and understands which for ipfs:// case it doesn't. It just recognize it as a protocol.

I think the best wat is still covering as little as possible but still covering the needed parts.

I disagree, this addon should provide at least the same level of the browser itself provides.

http/s and ws/s are http protocol still.
so that's is proxy territory but other are not http so covering them would at best be wasteful and at worst introduce bugs

You are forgetting Socks proxy protocol. This means that it doesn't matter the request is http or not, if user wants, he can apply proxy for it using Socks proxy.

Also the ProxyAPI that extensions have access to is limited by nature, in Firefox case a handful of protocols are allowed which I have listed above.
More read:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/RequestFilter

[rezad1393]

oh yea I forgot the socks one.
but as you saw proxy still got involved with the data: urls which it shouldnt have.
and the ipfs and so on will get more ingetrated and they are NOT the target of http(or socks) proxy.
they are specifically intended to get the resource from some other program.
like mailto: was before them.

but ,again, you are the developer and I think you know more than me and should do what you think is best. I was just trying to tell what I knew so that maybe it would help.

I am gonna unsubscribe from this now that the issue is resolved.

again thanks you for fast replied and fixes.

[salarcode]

No worries,
Just be clear again, ipfs should be supported by Extensions Proxy API in the first place which will never be because of its nature.

And it is safe to assume whatever implemented by proxy API is web request hence the name webRequest, they can go through a proxy.
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest