Browser crashes on accessing big files

[MarionMoseby]

• Gitea version (or commit ref): Codeberg?
• Git version: Codeberg?
• Operating system: Ubuntu 20.04
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (Warning: The following link might crash your browser: LINK)
  • No
  • Not relevant
• Log gist: Im sorry I dont know what this is

Description

I've previously reported this bug on codeberg, a gitea instance, and I have been redirected to this issue tracker to report it here as well. The thing is, When I open this file for example, firefox becomes unresponsive due to the file being too big. Maybe the maximum file size to be shown in the preview should be decreased? In GH at least I have no option to preview such a large file.

Screenshots

I cannot add an screenshot as essentially my computer crashes, but basically the tab turns white and all slows down

[lafriks]

How big the file is?

[silverwind]

Big files are certainly an issue. GitHub has a limit of 1MB for file display above which it offers only a link to the raw file.

Both the number of DOM nodes and the syntax highlighter can cause slowdowns. We should evaluate and add a configurable limit. I think default could need to be around 512kB in our case.

[mrsdizzie]

GitHub limit For display is 5mb and stop highlighting at 1MB

https://help.github.com/en/github/creating-cloning-and-archiving-repositories/limits-for-viewing-content-and-diffs-in-a-repository

Gitea def has some other bug with larger files I've seen when working on syntax highlighting but not time to look into yet. loading a few mb of non highlighted text file should be fine and is no issue on github or other sites.

Might be something JavaScript side as I get one of those "this page is using too much memory" warnings sometimes. Will try and see if I can reproduce and find cause.

[silverwind]

GitHub limit For display is 5mb and stop highlighting at 1MB

Not true or outdated info. See these two examples. One is slightly below 1MB, one is above:

https://github.com/StylishThemes/GitHub-Dark/blob/master/github-dark.css
https://github.com/StylishThemes/GitHub-Dark/blob/master/github-dark.user.css

[mrsdizzie]

@silverwind interesting yea, I guess they have changed that. Not sure why they say "right now" if that is always the limit.

If that is the case, it seems like they just have a single limit of 1MB and there isn't a situation where they show file content but don't highlight it. We could have that by just changing this default to 1MB instead of the current 8:

gitea/modules/setting/setting.go

Line 214 in c5b08f6

MaxDisplayFileSize: 8388608,

Which would then match the already 1MB limit for highlighting.

Looking at the actual cause of slowness for large files in Gitea it is all 'recalculating styles' in the browser -- so there is something which doesn't scale there (and maybe thats fine, probably why Github has a 1MB limit too then).

[silverwind]

GitHub also has a limit at 20000 line number after which they just stop highlighting. See this example. You'll notice that highlighting stops at line 20k. It's also visible in simple diffs like this one when changes happen after line 20k. I'm not sure if we'll need a similar limit with our approach.

'recalculating styles'

Maybe some JS interference. Probably worth a try viewing diffs with scripts disabled in the browser. There's also this nasty bit of JS we should get rid of for split diff.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

[MarionMoseby]

Any progress with this? I feel like this is a valid issue and should not be closed

[schorsch13]

Another codeberg user reported a related issue https://codeberg.org/Codeberg/Community/issues/531 where the browser crashes when accessing this link WARNING: your browser might crash! But this link directs you to a git-diff instead of a big file

[silverwind]

Yes, we probably need some kind of lazy-loading solution. The HTML renders pretty fast in this case at least:

$ time curl -s 'https://codeberg.org/boud/gevolution/compare/0906a1e...fe756e7#diff-6' > /dev/null
curl -s  > /dev/null  0.03s user 0.04s system 3% cpu 2.200 total

[zeripath]

We already have a lazy load system on 1.16 (#17739 #16829)

Reduce the MAX_GIT_DIFF_FILES and MAX_GIT_DIFF_LINES as appropriate and use the functionality in 1.16 to load large diffs as needed.

Iirc in 1.16 I also made split diffs a whole lot less burdensome to render.