Use git plumbing for upload rather than porcelain

[zeripath]

When uploading files using the GUI, Gitea currently does:

a) A full clone - Including the whole object DB
b) A full branch checkout - every file
c) Copies the uploaded file to this cloned repository
d) Add/stages the changes
e) Commits
f) Pushes to the real repository
g) Cleans up the cloned repository - deleting the whole object db and checkout.

This is extremely inefficient.

This PR takes a different approach and, instead of using the git porcelain commands (git add and the like), uses the plumbing commands to (hopefully) significantly speed up this process. See my comments in #601 (#601 (comment)).

Currently my approach is to use the git-cli commands which can/could be migrated to go-git commands later.

Will fix #5600.

[codecov-io]

Codecov Report

Merging #5621 into master will increase coverage by 0.11%.
The diff coverage is 44.78%.

@@            Coverage Diff             @@
##           master    #5621      +/-   ##
==========================================
+ Coverage   37.81%   37.93%   +0.11%     
==========================================
  Files         322      322              
  Lines       47485    47682     +197     
==========================================
+ Hits        17957    18087     +130     
- Misses      26939    26995      +56     
- Partials     2589     2600      +11

Impacted Files	Coverage Δ
routers/repo/editor.go	28.51% <26.47%> (-0.51%)	⬇️
models/repo_editor.go	31.47% <45.9%> (+17.49%)	⬆️
models/repo_branch.go	58.27% <56%> (-0.46%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e90103...23b43cd. Read the comment docs.

[zeripath]

I don't currently set the SSH_ORIGINAL_COMMAND environment variable, meaning that although the gitea post-receive hooks et al are run, they short circuit and don't do anything.

It wouldn't be hard to set this - the value isn't actually checked (it just needs to be set) - however, that will make a change from the previous settings.

[zeripath]

I've completed reimplementing repo_editor.go. So it's good to look at but I don't think it's ready for merging.

[zeripath]

My simple tests show a 10x speed up for preview diff of a very small file in a very small repo, from 340ms to 34ms. I saw similar speed ups for repouploadfile. I didn't actually check the timings for repouploadfiles but expect it to be similar.

Next steps are to move most, if not all of the helper functionality into the git module.

Having to do git ls-tree -r | git update-index --index-info (or thereabouts) seems wasteful and I think it should be possible to further optimise this. However, the timings above seem to indicate that there's still significant speed ups even with this.

---

OK, this command can be replaced by:

git read-tree HEAD with suitable options.

Further if you place your working directory as the bare repository then commands become a lot simpler.

I've therefore rebased my work on this.

[ghost]

Will this PR fix #4778?

[zeripath]

My current implementation doesn't account for LFS. It should be relatively easy to add though.

[zeripath]

@filipnavara would it be possible for you to take a look at this and tell me if it works for you and your large repos?

[filipnavara]

@zeripath I just finished updating my Gitea installation and called it a day... Will review and give it a try tomorrow. :-)

[zeripath]

@yasuokav ok, it looks like this might be possible however I need to refactor these code paths up and out models to allow us to use the LFS module, otherwise when it comes to store we have to reimplement the ContentStore. (Finding that out was this evening's work.) I might be able to take a look again at this tomorrow.

[zeripath]

Ok I would have had an updated pr to push if I hadn't spent an hour debugging failing uploads because I had written if err == nil instead of != however it looks like it will work.

[zeripath]

OK, I have made this work with LFS. However, in order to make this PR a bit more readable I'm going to close it and reopen with a cleaner history.

[zeripath]

@filipnavara @yasuokav @xxxTonixxx @techknowlogick @lunny @typeless I've decided this PR is too busy and have reopened in #5702 with LFS support.