Package registry: large packages download with size of 0B #20605

dorianim · 2022-08-01T21:25:31Z

Description

When uploading a package to the generic registry, the size is correctly stated in the UI:

When downloading it, however, it has a size of 0B:

Gitea logs:
2022/08/01 23:21:08 [62e843c4] router: completed GET /bluerock/-/packages/generic/app/0-preview-2f3f0488da9755d3e1378fd33d95aceec86a1058/files/160 for 46.114.95.2:0, 0 in 25.2ms @ user/package.go:370(user.DownloadPackageFile)

Gitea Version

1.17.0

Can you reproduce the bug on the Gitea demo site?

No, maybe because it is on 1.18.0+dev-198-g4f14c6de1?

How are you running Gitea?

I am using the gitea/gitea docker image on the latest tag

Database

PostgreSQL

The text was updated successfully, but these errors were encountered:

wxiaoguang · 2022-08-02T05:40:23Z

Are you using any reverse-proxy, or can you try to download the file from Gitea HTTP port directly? The content-length header doesn't seem to be generated by Gitea.

dorianim · 2022-08-02T06:21:37Z

Thanks for your reply! Unfortunately, this does not seem to be the reason.
My instance is indeed running behind a trafik reverse proxy, but when curling it directly, the content length header still exists and is 0:

wxiaoguang · 2022-08-02T06:56:32Z

I guess the problem is that the uploaded file was empty already. There is a bug when uploading packages.

When a large package is uploaded, the pfci.Data is already EOF before contentStore.Save, so there will an empty file saved in the store.

@KN4CK3R

If I add a pfci.Data.(*packages_module.HashedBuffer).Seek(0, io.SeekStart) before if err := contentStore.Save(packages_module.BlobHash256Key(pb.HashSHA256), pfci.Data, pfci.Data.Size()); err != nil { in addFileToPackageVersion, then the problem is resolved.

KN4CK3R · 2022-08-02T08:31:44Z

That should not be EOF. My old test upload does not have a problem. I will have a look.

Edit: The error occurs only with larger files which are temporarly stored on the filesystem, so my FileBackedBuffer may be broken.

KN4CK3R · 2022-08-02T09:08:30Z

Fixed with #20622

dorianim · 2022-08-02T15:32:27Z

Thanks for the quick fix!
In which version will this be released?

wxiaoguang · 2022-08-02T15:33:23Z

Now it on 1.18-dev

There will be a backport for 1.17.1

KN4CK3R · 2022-08-02T17:51:51Z

Backport in #20635

f403 · 2022-10-11T13:40:48Z

I fear, this is not solved, or there is another bug.
When uploading very big files (in my case 1.4G) they do not appear on disk, and downloading results in a 0B size.
Without any warnings during upload/download.

Gitea 1.17.2 (docker)
Steps to reproduce:

$ ll
total 1,4G
drwxrwxr-x  2 someuser someuser 4,0K Okt 11 14:47 .
drwxrwxrwt 41 root     root      12K Okt 11 14:50 ..
-rw-rw-r--  1 someuser someuser 1,4G Aug  9 18:48 ubuntu-22.04.1-live-server-amd64.iso

$ curl --user user:tockentockentockentockentockentockentock --upload-file ubuntu-22.04.1-live-server-amd64.iso https://gitea.host/api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso

$ wget -O downloaded.iso https://gitea.host/user/-/packages/generic/ubuntu-live-server/22.04.1/files/30
--2022-10-11 14:57:57--  https://gitea.host/user/-/packages/generic/ubuntu-live-server/22.04.1/files/30
Resolving gitea.host (gitea.host)... 198.51.100.1
Connecting to gitea.host (gitea.host)|198.51.100.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [application/octet-stream]
Saving to: ‘downloaded.iso’

downloaded.iso                                           [ <=>                                                                                                                  ]       0  --.-KB/s    in 0s      

2022-10-11 14:57:57 (0,00 B/s) - ‘downloaded.iso’ saved [0/0]

$ ll
total 1,4G
drwxrwxr-x  2 someuser someuser 4,0K Okt 11 14:57 .
drwxrwxrwt 41 root     root      12K Okt 11 14:57 ..
-rw-rw-r--  1 someuser someuser    0 Okt 11 14:57 downloaded.iso
-rw-rw-r--  1 someuser someuser 1,4G Aug  9 18:48 ubuntu-22.04.1-live-server-amd64.iso

And the logs at this time:

gitea     | 2022/10/11 12:53:36 [63456750] router: started   PUT /api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso for 172.19.0.3:55322      
gitea     | 2022/10/11 12:53:36 ...rvices/auth/basic.go:67:Verify() [T] [63456750] Basic Authorization: Attempting login for: user                                                                                   
gitea     | 2022/10/11 12:53:36 ...rvices/auth/basic.go:90:Verify() [T] [63456750] Basic Authorization: Valid AccessToken for user[0]                                                                              
gitea     | 2022/10/11 12:53:39 [633ef465-25] router: slow      PUT /api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso for 172.19.0.3:55322, elapsed 3888.4ms @ generic/gener
ic.go:67(generic.UploadPackage)                                                                                                                                                                                    
gitea     | 2022/10/11 12:53:50 ...packages/packages.go:112:createPackageAndVersion() [T] [63456750] Creating package: 3, 3, generic, ubuntu-live-server, 22.04.1, map[], map[], false                             
gitea     | 2022/10/11 12:53:51 ...packages/packages.go:226:addFileToPackageVersion() [T] [63456750] Adding package file: 24, ubuntu-22.04.1-live-server-amd64.iso
gitea     | 2022/10/11 12:53:52 [63456750] router: completed PUT /api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso for 172.19.0.3:55322, 201 Created in 16835.7ms @ generic/
generic.go:67(generic.UploadPackage)

gitea     | 2022/10/11 12:57:58 [63456856] router: started   GET /user/-/packages/generic/ubuntu-live-server/22.04.1/files/30 for 172.19.0.3:37902
gitea     | 2022/10/11 12:57:58 [63456856] router: completed GET /user/-/packages/generic/ubuntu-live-server/22.04.1/files/30 for 172.19.0.3:37902, 0  in 43.2ms @ user/package.go:370(user.DownloadPackageFile)

And there no new files on disk:

$ touch --date "2022-10-10" /tmp/start
$ find /data/gitea/gitea/packages/ -type f -newer /tmp/start
$

KN4CK3R · 2022-10-11T13:50:16Z

Do you use a reverse proxy?

Tested it with a 1.5gb file and it worked without problems.

f403 · 2022-10-11T14:08:35Z

Yes, but same happens when uploading without proxy.

$ curl --user user:tockentockentockentockentockentockentock --upload-file ubuntu-22.04.1-live-server-amd64.iso http://172.19.0.4:3000/api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso
$ wget -O downloaded.iso http://172.19.0.4:3000/api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso
--2022-10-11 14:03:51--  http://172.19.0.4:3000/api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso
Connecting to 172.19.0.4:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [application/octet-stream]
Saving to: ‘downloaded.iso’

downloaded.iso                                           [ <=>                                                                                                                  ]       0  --.-KB/s    in 0s      

2022-10-11 14:03:51 (0.00 B/s) - ‘downloaded.iso’ saved [0/0]

$ ll
total 1.4G
drwxrwxr-x  2 someuser someuser 4.0K Oct 11 14:03 ./
drwxrwxrwt 15 root     root     4.0K Oct 11 13:53 ../
-rw-rw-r--  1 someuser someuser    0 Oct 11 14:03 downloaded.iso
-rw-rw-r--  1 someuser someuser 1.4G Aug  9 16:48 ubuntu-22.04.1-live-server-amd64.iso

f403 · 2022-10-11T14:37:36Z

Hmm...
It worked in a fresh container on my laptop.

Can you, please, give me a hint, where to look for the uploaded files (temp path, disk path for a certain package file)?

KN4CK3R · 2022-10-11T14:41:00Z

When uploading files larger 32mb the content gets temporarly stored in %temp%/gitea-buffer-xxx with %temp% whatever is the temporary directory of the current user and xxx some random characters.

And the file is then stored in data/packages/xxx by default. This can be changed in the app.ini. xxx is a path depeding of the file sha256 hash.

f403 · 2022-10-11T15:13:13Z

Thank you!

There was an empty file under data/packages/xxx created before updating to 1.17.2.
Deleting the file and restarting Gitea helped. (And running "Cleanup expired packages", not sure if it was needed)

(Not overwriting existing files is correct, otherwise there is a possibility for a hash-collision attack)

dorianim added the type/bug label Aug 1, 2022

wxiaoguang added the issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented label Aug 2, 2022

wxiaoguang added the issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP label Aug 2, 2022

KN4CK3R added the topic/packages label Aug 2, 2022

KN4CK3R mentioned this issue Aug 2, 2022

Fix package upload for files >32mb #20622

Merged

wxiaoguang closed this as completed in #20622 Aug 2, 2022

wxiaoguang reopened this Aug 2, 2022

wxiaoguang added this to the 1.17.1 milestone Aug 2, 2022

lafriks closed this as completed Aug 2, 2022

wxiaoguang mentioned this issue Aug 9, 2022

Error while extracting downloaded file #20731

Closed

wxiaoguang changed the title ~~Package registry: generic packages download with size of 0B~~ Package registry: large packages download with size of 0B Aug 9, 2022

go-gitea locked and limited conversation to collaborators May 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package registry: large packages download with size of 0B #20605

Package registry: large packages download with size of 0B #20605

dorianim commented Aug 1, 2022 •

edited

Loading

wxiaoguang commented Aug 2, 2022

dorianim commented Aug 2, 2022 •

edited by wxiaoguang

Loading

wxiaoguang commented Aug 2, 2022

KN4CK3R commented Aug 2, 2022 •

edited

Loading

KN4CK3R commented Aug 2, 2022

dorianim commented Aug 2, 2022

wxiaoguang commented Aug 2, 2022

KN4CK3R commented Aug 2, 2022

f403 commented Oct 11, 2022

KN4CK3R commented Oct 11, 2022 •

edited

Loading

f403 commented Oct 11, 2022

f403 commented Oct 11, 2022

KN4CK3R commented Oct 11, 2022 •

edited

Loading

f403 commented Oct 11, 2022 •

edited

Loading

Package registry: large packages download with size of 0B #20605

Package registry: large packages download with size of 0B #20605

Comments

dorianim commented Aug 1, 2022 • edited Loading

Description

Gitea Version

Can you reproduce the bug on the Gitea demo site?

How are you running Gitea?

Database

wxiaoguang commented Aug 2, 2022

dorianim commented Aug 2, 2022 • edited by wxiaoguang Loading

wxiaoguang commented Aug 2, 2022

KN4CK3R commented Aug 2, 2022 • edited Loading

KN4CK3R commented Aug 2, 2022

dorianim commented Aug 2, 2022

wxiaoguang commented Aug 2, 2022

KN4CK3R commented Aug 2, 2022

f403 commented Oct 11, 2022

KN4CK3R commented Oct 11, 2022 • edited Loading

f403 commented Oct 11, 2022

f403 commented Oct 11, 2022

KN4CK3R commented Oct 11, 2022 • edited Loading

f403 commented Oct 11, 2022 • edited Loading

dorianim commented Aug 1, 2022 •

edited

Loading

dorianim commented Aug 2, 2022 •

edited by wxiaoguang

Loading

KN4CK3R commented Aug 2, 2022 •

edited

Loading

KN4CK3R commented Oct 11, 2022 •

edited

Loading

KN4CK3R commented Oct 11, 2022 •

edited

Loading

f403 commented Oct 11, 2022 •

edited

Loading