Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix storage Iterate bug and Add storage doctor to delete garbage attachments #16971

Merged
merged 4 commits into from
Sep 6, 2021
Merged

Fix storage Iterate bug and Add storage doctor to delete garbage attachments #16971

merged 4 commits into from
Sep 6, 2021

Conversation

lunny
Copy link
Member

@lunny lunny commented Sep 6, 2021

For IterateObjects function, the first parameter should be the relative path of the object, and the second parameter Object' Name() method should only return the base name of the object. This PR fix it and this should be also back port to v1.15

This PR also introduced a new doctor check: storages, at first, only attachments are checked. More checks(LFS, repo-archives and etc.) could be added in other PR.
The new check will iterate all objects in attachment storage and check if it's recorded in database. When with fix option, it will delete all the missed information objects.

@lunny lunny added type/bug type/enhancement An improvement of existing functionality backport/v1.15 labels Sep 6, 2021
@lunny lunny added this to the 1.16.0 milestone Sep 6, 2021
@codecov-commenter
Copy link

codecov-commenter commented Sep 6, 2021 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (main@0767fe0). Click here to learn what that means.
The diff coverage is 19.60%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main   #16971   +/-   ##
=======================================
  Coverage        ?   45.15%           
=======================================
  Files           ?      763           
  Lines           ?    86243           
  Branches        ?        0           
=======================================
  Hits            ?    38939           
  Misses          ?    41006           
  Partials        ?     6298
Impacted Files Coverage Δ
cmd/doctor.go 0.00% <ø> (ø)
models/attachment.go 57.74% <0.00%> (ø)
modules/storage/minio.go 58.18% <0.00%> (ø)
modules/doctor/storage.go 21.27% <21.27%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0767fe0...4b55635. Read the comment docs.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Sep 6, 2021
axifive
axifive approved these changes Sep 6, 2021
View reviewed changes
@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Sep 6, 2021
6543 and others added 2 commits September 6, 2021 16:02
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Sep 6, 2021
@6543 6543 merged commit a807031 into go-gitea:main Sep 6, 2021
@lunny lunny deleted the lunny/doctor_storage branch September 7, 2021 01:01
lunny added a commit to lunny/gitea that referenced this pull request Sep 7, 2021
@lunny
Fix storage Iterate bug and Add storage doctor to delete garbage atta…
1e1741f 
…chments (go-gitea#16971)

* Fix storage Iterate bug and Add storage doctor to delete garbage attachments

* Close object when used
@lunny lunny mentioned this pull request Sep 7, 2021
Merged
@lunny lunny added the backport/done All backports for this PR have been created label Sep 7, 2021
zeripath pushed a commit that referenced this pull request Sep 7, 2021
@lunny
Fix storage Iterate bug and Add storage doctor to delete garbage atta…
07489d0 
…chments (#16971) (#16977)

* Fix storage Iterate bug and Add storage doctor to delete garbage attachments

* Close object when used
zeripath added a commit to zeripath/gitea that referenced this pull request Sep 19, 2021
@zeripath
Changelog 1.15.3
b0c5677 
## [1.15.3](https://github.com/go-gitea/gitea/releases/tag/v1.15.3) - 2021-09-19

* ENHANCEMENTS
  * Add fluid to ui container class to remove margin (go-gitea#16396) (go-gitea#16976)
  * Add caller to cat-file batch calls (go-gitea#17082) (go-gitea#17089)
* BUGFIXES
  * Render full plain readme. (go-gitea#17083) (go-gitea#17090)
  * Upgrade xorm to v1.2.4 (go-gitea#17059)
  * Fix bug of migrate comments which only fetch one page (go-gitea#17055) (go-gitea#17058)
  * Do not show issue context popup on external issues (go-gitea#17050) (go-gitea#17054)
  * Decrement Fork Num when converting from Fork (go-gitea#17035) (go-gitea#17046)
  * Correctly rollback in ForkRepository (go-gitea#17034) (go-gitea#17045)
  * Fix missing close in WalkGitLog (go-gitea#17008) (go-gitea#17009)
  * Add prefix to SVG id/class attributes (go-gitea#16997) (go-gitea#17000)
  * Fix bug of migrated repository not index (go-gitea#16991) (go-gitea#16996)
  * Skip AllowedUserVisibilityModes validation on update user if it is an organisation (go-gitea#16988) (go-gitea#16990)
  * Fix storage Iterate bug and Add storage doctor to delete garbage attachments (go-gitea#16971) (go-gitea#16977)
  * Fix issue with issue default mail template (go-gitea#16956) (go-gitea#16975)
  * Ensure that rebase conflicts are handled in updates (go-gitea#16952) (go-gitea#16960)
  * Prevent panic on diff generation (go-gitea#16950) (go-gitea#16951)

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath mentioned this pull request Sep 19, 2021
Merged
lunny pushed a commit that referenced this pull request Sep 20, 2021
@zeripath @techknowlogick
Changelog 1.15.3 (#17091)
7ce938b 
## [1.15.3](https://github.com/go-gitea/gitea/releases/tag/v1.15.3) - 2021-09-19

* ENHANCEMENTS
  * Add fluid to ui container class to remove margin (#16396) (#16976)
  * Add caller to cat-file batch calls (#17082) (#17089)
* BUGFIXES
  * Render full plain readme. (#17083) (#17090)
  * Upgrade xorm to v1.2.4 (#17059)
  * Fix bug of migrate comments which only fetch one page (#17055) (#17058)
  * Do not show issue context popup on external issues (#17050) (#17054)
  * Decrement Fork Num when converting from Fork (#17035) (#17046)
  * Correctly rollback in ForkRepository (#17034) (#17045)
  * Fix missing close in WalkGitLog (#17008) (#17009)
  * Add prefix to SVG id/class attributes (#16997) (#17000)
  * Fix bug of migrated repository not index (#16991) (#16996)
  * Skip AllowedUserVisibilityModes validation on update user if it is an organisation (#16988) (#16990)
  * Fix storage Iterate bug and Add storage doctor to delete garbage attachments (#16971) (#16977)
  * Fix issue with issue default mail template (#16956) (#16975)
  * Ensure that rebase conflicts are handled in updates (#16952) (#16960)
  * Prevent panic on diff generation (#16950) (#16951)

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
6543 pushed a commit to 6543-forks/gitea that referenced this pull request Sep 21, 2021
@zeripath @techknowlogick @6543
Changelog 1.15.3 (go-gitea#17091)
a0bd752 
## [1.15.3](https://github.com/go-gitea/gitea/releases/tag/v1.15.3) - 2021-09-19

* ENHANCEMENTS
  * Add fluid to ui container class to remove margin (go-gitea#16396) (go-gitea#16976)
  * Add caller to cat-file batch calls (go-gitea#17082) (go-gitea#17089)
* BUGFIXES
  * Render full plain readme. (go-gitea#17083) (go-gitea#17090)
  * Upgrade xorm to v1.2.4 (go-gitea#17059)
  * Fix bug of migrate comments which only fetch one page (go-gitea#17055) (go-gitea#17058)
  * Do not show issue context popup on external issues (go-gitea#17050) (go-gitea#17054)
  * Decrement Fork Num when converting from Fork (go-gitea#17035) (go-gitea#17046)
  * Correctly rollback in ForkRepository (go-gitea#17034) (go-gitea#17045)
  * Fix missing close in WalkGitLog (go-gitea#17008) (go-gitea#17009)
  * Add prefix to SVG id/class attributes (go-gitea#16997) (go-gitea#17000)
  * Fix bug of migrated repository not index (go-gitea#16991) (go-gitea#16996)
  * Skip AllowedUserVisibilityModes validation on update user if it is an organisation (go-gitea#16988) (go-gitea#16990)
  * Fix storage Iterate bug and Add storage doctor to delete garbage attachments (go-gitea#16971) (go-gitea#16977)
  * Fix issue with issue default mail template (go-gitea#16956) (go-gitea#16975)
  * Ensure that rebase conflicts are handled in updates (go-gitea#16952) (go-gitea#16960)
  * Prevent panic on diff generation (go-gitea#16950) (go-gitea#16951)

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
lunny pushed a commit that referenced this pull request Sep 22, 2021
@6543 @zeripath @techknowlogick
Changelog 1.15.3 (#17091) (#17113)
fc1e9e6 
## [1.15.3](https://github.com/go-gitea/gitea/releases/tag/v1.15.3) - 2021-09-19

* ENHANCEMENTS
  * Add fluid to ui container class to remove margin (#16396) (#16976)
  * Add caller to cat-file batch calls (#17082) (#17089)
* BUGFIXES
  * Render full plain readme. (#17083) (#17090)
  * Upgrade xorm to v1.2.4 (#17059)
  * Fix bug of migrate comments which only fetch one page (#17055) (#17058)
  * Do not show issue context popup on external issues (#17050) (#17054)
  * Decrement Fork Num when converting from Fork (#17035) (#17046)
  * Correctly rollback in ForkRepository (#17034) (#17045)
  * Fix missing close in WalkGitLog (#17008) (#17009)
  * Add prefix to SVG id/class attributes (#16997) (#17000)
  * Fix bug of migrated repository not index (#16991) (#16996)
  * Skip AllowedUserVisibilityModes validation on update user if it is an organisation (#16988) (#16990)
  * Fix storage Iterate bug and Add storage doctor to delete garbage attachments (#16971) (#16977)
  * Fix issue with issue default mail template (#16956) (#16975)
  * Ensure that rebase conflicts are handled in updates (#16952) (#16960)
  * Prevent panic on diff generation (#16950) (#16951)

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
@go-gitea go-gitea locked and limited conversation to collaborators Oct 19, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@axifive axifive axifive approved these changes

@6543 6543 6543 approved these changes

Assignees
No one assigned
Labels
backport/done All backports for this PR have been created lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/bug type/enhancement An improvement of existing functionality
Projects
None yet
Milestone
1.16.0
Development

Successfully merging this pull request may close these issues.
5 participants
@lunny @codecov-commenter @axifive @6543 @GiteaBot