Add LFS Migration and Mirror #14726
Add LFS Migration and Mirror #14726
Conversation
6543
added
the
type/feature
|
After talking with @zeripath yesterday on Discord I moved some parts of the old LFS infrastructure to |
GiteaBot
added
the
lgtm/need 2
modules/lfs/content_store.go
Outdated
| return false, err | ||
| } | ||
|
|
||
| return true, nil | ||
| } | ||
|
|
||
| // ReadMetaObject will read a models.LFSMetaObject and return a reader | ||
| func ReadMetaObject(pointer *Pointer) (io.ReadCloser, error) { |
There was a problem hiding this comment.
May need to consider cancellation here and elsewhere in this file - but that probably can wait for another pr.
|
There are lots of places here where you are simply checking for an error and returning it straight up. This is the equivalent of try-catch exception-throw exception. Some of these errors shouldn't happen eg. Json errors but some of them will be almost indistinguishable from each other once they bubble up and get logged. The simplest thing you can do is stick a log.Error before passing up the err. That is the absolute minimum that needs to happen around the http requests - unless some error is expected here. That would at least allow us to find the error. Some errors though should be being transformed to gain extra context. I would also argue that some log.Debugs and Traces wouldn't go amiss. Tracing the http request URLs might not be a bad idea at all. In general we're way to keen to make things * pointers in gitea - not everything needs to be a pointer - in particular immutable things probably shouldn't be. |
routers/repo/migrate.go
Outdated
| ctx.RenderWithErr(ctx.Tr("repo.migrate.invalid_lfs_endpoint"), tpl, &form) | ||
| return | ||
| } | ||
| err = migrations.IsMigrateURLAllowed(form.LFSEndpoint, ctx.User) |
There was a problem hiding this comment.
may need to run this on the ep instead.
routers/repo/lfs.go
Outdated
| errChan := make(chan error, 1) | ||
| wg := sync.WaitGroup{} | ||
| wg.Add(5) | ||
| for i, pointerBlob := range pointerBlobs { |
There was a problem hiding this comment.
pointerBlobs likely needs to become a channel
GiteaBot
added
lgtm/done
|
🚀 |
|
Maybe it escaped my knowledge (which is quite limited in terms of how git works under the hood, granted), but apparently this doesn't automatically enable me to mirror releases, huh? Or does the access token I generated on GitHub not have enough privileges? (I am deploying this on my personal LAN-only server, so I was already quite generous on the permission barring only the writes and some that seemed unrelated) I am running the latest tag in docker as of now if that helps. |
|
@GlassedSilver This PR is not for releases, but rather LFS. Release mirroring is not currently possible. Please also don't ask for help in closed tickets as they have a possibility of getting lost. For support please use our discourse forum or the discord chat. |
Implemented downloading of remote LFS files with a custom HTTP/Filesystem client. The GIT LFS Client isn't flexible enough to integrate into the Gitea model.
LFS is supported for migrations and mirrors files periodicaly if mirroring is enabled. The client respects file size limits set in the Gitea config and uses the internal LFS structure to store the files. The client determines the LFS endpoint from the clone url if not specified. Migration of local filesystem repositories is supported too. In case of a mirrored repository LFS support can be (de)activated later in the repository settings.
Migration page:
Migration of local repositories is supported:
Repository settings page:
ToDo:
modules/servicesseperationuse different credentialsshould be implemented with Add Gitea secrets storage and management #14483close #14718
close #849