Failed to synchronize tags to release in SyncMirror with error: "pushUpdateAddTag: GetTag: Length must be 40:" #19038

somera · 2022-03-09T11:44:10Z

Feature Description

In gitea.log I see sometimes:

==> gitea.log <==
2022/03/09 12:06:37 ...irror/mirror_pull.go:261:runSync() [E] Failed to synchronize tags to releases for repository: pushUpdateAddTag: GetTag: Length must be 40:
        /source/services/mirror/mirror_pull.go:261 (0x1cdf3b8)
        /source/services/mirror/mirror_pull.go:376 (0x1ce10b8)
        /source/services/mirror/mirror.go:42 (0x1cdb344)
        /source/services/mirror/mirror.go:144 (0x1cdbe31)
        /source/modules/queue/unique_queue_channel.go:76 (0x14954b6)
        /source/modules/queue/workerpool.go:437 (0x149e1c5)
        /source/modules/queue/workerpool.go:280 (0x149d096)
        /usr/local/go/src/runtime/asm_amd64.s:1581 (0x471520)

But you can't see:

which project
which tag?
how long is it?

Screenshots

No response

The text was updated successfully, but these errors were encountered:

Yet another issue has come up where the logging from SyncMirrors does not provide enough context. This PR adds more context to these logging events. Related go-gitea#19038 Signed-off-by: Andrew Thornton <art27@cantab.net>

zeripath · 2022-03-09T21:54:57Z

You have to tell us what version of Gitea you're reporting from.

Assuming you're using 1.16 - you can add a trace logger for the services/mirror as I have previously explained which will add significant amounts of tracing.

How to add trace logging for services/mirror

You can adjust your ini like this:

[log]
MODE=console, traceconsole
...

[log.traceconsole]
MODE=console
LEVEL=trace
EXPRESSION=services/mirror

Or you can run gitea manager logging add console -n traceconsole -l trace -e "services/mirror"

You can use a file logger too. Look at gitea manager logging add --help gitea manager logging add file --help

This is an interesting case. The error is coming from within:

gitea/modules/repository/repo.go

Lines 293 to 295 in 2448c5a

    
           // PushUpdateAddTag must be called for any push actions to add tag 
        
           func PushUpdateAddTag(repo *repo_model.Repository, gitRepo *git.Repository, tagName string) error { 
        
           	tag, err := gitRepo.GetTag(tagName)

The error string indicates that the ID it is trying to look for is actually an empty string but that doesn't make any sense at all.

I've created another PR to further improve the logging here, but if you add the tracelogger as above we can find out what is special about the repo that is causing this.

somera · 2022-03-10T07:44:04Z

I'm using 1.16.3.

zeripath · 2022-03-10T09:12:57Z

Unable to open checker for a5f1ab1c2af1bb0866438c793c390062bf0e39ec. Error: failed to run attr-check. Error: context canceled

Ignore these. They are nothing to do with your initial report. They are incorrectly being reported as an error when they're normal. PRs to fix them have already been merged and I recently closed a duplicate issue you opened about these.

zeripath · 2022-03-10T09:30:10Z

We need to find cases where:

Failed to synchronize tags to releases for repository: pushUpdateAddTag: GetTag: Length must be 40:

Is reported. Not the check-attr false positive.

somera · 2022-03-10T09:36:35Z

Unable to open checker for a5f1ab1c2af1bb0866438c793c390062bf0e39ec. Error: failed to run attr-check. Error: context canceled
Ignore these. They are nothing to do with your initial report. They are incorrectly being reported as an error when they're normal. PRs to fix them have already been merged and I recently closed a duplicate issue you opened about these.

Sorry. Wrong post.

Yet another issue has come up where the logging from SyncMirrors does not provide enough context. This PR adds more context to these logging events. Related #19038 Signed-off-by: Andrew Thornton <art27@cantab.net>

somera · 2022-03-10T10:10:17Z

I got this error on run "Garbage-Collection" cron. I will start it again.

Backport go-gitea#19045 Yet another issue has come up where the logging from SyncMirrors does not provide enough context. This PR adds more context to these logging events. Related go-gitea#19038 Signed-off-by: Andrew Thornton <art27@cantab.net>

Backport #19045 Yet another issue has come up where the logging from SyncMirrors does not provide enough context. This PR adds more context to these logging events. Related #19038 Signed-off-by: Andrew Thornton <art27@cantab.net>

somera · 2022-03-24T13:20:06Z

@zeripath I got this

2022/03/24 02:07:27 ...irror/mirror_pull.go:261:runSync() [E] SyncMirrors [repo: 225:games/triplea]: failed to synchronize tags to releases: unable to PushUpdateAddTag: "2.0.17694" to Repo[225:games/triplea]: unable to GetTag: Length must be 40:
        /source/services/mirror/mirror_pull.go:261 (0x1d49ddc)
        /source/services/mirror/mirror_pull.go:376 (0x1d4bc4d)
        /source/services/mirror/mirror.go:46 (0x1d45cb5)
        /source/services/mirror/mirror.go:148 (0x1d467d1)
        /source/modules/queue/unique_queue_channel.go:76 (0x14ffd76)
        /source/modules/queue/workerpool.go:406 (0x150863a)
        /source/modules/queue/workerpool.go:280 (0x1507956)
        /usr/local/go/src/runtime/asm_amd64.s:1581 (0x471520)

repo: https://github.com/triplea-game/triplea

zeripath · 2022-03-27T18:29:40Z

Sorry I missed your response.

Let's break down the log message

2022/03/24 02:07:27 ...irror/mirror_pull.go:261:runSync() [E] SyncMirrors [repo: 225:games/triplea]: failed to synchronize tags to releases: unable to PushUpdateAddTag: "2.0.17694" to Repo[225:games/triplea]: unable to GetTag: Length must be 40:

In repo with ID: 225 which has the path games/triplea and would therefore be GITEA_REPOSITORY_ROOT/games/triplea.git whilst PushUpdateAddTag was trying to work it tried to push the tag 2.0.17694 but the tag was not found.

So...

Does that tag exist in that repository?

somera · 2022-03-27T18:40:45Z

@zeripath I don't know. But the error happens during mirror update cron.

My local mirror:

GutHub repo:

https://github.com/triplea-game/triplea/tags?after=2.0.17751

zeripath · 2022-03-27T18:42:19Z

What are the results of:

git show-ref --tags | grep 2.0.17694

git show-ref --tags -- 2.0.17694

somera · 2022-03-27T18:56:01Z

$ git show-ref --tags | grep 2.0.17694
ef59fccf7e34a7f036829d2fdc4f5e09006ddb78 refs/tags/2.0.17694

$ git show-ref --tags -- 2.0.17694
ef59fccf7e34a7f036829d2fdc4f5e09006ddb78 refs/tags/2.0.17694

zeripath · 2022-03-27T19:35:57Z

This is on the gitea repository on the server right?

somera · 2022-03-27T19:38:52Z

I started new mirror and got this:

==> gitea.log <==
2022/03/27 21:37:54 ...s/repository/repo.go:146:MigrateRepositoryGitData() [E] Failed to synchronize tags to releases for repository: unable to PushUpdateAddTag: "1.9.0.0.9102" to Repo[10988:gitea_admin/triplea]: unable to GetTag: Length must be 40:
        /source/modules/repository/repo.go:146 (0x14fea5f)
        /source/services/migrations/gitea_uploader.go:116 (0x1dcb0b5)
        /source/services/migrations/migrate.go:201 (0x1de2f6c)
        /source/services/migrations/migrate.go:129 (0x1de2579)
        /source/services/task/migrate.go:112 (0x1e898f5)
        /source/services/task/task.go:33 (0x1e8a7a4)
        /source/services/task/task.go:55 (0x1e8aa6f)
        /source/modules/queue/workerpool.go:445 (0x1607ade)
        /source/modules/queue/workerpool.go:280 (0x1606c16)
        /usr/local/go/src/runtime/asm_amd64.s:1571 (0x46ffe0)

somera · 2022-03-27T19:40:44Z

This is on the gitea repository on the server right?

[So Mär 27][21:39:55][git]@[nuc-mini-celeron]:[~/data/repositories/games/triplea.git]$ git show-ref --tags | grep 2.0.17694
ef59fccf7e34a7f036829d2fdc4f5e09006ddb78 refs/tags/2.0.17694
[So Mär 27][21:40:19][git]@[nuc-mini-celeron]:[~/data/repositories/games/triplea.git]$ git show-ref --tags | grep 2.0.17694
ef59fccf7e34a7f036829d2fdc4f5e09006ddb78 refs/tags/2.0.17694

Strangely go-gitea#19038 appears to relate to an issue whereby a tag appears to be listed in `git show-ref --tags` but then does not appear when `git show-ref --tags -- short_name` is called. As a solution though I propose to stop the second call as it is unnecessary and only likely to cause problems. Fix go-gitea#19038 Signed-off-by: Andrew Thornton <art27@cantab.net>

zeripath · 2022-03-27T20:01:38Z

Pull and build #19236 and see if the problem still happens.

zeripath · 2022-03-27T20:02:07Z

The second line there is not right they are not supposed to be the same it should be git show-ref --tags -- 2.0.17694 .

However, I don't understand what is happening here.

The ref is being listed by git show-ref --tags but then when we try to use git show-ref --tags -- TAG_NAME it doesn't work.

I dunno.

I've got a patch that tries to avoid the second show-ref so I guess we try that?

somera · 2022-03-27T20:03:59Z

Pull and build #19236 and see if the problem still happens.

I can't do this on my gite instance.

Can you check this on https://try.gitea.io/ ?

somera · 2022-03-27T20:06:22Z

The second line there is not right they are not supposed to be the same it should be git show-ref --tags -- 2.0.17694 .

wrong copy & paste

So Mär 27][22:05:49][git]@[nuc-mini-celeron]:[~/data/repositories/games/triplea.git]$ git show-ref --tags -- 2.0.17694
ef59fccf7e34a7f036829d2fdc4f5e09006ddb78 refs/tags/2.0.17694

zeripath · 2022-03-27T20:44:03Z

Pull and build #19236 and see if the problem still happens.

I can't do this on my gite instance.

Can you check this on https://try.gitea.io/ ?

If I could replicate your issue I would have done so myself and I wouldn't have asked.

"I can't do this on my gitea instance" does not make sense. You have already determined that this is a bug of significant importance to you and your workflow. Running a small patch temporarily in order to test if a bug fix works, resolves or helps to explain your problem further cannot therefore break your instance further.

Honestly it's like working blindfolded in the dark and it's deeply infuriating - it unnecessarily delays the resolution of your issues and leads to these months long issue threads where I or others attempt to fix the problems that affect you - often unsuccessfully and if the issue is ever fixed it is because someone else has finally managed to replicate or reported the problem in a different way and has tested a bug fix.

Having just managed to successfully migrate https://github.com/triplea-game/triplea locally - I can see a number of potential issues with the code - not least that it is repeatedly calling git commands instead of reusing the open cat-files. This may be a cause of fd exhaustion so I guess I will take a look at reducing that and will continue to work on #19236 as an efficiency improvement.

However, if you are unwilling to try patches - I am afraid I don't know what to do.

PushAddUpdateTag only runs on tags which have been emitted by git show-ref --tags first. The error you receive indicates that a subsequent git show-ref --tags -- TAGNAME is then returning no id. How this could happen I don't know and it doesn't make sense to me. I guess all I can say is I will have a think and will comment again if again comes to me.

somera · 2022-03-27T20:59:57Z

@zeripath I don't want test unofficial release on my private prod instance. This is the main reason.

However, if you are unwilling to try patches - I am afraid I don't know what to do.

Hm ... you are developing software or fixes without testing them? In this case you have to add new mirror and check the log.

But it's ok. I will check it on other computer. But not today.

I need some infos:

should I compile the 1.16 branch? If not, which commit is it?
how to compile it for PostgreSQL? I'm not working with go.
which go version I need?

Yet another issue has come up where the logging from SyncMirrors does not provide enough context. This PR adds more context to these logging events. Related go-gitea#19038 Signed-off-by: Andrew Thornton <art27@cantab.net>

somera · 2022-03-28T19:07:58Z

@zeripath now I can test your fix on other temp instance. But please answer my questions. Thx.

zeripath · 2022-03-28T19:24:50Z

However, if you are unwilling to try patches - I am afraid I don't know what to do.

Hm ... you are developing software or fixes without testing them? In this case you have to add new mirror and check the log.

As I said - I am unable to replicate your problem on my testing machine.

I do not know why but the code works for me.

I need some infos:

should I compile the 1.16 branch? If not, which commit is it?

#19236 is the attempted fix PR backported to v1.16. Its branch name is backport-19235-v1.16 on my fork zeripath/gitea.git

The github ssh url is therefore git@github.com:zeripath/gitea

If you have the gh github cli tool:

gh pr checkout 19235

if you have a clone of the gitea repo you can either add my fork as a remote or use fetch to get the branch.

git fetch git@github.com:zeripath/gitea.git backport-19236-v1.16:backport-19236-v1.16
git checkout backport-19236-v1.16
make vendor

how to compile it for PostgreSQL? I'm not working with go.

TAGS="bindata sqlite sqlite_unlock_notify" make

You will need also need Node.js LTS

which go version I need?

It is probably best to just get 1.18

somera · 2022-03-28T19:35:37Z

hm ... I try this. My system is optimized for Java development.

It's possible to compile the code with go in docker?

docker run --rm -i -t -v $PWD:/v -w /v golang:1.18 make

Or you can attach an gitea-1.16.x-linux-amd64 to the ticket?

somera · 2022-03-28T20:10:08Z

My instance for testing hat at the moment problem with

2022/03/28 21:56:04 ...ervices/task/task.go:56:handle() [E] Run task failed: pq: doppelter Schlüsselwert verletzt Unique-Constraint »topic_pkey«
        /source/services/task/task.go:56 (0x1e8aa75)
        /source/modules/queue/workerpool.go:445 (0x1607ade)
        /source/modules/queue/workerpool.go:280 (0x1606c16)
        /usr/local/go/src/runtime/asm_amd64.s:1571 (0x46ffe0)

I updated this

from gitea-1.14.5-linux-amd64
to gitea-1.15.11-linux-amd64
to gitea-1.16.5-linux-amd64

zeripath · 2022-03-28T20:16:58Z

gitea doctor recreate-table topic

zeripath · 2022-03-28T20:53:09Z

I've managed to hopefully create a docker image at https://eldritchkitty.com/~andrew/backport-19235-v1.16.tar.gz

somera · 2022-03-28T20:54:33Z

I can't reproduce the problem on my 2nd gitea 1.16.5 instance. Strange.

I've managed to hopefully create a docker image at https://eldritchkitty.com/~andrew/backport-19235-v1.16.tar.gz

thx.

…ag (#19235) Strangely #19038 appears to relate to an issue whereby a tag appears to be listed in `git show-ref --tags` but then does not appear when `git show-ref --tags -- short_name` is called. As a solution though I propose to stop the second call as it is unnecessary and only likely to cause problems. I've also noticed that the tags calls are wildly inefficient and aren't using the common cat-files - so these have been added. I've also noticed that the git commit-graph is not being written on mirroring - so I've also added writing this to the migration which should improve mirror rendering somewhat. Fix #19038 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>

…ag (go-gitea#19235) Strangely go-gitea#19038 appears to relate to an issue whereby a tag appears to be listed in `git show-ref --tags` but then does not appear when `git show-ref --tags -- short_name` is called. As a solution though I propose to stop the second call as it is unnecessary and only likely to cause problems. I've also noticed that the tags calls are wildly inefficient and aren't using the common cat-files - so these have been added. I've also noticed that the git commit-graph is not being written on mirroring - so I've also added writing this to the migration which should improve mirror rendering somewhat. Fix go-gitea#19038 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>

…ag (#19235) (#19236) * Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235) Strangely #19038 appears to relate to an issue whereby a tag appears to be listed in `git show-ref --tags` but then does not appear when `git show-ref --tags -- short_name` is called. As a solution though I propose to stop the second call as it is unnecessary and only likely to cause problems. I've also noticed that the tags calls are wildly inefficient and aren't using the common cat-files - so these have been added. I've also noticed that the git commit-graph is not being written on mirroring - so I've also added writing this to the migration which should improve mirror rendering somewhat. Fix #19038 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> * fix rebase relict Co-authored-by: 6543 <6543@obermui.de>

zeripath mentioned this issue Mar 9, 2022

Improve SyncMirrors logging #19045

Merged

zeripath changed the title ~~Increase tag length~~ Failed to synchronize tags to release in SyncMirror with error: "pushUpdateAddTag: GetTag: Length must be 40:" Mar 9, 2022

This comment was marked as off-topic.

Sign in to view

zeripath mentioned this issue Mar 10, 2022

Improve SyncMirrors logging (#19045) #19050

Merged

noerw added the issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail label Mar 12, 2022

plaimbock mentioned this issue Mar 16, 2022

WebUI shows incorrect number of tags after migration (mirror) of git-lfs #17509

Closed

zeripath closed this as completed Mar 27, 2022

zeripath reopened this Mar 27, 2022

zeripath mentioned this issue Mar 27, 2022

Use full output of git show-ref --tags to get tags for PushUpdateAddTag #19235

Merged

zeripath mentioned this issue Mar 27, 2022

Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235) #19236

Merged

6543 closed this as completed in #19235 Mar 29, 2022

go-gitea locked and limited conversation to collaborators Apr 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to synchronize tags to release in SyncMirror with error: "pushUpdateAddTag: GetTag: Length must be 40:" #19038

Failed to synchronize tags to release in SyncMirror with error: "pushUpdateAddTag: GetTag: Length must be 40:" #19038

somera commented Mar 9, 2022

zeripath commented Mar 9, 2022

somera commented Mar 10, 2022

This comment was marked as off-topic.

zeripath commented Mar 10, 2022 •

edited

Loading

zeripath commented Mar 10, 2022

somera commented Mar 10, 2022

somera commented Mar 10, 2022

somera commented Mar 24, 2022

zeripath commented Mar 27, 2022

somera commented Mar 27, 2022

zeripath commented Mar 27, 2022

somera commented Mar 27, 2022

zeripath commented Mar 27, 2022

somera commented Mar 27, 2022

somera commented Mar 27, 2022

zeripath commented Mar 27, 2022

zeripath commented Mar 27, 2022

somera commented Mar 27, 2022

somera commented Mar 27, 2022

zeripath commented Mar 27, 2022

somera commented Mar 27, 2022 •

edited

Loading

somera commented Mar 28, 2022

zeripath commented Mar 28, 2022

somera commented Mar 28, 2022

somera commented Mar 28, 2022

zeripath commented Mar 28, 2022

zeripath commented Mar 28, 2022

somera commented Mar 28, 2022

Failed to synchronize tags to release in SyncMirror with error: "pushUpdateAddTag: GetTag: Length must be 40:" #19038

Failed to synchronize tags to release in SyncMirror with error: "pushUpdateAddTag: GetTag: Length must be 40:" #19038

Comments

somera commented Mar 9, 2022

Feature Description

Screenshots

zeripath commented Mar 9, 2022

somera commented Mar 10, 2022

This comment was marked as off-topic.

zeripath commented Mar 10, 2022 • edited Loading

zeripath commented Mar 10, 2022

somera commented Mar 10, 2022

somera commented Mar 10, 2022

somera commented Mar 24, 2022

zeripath commented Mar 27, 2022

somera commented Mar 27, 2022

zeripath commented Mar 27, 2022

somera commented Mar 27, 2022

zeripath commented Mar 27, 2022

somera commented Mar 27, 2022

somera commented Mar 27, 2022

zeripath commented Mar 27, 2022

zeripath commented Mar 27, 2022

somera commented Mar 27, 2022

somera commented Mar 27, 2022

zeripath commented Mar 27, 2022

somera commented Mar 27, 2022 • edited Loading

somera commented Mar 28, 2022

zeripath commented Mar 28, 2022

somera commented Mar 28, 2022

somera commented Mar 28, 2022

zeripath commented Mar 28, 2022

zeripath commented Mar 28, 2022

somera commented Mar 28, 2022

zeripath commented Mar 10, 2022 •

edited

Loading

somera commented Mar 27, 2022 •

edited

Loading