When pushing only the first branch/tag gets PR/Release marked

[mixman68]

• Gitea version (or commit ref): 1.8.1
• Git version: 2.11.0
• Operating system: Debian 9 stretch
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL) https://try.gitea.io/djgreg13/bugtag
  • No
  • Not relevant
• Log gist:

Description

If i push a commit in master and a tag with --all options, the front is desynchronised with my repo.
https://try.gitea.io/djgreg13/bugtag is the example, i push master and tag with --all

If i clone the repo, i see the tag, but on front no tags is shown in Releases page

If i push only a new tag, an automatic entry will be created on Releases page

I except to see in this page tag 1.0.0 (pushed with --all) and tag 1.0.1 (pushed with --tags) or i see only 1.0.1

Screenshots

you can see on https://try.gitea.io/djgreg13/bugtag

[zeripath]

This is due to the breaks in:

gitea/cmd/hook.go

Lines 193 to 217 in 54bd63c

	if newCommitID != git.EmptySHA && strings.HasPrefix(refFullName, git.BranchPrefix) {
	branch := strings.TrimPrefix(refFullName, git.BranchPrefix)
	repo, pullRequestAllowed, err := private.GetRepository(repoID)
	if err != nil {
	log.GitLogger.Error("get repo: %v", err)
	break
	}
	if !pullRequestAllowed {
	break
	}
	baseRepo := repo
	if repo.IsFork {
	baseRepo = repo.BaseRepo
	}
	if !repo.IsFork && branch == baseRepo.DefaultBranch {
	break
	}
	pr, err := private.ActivePullRequest(baseRepo.ID, repo.ID, baseRepo.DefaultBranch, branch)
	if err != nil {
	log.GitLogger.Error("get active pr: %v", err)
	break
	}

[zeripath]

Most of these should probably be continue

[zeripath]

However this problem is fixed in my #6993

[mrsdizzie]

Wouldn't this be happening in:

gitea/models/update.go

Lines 187 to 282 in 181b7c9

	func pushUpdate(opts PushUpdateOptions) (repo *Repository, err error) {
	isNewRef := opts.OldCommitID == git.EmptySHA
	isDelRef := opts.NewCommitID == git.EmptySHA
	if isNewRef && isDelRef {
	return nil, fmt.Errorf("Old and new revisions are both %s", git.EmptySHA)
	}
	repoPath := RepoPath(opts.RepoUserName, opts.RepoName)
	gitUpdate := exec.Command("git", "update-server-info")
	gitUpdate.Dir = repoPath
	if err = gitUpdate.Run(); err != nil {
	return nil, fmt.Errorf("Failed to call 'git update-server-info': %v", err)
	}
	owner, err := GetUserByName(opts.RepoUserName)
	if err != nil {
	return nil, fmt.Errorf("GetUserByName: %v", err)
	}
	repo, err = GetRepositoryByName(owner.ID, opts.RepoName)
	if err != nil {
	return nil, fmt.Errorf("GetRepositoryByName: %v", err)
	}
	gitRepo, err := git.OpenRepository(repoPath)
	if err != nil {
	return nil, fmt.Errorf("OpenRepository: %v", err)
	}
	if err = repo.UpdateSize(); err != nil {
	log.Error("Failed to update size for repository: %v", err)
	}
	var commits = &PushCommits{}
	if strings.HasPrefix(opts.RefFullName, git.TagPrefix) {
	// If is tag reference
	tagName := opts.RefFullName[len(git.TagPrefix):]
	if isDelRef {
	err = pushUpdateDeleteTag(repo, gitRepo, tagName)
	if err != nil {
	return nil, fmt.Errorf("pushUpdateDeleteTag: %v", err)
	}
	} else {
	// Clear cache for tag commit count
	cache.Remove(repo.GetCommitsCountCacheKey(tagName, true))
	err = pushUpdateAddTag(repo, gitRepo, tagName)
	if err != nil {
	return nil, fmt.Errorf("pushUpdateAddTag: %v", err)
	}
	}
	} else if !isDelRef {
	// If is branch reference
	// Clear cache for branch commit count
	cache.Remove(repo.GetCommitsCountCacheKey(opts.RefFullName[len(git.BranchPrefix):], true))
	newCommit, err := gitRepo.GetCommit(opts.NewCommitID)
	if err != nil {
	return nil, fmt.Errorf("gitRepo.GetCommit: %v", err)
	}
	// Push new branch.
	var l *list.List
	if isNewRef {
	l, err = newCommit.CommitsBeforeLimit(10)
	if err != nil {
	return nil, fmt.Errorf("newCommit.CommitsBeforeLimit: %v", err)
	}
	} else {
	l, err = newCommit.CommitsBeforeUntil(opts.OldCommitID)
	if err != nil {
	return nil, fmt.Errorf("newCommit.CommitsBeforeUntil: %v", err)
	}
	}
	commits = ListToPushCommits(l)
	}
	if opts.RefFullName == git.BranchPrefix+repo.DefaultBranch {
	UpdateRepoIndexer(repo)
	}
	if err := CommitRepoAction(CommitRepoActionOptions{
	PusherName: opts.PusherName,
	RepoOwnerID: owner.ID,
	RepoName: repo.Name,
	RefFullName: opts.RefFullName,
	OldCommitID: opts.OldCommitID,
	NewCommitID: opts.NewCommitID,
	Commits: commits,
	}); err != nil {
	return nil, fmt.Errorf("CommitRepoAction: %v", err)
	}
	return repo, nil
	}

Calling pushUpdateAddTag() etc...?

gitea/models/update.go

Line 111 in 181b7c9

func pushUpdateAddTag(repo *Repository, gitRepo *git.Repository, tagName string) error {

I suspect what happens is that for whatever reason sometimes when a tag gets pushed that pushUpdateAddTag() fails or isn't called, and then the tag is present on the remote and never run through pushUpdateAddTag() again. There are several places pushUpdateTag() could error, and several more even before that call, but you would have already needed error logging turned on to see I think.

Migrating or mirroring these example repos like the one above shows the proper releases (both 1.0.0 and 1.0.1) because it runs SyncReleasesWithTags (which then runs pushUpdateAddTag() for each tag )which does see that tag and create a proper release:

https://try.gitea.io/mrsdizzie/bugtag

FWIW the initial comment says "I except to see in this page tag 1.0.0 (pushed with --all) "
but git push --all doesn't push tags. However, it should have then pushed both tags on the next git push --tags (which it does in my testing, and I've yet to be able to reproduce pushing a tag, or several tags at once, that doesn't generate a release for each tag).

I want to find the cause of this and fix in case of a bug, but I also wonder if it would be helpful to add a button in the repo settings that can run SyncReleasesWithTags in case of any errors like this, which would at least provide a way to recover.

[zeripath]

That never gets called.

Look again at hook post-receive.

If you are pushing multiple things the breaks will stop the processing of everything after them.

To reproduce this just clone his repo and git push --tags to a local Gitea - only the first tag received will get a release made.

[zeripath]

#6993 fixes this because the private.HookPostReceive is called for every line in the push - but if you wanted to fix it without that just change those breaks to continue and it's fixed, however I'm not doing that as I'm certain that #6993 is a better way of doing hooks.

[mrsdizzie]

Hm when I clone the test repo and push it to a local repo then do git push --tagsit shows both of the tags as releases.

In addition, I can do something like this:

$ git tag test1
$ git tag test2
$ git tag test3
$ git push --tags

and it will create a release for test1, test2, test3 so I don't quite understand the only first tag gets processed part. Those hooks are running but they aren't stopping pushing multiple tags.

Are you not seeing that?

[zeripath]

Weird I don't get that on master. Did you push --all before pushing --tags?

[zeripath]

Yeah you're right the breaks shouldn't affect tag only pushing as they should only due for branch pushing.

I was definitely able to reproduce the problem on master with:

git push --all local
git push --tags local

[mrsdizzie]

Interesting, if I do the same thing it shows the release fine. Just so we're on the same page, I'm creating a new empty repo locally ("reproduce") and then doing this:

[mrsdizzie@mbp ~/Documents/source/github ] >  git clone https://try.gitea.io/djgreg13/bugtag.git
Cloning into 'bugtag'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
[mrsdizzie@mbp ~/Documents/source/github ] >  cd bugtag/
[mrsdizzie@mbp ~/Documents/source/github/bugtag (master) ] >  git remote add local http://localhost:3000/mrsdizzie/reproduce.git
[mrsdizzie@mbp ~/Documents/source/github/bugtag (master) ] >  git push local --all
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 212 bytes | 212.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To http://localhost:3000/mrsdizzie/reproduce.git
 * [new branch]      master -> master
[mrsdizzie@mbp ~/Documents/source/github/bugtag (master) ] >  git push local --tags
Total 0 (delta 0), reused 0 (delta 0)
To http://localhost:3000/mrsdizzie/reproduce.git
 * [new tag]         1.0.0 -> 1.0.0
 * [new tag]         1.0.1 -> 1.0.1

And then I can see both releases in Gitea. Tried via SSH too just to rule out any possible difference. Do you get any logging from the pushUpdate function on your end? Or maybe you see something obvious that I'm missing in the way I am testing it?

[zeripath]

Yeah that's what I did.

Hmm. I'll have to try it again.

Maybe it's an intermittent locking Heisenbug?

[zeripath]

Just realised that in post-receive we probably shouldn't stop on error because it's too late as the commits are in the repo - so we have to complete.

It's almost like we need a transaction log or some way of getting Gitea to just refresh the whole repo - as you suggested. (This would also be useful for allowing gitea to just take over repos.)

[mrsdizzie]

I'd be curious if you have error logging on to see if there is anything coming from the pushUpdate or pushUpdateAddTag functions (as there are places it could exit before creating the release but after pushing the tag). Or just anywhere I guess! I was hoping to be able to reproduce it as you are to see whats actually happening but I can't :(

And yea perhaps SyncReleasesWithTags should be made available to the user in some way, because as of now even if you found why the release wasn't made and fixed it (or explained why it was a valid error) there is no way to make it try again.

[mixman68]

Since git 2.4 there is an option for push tags with commits but it is disabled by default git config --global push.followTags true So if I do push --all as i tell, the follow-tags is by default to true ;) i tested git push --mirror, this commang bugs too with tags, there are missing into release tab. Only commits are shown in timeline. I tested with gitea 1.5, it's works fine, our previous version. So it is a regression with newer version

[mrsdizzie]

Ah, ok that is a very helpful detail! I can reproduce this now with specific steps.

The git config --global push.followTags true / --follow-tags option only works for annotated tags:

https://git-scm.com/docs/git-push#Documentation/git-push.txt---follow-tags

Assuming your tag was annotated. In testing, if I create an annotated tag and push it with the --follow-tags option then gitea doesn't create a release (but the tag is pushed to the server successfully). If I create an annotated tag and push it with git push --tags then it does see it.

This appears to be because of this check:

gitea/models/update.go

Lines 222 to 223 in 181b7c9

	if strings.HasPrefix(opts.RefFullName, git.TagPrefix) {
	// If is tag reference

When using git push --tags the opts.RefFullName: is refs/tags/tagname which passes that check. When using git push --all it is refs/heads/branchname and doesn't pass that test so it never runs pushUpdateAddTag.

Not sure the best way to check for tags in the git push --all situation (there doesn't appear to be enough info in opts for that currently), but that seems to be the likely cause here.

[zeripath]

So when I first looked at this I thought this might be an annotated Vs unannotated issue but looking at the repo on try I'm not convinced either of those tags are annotated.

[mrsdizzie]

It isn't about annotate as much as it is about the RefFullName not starting with /refs/tags.

git push --all just seems to only push annotated tags (but you can push them fine with git push --tags). There are maybe other situations where you can push a tag and that RefFullName value might not start with /refs/tags also (I'm not that familiar)

[mixman68]

@mrsdizzie ,
with the --mirror option of git, i have the same issue,

In the git documentation, they tell :

--mirror

    Instead of naming each ref to push, specifies that all refs under refs/ (which includes but is not limited to refs/heads/, refs/remotes/, and refs/tags/) be mirrored to the remote repository. Newly created local refs will be pushed to the remote end, locally updated refs will be force updated on the remote end, and deleted refs will be removed from the remote end. This is the default if the configuration option remote.<remote>.mirror is set.

So in the mirror push, the tag is not show in Releases tab as follow-tags behaviour, so in mirror push, /refs/tags is pushed

[zeripath]

@djgreg13 would you be able to build #6993 and confirm that it solves this issue?

[mrsdizzie]

@zeripath I built that PR and it does fix the test case that I was able to reproduce as those tags are now seen by pushUpdate

[mixman68]

@mrsdizzie, thanks you for the test My connection is too bad I need to wait Monday for build :/

…

Le 24 mai 2019 à 17:48, mrsdizzie ***@***.***> a écrit : @zeripath I built that PR and it does fix the test case that I was able to reproduce as those tags are now seen by pushUpdate — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mixman68]

Hello, how can resync missed tags from version 1.8.1 to new 1.9.0 ?