cron.update_mirrors broken in Gitea 1.16.0 #18607

somera · 2022-02-04T17:33:56Z

Gitea Version

1.16.0

Git Version

2.25.1

Operating System

Ubuntu 20.04.3

How are you running Gitea?

Precompiled gitea-1.16.0-linux-amd64

Database

PostgreSQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Description

I'm hosting a lot of mirrors on my Gitea instance. Some days ago I updated to 1.16.0.

I see -> it started at 04:00 AM, but ...

Last run with Gitea 1.15.x:

cron.update_mirrors needed with Gitea 1.15.x ~3h to update all mirrors. The cron runs fridays at 04:00 AM. And today was the first run.

I saw in gitea that the cron runs -> run count was 1. But the monitoring data shows no activity (RAM usage, CPU usage, Disk usage) between 04:00 and 07:00 AM.

Than I started (4-6x times) the cron in Gitea admin web console. It runs for some seconds. and I saw some activity.

But there is no date for the last run of the cron

After the 4-6 runs the sorting by last update time isn't showing any changes

All the projects with Feb 04 date are created today. But I can't see the updated mirrors when I change the sort order.

Why cron.update_mirrors is not updating all my mirrors? I can't see any errors in the Gitea logs.

Screenshots

No response

somera · 2022-02-04T20:49:25Z

@zeripath should I set now the new parameters from #16982?

Cause I added now

PULL_LIMIT = 100

and it runs longer.

After an update (1.15.x -> 1.16.0) I rather expect the old behavior.

somera · 2022-02-04T21:10:50Z

I increased it to

PULL_LIMIT = 2000

and the cron.update_mirrors run's ~8-10 minutes. Second time ~12 minutes. Means for me: 2 runns -> 4000 updated 2000 mirrors.

but ~20 minutes for 4000 mirrors it's to fast for me.

Questions:

I'm right or wrong?
Are there some optimizations? If yes -> which?

Sometimes I see what Gitea is doing

and sometimes not

Just missing the git call for a lot projects.

And it will be good to know that this cron was running (start date, finish date) and how many repos was updated.

You are showing some infos for some crons. But not for the mirror cron.

I just have to know: is the process working fine or not? Can I lost data?

somera · 2022-02-04T21:22:43Z

I run the cron 4x manually.

1 time with PULL_LIMIT = 100 and 3 times with PULL_LIMIT = 2000

And in the DB I see

I expected ~6100 repos with update_unix = 2022-02-04. But there are only 1675.

somera · 2022-02-04T21:36:59Z

Now I'm sure. The new process is broken.

6 runs: 1 time with PULL_LIMIT = 100 and 5 times with PULL_LIMIT = 2000

But

And if I start the cron again -> nothing happens.

And in the mirror table (next_update_unix column) I see, how many mirrors can be updated

somera · 2022-02-05T09:55:39Z

Since yesterday klicked x times on "Now sync" -> nothing happens

somera · 2022-02-05T10:17:23Z

I run today manually the cron. Gitea updated run some mirror updates. But the mirror.updated_unix column hasn't after the run ZERO changes.

Now I call the curl

curl -X 'POST' 'http://nuc-mini-celeron:3000/api/v1/repos/gaphor/in-app-notification-demo/mirror-sync' -H 'accept: application/json' -H 'Authorization: token xxxxx' -d ''
call for evry mirror again and the mirror.updated_unix is changing

But the whole mirror update process looks brocken.

somera · 2022-02-05T13:39:40Z

After spending some hours with Gitea 1.16.0 I'm disapointed. Disapointed cause Gitea is not working like in 1.15.x for mirror sync. Yes, there was changes. But the changes shouldn't change the existing behavior.

I sended ~40.000 curl calls (4x for all mirrors) to get this mirror_update status

1785 mirrors are not up2date.

This is my current config:

[repository]
ROOT = /var/lib/gitea/repositories
; Mirror sync queue length, increase if mirror syncing starts hanging
; DEPRECATED!!!
MIRROR_QUEUE_LENGTH = 20000

[mirror]
DEFAULT_INTERVAL = 8h
LENGTH = 20000

; Update mirrors
[cron.update_mirrors]
SCHEDULE = 0 0 4 * * 5 ; -> on 04.02.2022 04:00:00 it runs only for 8-10 minutes
PULL_LIMIT = -1
PUSH_LIMIT = -1

If I now start the update_mirrors cron manually, the oldest mirrors don't be updated. Nothing happens. Why?

If I open the repo with the oldest sync date

and I click on sync. It's not syncing. Why not? How long have I to wait?

somera · 2022-02-06T12:55:53Z

Same status after the very short cron run today at 04:00AM

lunny · 2022-02-06T13:42:27Z

Are there any logs?

somera · 2022-02-06T14:15:53Z

Are there any logs?

Hm ... if I switch TO DEUBUG I see a lot of SQL's.

Or this: #16982 (comment)

I think, the "problem" is the new feature implemented in #16982. Or you did other big changes.

somera · 2022-02-06T17:02:38Z

I added

[queue.mirror]
LENGTH = 20000

and than

./gitea manager flush-queues

and restart gitea. After I run mirror-cron manually I see no changes.

And I can bet, that if I would do my ~9000 curl calls, some of the mirrors will be synced.

The 1 repo with mirror.updated_unix = 2022-02-06 is a new repo which I added today.

zeripath · 2022-02-06T17:29:11Z

I've held off replying to this issue because I'm struggling to completely understand what is the problem and what is actually happening.

This is despite you having posted 11 comments...

So can we please have a succinct three line description as to what is happening, what you want to happen and what your configuration is? (that includes cron configuration.)

I will also remind you that you SPECIFICALLY asked for LIMIT_SIZE - I checked with you as to how it was supposed to work and I wrote a very long explanation as to what it was doing.

There have been many changes to mirroring this is not the only thing.

somera · 2022-02-06T17:46:51Z

I've held off replying to this issue because I'm struggling to completely understand what is the problem and what is actually happening.

This is despite you having posted 11 comments...

Sorry ... I was tested and I posted every try. And there are some problems now.

Started two times after gitea restart. But no last start date set

So can we please have a succinct three line description as to what is happening, what you want to happen and what your configuration is? (that includes cron configuration.)

I will also remind you that you SPECIFICALLY asked for LIMIT_SIZE - I checked with you as to how it was supposed to work and I wrote a very long explanation as to what it was doing.

Yes, this was a wish. But this implementation should not change the "normal process". This should be only for small optimization for people with a lot of mirrors on the gitea instance.

There have been many changes to mirroring this is not the only thing.

Short version on my problem.

I have 9000+ mirrors. The mirror sync cron was running one time a week and needed 3h to update all mirrors. And after update to gitea 1.16.0 it's not working anymore. The cron runs only for 8-10 seconds.

To update ~90% of the repos I sended yesterday 4 times 9000+ curl calls

curl -X 'POST' 'http://nuc-mini-celeron:3000/api/v1/repos/gaphor/in-app-notification-demo/mirror-sync' -H 'accept: application/json' -H 'Authorization: token xxxxx' -d ''

In this case the sync worked.

And now after flush-queue (cause it could be corrupt) if I start the mirror cron manually no one mirror will be updated. I tryied this 3 times.

My config now:

[repository]
ROOT = /var/lib/gitea/repositories
; Mirror sync queue length, increase if mirror syncing starts hanging
; DEPRECATED
MIRROR_QUEUE_LENGTH = 20000

[mirror]
DEFAULT_INTERVAL = 8h
LENGTH = 20000

[queue.mirror]
LENGTH = 20000

; Update mirrors
[cron.update_mirrors]
; Every day at 4AM
SCHEDULE = 0 0 4 * * *
PULL_LIMIT = -1
PUSH_LIMIT = -1

somera · 2022-02-06T19:31:50Z

@zeripath when I change the loglevel to

[log]
LEVEL            = Info

than I see 9000+ (each for every mirror) select statements in the log

2022/02/06 20:24:32 models/repo/repo.go:635:getRepositoryByID() [I] [SQL] SELECT "id", "owner_id", "owner_name", "lower_name", "name", "description", "website", "original_service_type", "original_url", "default_branch", "num_watches", "num_stars", "num_forks", "num_issues", "num_closed_issues", "num_pulls", "num_closed_pulls", "num_milestones", "num_closed_milestones", "num_projects", "num_closed_projects", "is_private", "is_empty", "is_archived", "is_mirror", "status", "is_fork", "fork_id", "is_template", "template_id", "size", "is_fsck_enabled", "close_issues_via_commit_in_any_branch", "topics", "trust_model", "avatar", "created_unix", "updated_unix" FROM "repository" WHERE "id"=$1 LIMIT 1 [4445] - 848.612µs

and than this

2022/02/06 20:24:32 ...s/repo/pushmirror.go:111:PushMirrorsIterate() [I] [SQL] SELECT "id", "repo_id", "remote_name", "interval", "created_unix", "last_update", "last_error" FROM "push_mirror" WHERE (last_update + ("interval" / $1) <= $2) AND ("interval" != 0) ORDER BY last_update ASC [1s 1644175472] - 1.891748ms

select. and no one mirror will be updated.

This are the 8-10 seconds the cron.update_mirrors needs to run and finish.

zeripath · 2022-02-06T19:45:19Z

OK let's look at that configuration first.

You have way too many mirrors to for a persistable-channel queue to ever be the correct queue for you.

Therefore change your update_checker queue to use a level queue and get rid of the channel queue related things.

[repository]
ROOT = /var/lib/gitea/repositories

[mirror]
DEFAULT_INTERVAL = 8h 

[queue.mirror]
TYPE=level; <- You have way too many mirrors for a channel or persistent-channel to be the right queue for you

; Update mirrors
[cron.update_mirrors]
; Every day at 4AM
SCHEDULE = 0 0 4 * * *
PULL_LIMIT = -1
PUSH_LIMIT = -1

Next I think we need address what the update_mirrors cron task does,

update_mirrors looks through the list of mirrors which are due to be updated - (e.g. next_update_unix <= Now & next_update_unix != 0) and sorts them into a last_updated order. It will then iterate through them.

It will then queue PULL_LIMIT pull mirrors and PUSH_LIMIT push mirrors for update.

The updates will be done by the workers on the other end of the queue. This will depend on your general queue configuration but often this is a scaling worker pool up to a maximum of 10 workers.

So ... If you are finding that nothing is being queued ... It would be useful to check the values of next_update_unix in the mirror table.

The PULL_LIMIT and PUSH_LIMIT code is working perfectly as described and so your problem is elsewhere.

Now to explain why we changed the default PULL_LIMIT and PUSH_LIMIT.

The vast majority of installs will benefit from this change as most people do not:

Change their cron configuration to only check things once per day (instead of changing the repository mirror update time.)
Have 9000 mirrors.

Your situation represents an edgecase of edgecases.

But I personally have tried to provide you with ways to make your personal situation easier.

The change to use a normal queue for mirrors (#17326) allows you the option to use a level queue for your underlying queue thus prevent gitea from seizing up due to a blocked queue. The PULL_LIMIT and PUSH_LIMIT options were requested by you gives you other options to consider changing your cron configuration back to /10 minutes (but you might actually need a PULL_LIMIT/PUSH_LIMIT to be percentages of the total number of mirrors (not sure here.))

You're not running Gitea in a normal way and that means you will always need to carefully think about things. In your situation you need to tune things properly and that is what we have provided for you.

somera · 2022-02-06T19:56:46Z

OK let's look at that configuration first.

You have way too many mirrors to for a persistable-channel queue to ever be the correct queue for you.

Therefore change your update_checker queue to use a level queue and get rid of the channel queue related things.
[repository]
ROOT = /var/lib/gitea/repositories

[mirror]
DEFAULT_INTERVAL = 8h 

[queue.mirror]
TYPE=level; <- You have way too many mirrors for a channel or persistent-channel to be the right queue for you

This is not defined in

https://github.com/go-gitea/gitea/blob/main/custom/conf/app.example.ini

and

https://docs.gitea.io/en-us/config-cheat-sheet/#mirror-mirror

; Update mirrors
[cron.update_mirrors]
; Every day at 4AM
SCHEDULE = 0 0 4 * * *
PULL_LIMIT = -1
PUSH_LIMIT = -1


Next I think we need address what the `update_mirrors` cron task does,

`update_mirrors` looks through the list of mirrors which are due to be updated - (e.g. `next_update_unix` <= Now & `next_update_unix != 0`) and sorts them into a last_updated order. It will then iterate through them.

It will then queue PULL_LIMIT pull mirrors and PUSH_LIMIT push mirrors for update.

The updates will be done by the workers on the other end of the queue. This will depend on your general queue configuration but often this is a scaling worker pool up to a maximum of 10 workers.

So ... If you are finding that nothing is being queued ... It would be useful to check the values of `next_update_unix` in the mirror table.

My current status for the mirrors

This means, that all 9000+ mirrors should be synced now.

The PULL_LIMIT and PUSH_LIMIT code is working perfectly as described and so your problem is elsewhere.

Now to explain why we changed the default PULL_LIMIT and PUSH_LIMIT.

The vast majority of installs will benefit from this change as most people do not:

Change their cron configuration to only check things once per day (instead of changing the repository mirror update time.)

Have 9000 mirrors.

Your situation represents an edgecase of edgecases.

;)

But I personally have tried to provide you with ways to make your personal situation easier.

Thx!

The change to use a normal queue for mirrors (#17326) allows you the option to use a level queue for your underlying queue thus prevent gitea from seizing up due to a blocked queue. The PULL_LIMIT and PUSH_LIMIT options were requested by you gives you other options to consider changing your cron configuration back to /10 minutes (but you might actually need a PULL_LIMIT/PUSH_LIMIT to be percentages of the total number of mirrors (not sure here.))

You're not running Gitea in a normal way and that means you will always need to carefully think about things. In your situation you need to tune things properly and that is what we have provided for you.

I added

[queue.mirror]
TYPE=level;

and restarted my Gitea. But the cron do the same: nothing.

The sync update with curl and directly in repo settings is working.

zeripath · 2022-02-06T20:01:14Z

@zeripath when I change the loglevel to
[log]
LEVEL            = Info

Change from what? Info is one of the lowest log levels and it will not be giving us any special information.

[log]
MODE=console, traceconsole
LEVEL=info

[log.traceconsole]
MODE=console
LEVEL=trace
EXPRESSION=services/mirror

Would be more useful.

OR even whilst gitea is running you can simply run:

./gitea manager logging add console --name traceconsole --level TRACE --expression services/mirror

And it will add trace level console logger that will emit TRACE level logs from events in the services/mirror files

than I see 9000+ (each for every mirror) select statements in the log

2022/02/06 20:24:32 models/repo/repo.go:635:getRepositoryByID() [I] [SQL] SELECT "id", "owner_id", "owner_name", "lower_name", "name", "description", "website", "original_service_type", "original_url", "default_branch", "num_watches", "num_stars", "num_forks", "num_issues", "num_closed_issues", "num_pulls", "num_closed_pulls", "num_milestones", "num_closed_milestones", "num_projects", "num_closed_projects", "is_private", "is_empty", "is_archived", "is_mirror", "status", "is_fork", "fork_id", "is_template", "template_id", "size", "is_fsck_enabled", "close_issues_via_commit_in_any_branch", "topics", "trust_model", "avatar", "created_unix", "updated_unix" FROM "repository" WHERE "id"=$1 LIMIT 1 [4445] - 848.612µs

That is your 9000 repositories being loaded and then added to the update queue.

and than this

2022/02/06 20:24:32 ...s/repo/pushmirror.go:111:PushMirrorsIterate() [I] [SQL] SELECT "id", "repo_id", "remote_name", "interval", "created_unix", "last_update", "last_error" FROM "push_mirror" WHERE (last_update + ("interval" / $1) <= $2) AND ("interval" != 0) ORDER BY last_update ASC [1s 1644175472] - 1.891748ms

These are push mirrors and it is clear that you have none.

select. and no one mirror will be updated.

This are the 8-10 seconds the cron.update_mirrors needs to run and finish.

Yes this is correct because the cron.update_mirrors task is simply adding mirrors to the queue to be updated - that is all it has ever done. It has never represented the actual work of doing the updating.

The work of updating a mirror will be done by workers on the queue. The mirror queue will scale its workers to do account for the amount of things in the queue.

But the cron do the same: nothing

I've tried to explain this to you before - the cron task update_mirrors DOES NOT represent the actual work of doing the updating. It has never done that. Previously you've had a proxy of this because in your situation you've been blocking the whole queue due to the number of mirrors you have.

somera · 2022-02-06T20:13:03Z

@zeripath when I change the loglevel to
You're not running Gitea in a normal way and that means you will always need to carefully think about things. In your situation you need to tune things properly and that is what we have provided for you.

What do you mean with "You're not running Gitea in a normal way"? ;)

If someone (not I) is running Gitea which is used by a lot of people with a lot of repos and mirrors. Than that person get the same problems.

I've tried to explain this to you before - the cron task update_mirrors DOES NOT represent the actual work of doing the updating. It has never done that. Previously you've had a proxy of this because in your situation you've been blocking the whole queue due to the number of mirrors you have.

Is this an design "problem"?

Understand or not. I'm not understand what exactly is blocked? The workes for the mirrors queue?

{
  "Name": "mirror",
  "DataDir": "/var/lib/gitea/data/queues/common",
  "BatchLength": 20,
  "QueueLength": 20000,
  "Timeout": 90000000000,
  "MaxAttempts": 10,
  "Workers": 0,
  "MaxWorkers": 10,
  "BlockTimeout": 1000000000,
  "BoostTimeout": 300000000000,
  "BoostWorkers": 1
}

It works with Gitea 1.15.x. -> is the old way isn't implemented anymore?

And the solution now is? At the moment I can send one a week the curl calls to update the mirrors.

zeripath · 2022-02-06T23:36:32Z

next_update_unix in the mirror table is the key thing for things to be added by the Cron task to the queue. Not updated_unix and certainly not updated_unix on the repository.

gitea/models/repo/mirror.go

Lines 124 to 128 in f393bc8

    
           return db.GetEngine(db.DefaultContext). 
        
           	Where("next_update_unix<=?", time.Now().Unix()). 
        
           	And("next_update_unix!=0"). 
        
           	OrderBy("updated_unix ASC"). 
        
           	Iterate(new(Mirror), f)

The 9k select repository statements says that:

gitea/services/mirror/mirror.go

Line 115 in f393bc8

return handler(idx, bean, pullLimit)

Happens for each repo

Thus you get a

gitea/services/mirror/mirror.go

Lines 67 to 70 in f393bc8

    
           item = SyncRequest{ 
        
           	Type:   PullMirrorType, 
        
           	RepoID: m.RepoID, 
        
           }

This will get pushed to the queue unless it is already in the queue.

gitea/services/mirror/mirror.go

Lines 92 to 99 in f393bc8

    
           // Check if this request is already in the queue 
        
           has, err := mirrorQueue.Has(&item) 
        
           if err != nil { 
        
           	return err 
        
           } 
        
           if has { 
        
           	return nil 
        
           }

gitea/services/mirror/mirror.go

Lines 101 to 104 in f393bc8

    
           // Push to the Queue 
        
           if err := mirrorQueue.Push(&item); err != nil { 
        
           	return err 
        
           }

Now you assert that calling sync with curl works.

So let's follow what that does:

gitea/routers/api/v1/repo/mirror.go

Line 17 in f393bc8

func MirrorSync(ctx *context.APIContext) {

gitea/routers/api/v1/repo/mirror.go

Line 51 in f393bc8

mirror_service.StartToMirror(repo.ID)

gitea/services/mirror/mirror.go

Lines 151 to 165 in f393bc8

    
           // StartToMirror adds repoID to mirror queue 
        
           func StartToMirror(repoID int64) { 
        
           	if !setting.Mirror.Enabled { 
        
           		return 
        
           	} 
        
           	go func() { 
        
           		err := mirrorQueue.Push(&SyncRequest{ 
        
           			Type:   PullMirrorType, 
        
           			RepoID: repoID, 
        
           		}) 
        
           		if err != nil { 
        
           			log.Error("Unable to push sync request for to the queue for push mirror repo[%d]: Error: %v", repoID, err) 
        
           		} 
        
           	}() 
        
           }

Which pushes to the same queue

Now you might argue that that push doesn't have a Has wrapped around it but... There's a Has internally in the push.

So what have we found:

PULL_LIMIT -1 must be working otherwise you wouldn't have 9K sql selects
Similarly the 9K selects imply that at least one point the next_update_unix query found all the cases.
Your own assertion about how curl syncs things means the queue infrastructure works at doing the syncing.
Which leaves the Has test at lines 99 above but there's a has in the push anyway.

I guess the question I have is what is making you think this isn't working?

So... One thing you could do is simply change the next_update_unix for a mirror manually. Put the tracer logger I suggested above on. And then click the Cron task button and follow what the logs do.

somera · 2022-02-06T23:42:43Z

Results for next_update_unix

If I use curl, that do something, but not for all curl calls.

I made 547 curl calls for 547 repos. And after gitea was ready, 266 repos still was not updated. Than I made second run for the 266 repos.

And this show me, that the sync mirror process isn't working. Cause in this case the manual cron start will update all my mirrors.

Before new manual start

Manual start ...

And this are the changes

Can you explain this? Gitea updates only 4 mirrors with mirror_next_update_unix = '2022-02-06'

And if I start the cron 2nd time ... no one repo will be updated.

zeripath · 2022-02-06T23:49:21Z

Go to monitor and tell me how many workers you have in your mirror queue right now. (Not initial configuration)

somera · 2022-02-06T23:50:39Z

{
  "Name": "mirror",
  "DataDir": "/var/lib/gitea/data/queues/common",
  "BatchLength": 20,
  "QueueLength": 20000,
  "Timeout": 90000000000,
  "MaxAttempts": 10,
  "Workers": 0,
  "MaxWorkers": 10,
  "BlockTimeout": 1000000000,
  "BoostTimeout": 300000000000,
  "BoostWorkers": 1
}

{
  "QueueLength": 20000,
  "BatchLength": 20,
  "BlockTimeout": 1000000000,
  "BoostTimeout": 300000000000,
  "BoostWorkers": 1,
  "MaxWorkers": 10,
  "Workers": 0,
  "Name": "mirror-channel"
}

But the number of workers it not important. If I put 1000 items into the queue and I have only 10 workers, that it need longer. And Gitea is not updating more than 1 repo in a row.

zeripath · 2022-02-06T23:51:04Z

That is initial configuration

somera · 2022-02-06T23:54:12Z

If I refresh the config side for the mirror queue I don't see any changes.

somera · 2022-02-06T23:55:15Z

"This queue surrounds other queues and has no worker pool itself."

zeripath · 2022-02-06T23:56:03Z

So... Mirror-channel?

somera · 2022-02-06T23:57:02Z

zeripath · 2022-02-07T00:14:03Z

I guess a trick for that is to wait until that worker was due to timeout and then add another worker manually yourself and see if that finishes off the work

zeripath · 2022-02-07T00:15:37Z

Actually a flush worker would be better

zeripath · 2022-02-07T00:23:43Z

Yup I bet this is this problem. When the zero worker times out unless there's a push the lack of worker won't be noticed.

Workaround just set workers=1 in [Queue.mirrors] or flush the queue. I'll have a think - likely when the managedQueue loses its final worker and if there's something in the queue it should zeroboost again.

somera · 2022-02-07T00:31:56Z

Yup I bet this is this problem. When the zero worker times out unless there's a push the lack of worker won't be noticed.

Workaround just set workers=1 in [Queue.mirrors] or flush the queue. I'll have a think - likely when the managedQueue loses its final worker and if there's something in the queue it should zeroboost again.

I added this

[queue.mirror]
LENGTH = 20000
WORKERS = 1

and set the limits to 100 (only for testing)

[cron.update_mirrors]
; Every day at 4AM
SCHEDULE = 0 0 4 * * *
PULL_LIMITS = 100
PUSH_LIMITS = 100

Than restart gitea

And start the cron manually ... no repo was synced.

I see this

and no other activity. after 8-10 seconds this task is finished.

I made 4 tests

2022/02/07 01:22:00 ...ces/mirror/mirror.go:56:Update() [T] Doing: Update
2022/02/07 01:22:09 ...ces/mirror/mirror.go:129:Update() [T] Finished: Update
2022/02/07 01:24:19 ...ces/mirror/mirror.go:56:Update() [T] Doing: Update
2022/02/07 01:24:28 ...ces/mirror/mirror.go:129:Update() [T] Finished: Update
2022/02/07 01:28:16 ...ces/mirror/mirror.go:56:Update() [T] Doing: Update
2022/02/07 01:28:25 ...ces/mirror/mirror.go:129:Update() [T] Finished: Update
2022/02/07 01:31:05 ...ces/mirror/mirror.go:56:Update() [T] Doing: Update
2022/02/07 01:31:14 ...ces/mirror/mirror.go:129:Update() [T] Finished: Update

somera · 2022-02-07T00:48:57Z

Yup I bet this is this problem. When the zero worker times out unless there's a push the lack of worker won't be noticed.

Workaround just set workers=1 in [Queue.mirrors] or flush the queue. I'll have a think - likely when the managedQueue loses its final worker and if there's something in the queue it should zeroboost again.

But what is the difference to curl?

curl is an external call which put the mirror into the queue. And it works.

What is the difference between the cron call? If you try to explain this, you will find the "problem". ;)

lunny · 2022-02-07T07:24:05Z

OK. I found something. It seems the mirror address is not right. Which may caused by #15157 . PR #18649

somera · 2022-02-07T07:57:46Z

OK. I found something. It seems the mirror address is not right. Which may caused by #15157 . PR #18649

What does it mean?

Should cron.update_mirrors be disabled? And can something go wrong here?

It is possible for the zero worker to timeout before all the work is finished. This may mean that work may take a long time to complete because a worker will only be induced on repushing. Fix go-gitea#18607 Signed-off-by: Andrew Thornton <art27@cantab.net>

* Restart zero worker if there is still work to do It is possible for the zero worker to timeout before all the work is finished. This may mean that work may take a long time to complete because a worker will only be induced on repushing. Also ensure that requested count is reset after pulls and push mirror sync requests and add some more trace logging to the queue push. Fix #18607 Signed-off-by: Andrew Thornton <art27@cantab.net>

Backport go-gitea#18658 It is possible for the zero worker to timeout before all the work is finished. This may mean that work may take a long time to complete because a worker will only be induced on repushing. Also ensure that requested count is reset after pulls and push mirror sync requests and add some more trace logging to the queue push. Fix go-gitea#18607 Signed-off-by: Andrew Thornton <art27@cantab.net>

* Restart zero worker if there is still work to do (#18658) Backport #18658 It is possible for the zero worker to timeout before all the work is finished. This may mean that work may take a long time to complete because a worker will only be induced on repushing. Also ensure that requested count is reset after pulls and push mirror sync requests and add some more trace logging to the queue push. Fix #18607 Signed-off-by: Andrew Thornton <art27@cantab.net> * Update modules/queue/workerpool.go

somera · 2022-02-11T18:39:22Z

After

Run:

gitea manager flush-queues

Wait for it to finish.

Shutdown Gitea and delete the /data/queues/common folder.

Restart.

Gitea is syncing only ~220 mirrors.

zeripath · 2022-02-11T19:03:33Z

Are you running 1.16-head?

somera · 2022-02-11T19:06:11Z

Are you running 1.16-head?

No. I can't run this on my instance.

zeripath · 2022-02-11T19:09:40Z

I'm not suggesting that you run 1.17/main - I am simply suggesting that you move the 1.16-dev or 1.16 which tracks the backports and bug fixes that will become 1.16.2 in future in the next week.

Are you at least running with WORKERS=1 so that there is a permanent worker for the mirror queue?

somera · 2022-02-11T19:14:28Z

Powered by Gitea Version: 1.16.1

lrwxrwxrwx 1 git git        24 Feb  6 17:20 gitea -> gitea-1.16.1-linux-amd64
-rwxr-xr-x 1 git git 106011280 Feb  6 14:44 gitea-1.16.1-linux-amd64

somera · 2022-02-11T19:33:15Z

I'm not suggesting that you run 1.17/main - I am simply suggesting that you move the 1.16-dev or 1.16 which tracks the backports and bug fixes that will become 1.16.2 in future in the next week.

Are you at least running with WORKERS=1 so that there is a permanent worker for the mirror queue?

Yes.

I restarted repeated flush-queues ... and started the mirror process again.

somera · 2022-02-11T19:37:56Z

It looks better now. And Gitea is syncing the mirrors.

somera · 2022-02-11T19:51:38Z

It's possible to stop the whole sync mirror process?

If I press on yellow marked trash. Will this stop only for the one mirror or whole cron?

zeripath · 2022-02-11T20:57:20Z

Just one mirror - if you want to stop all mirroring you'll need to stop the queue worker.

somera · 2022-02-12T15:27:57Z

Just one mirror - if you want to stop all mirroring you'll need to stop the queue worker.

Thx. I will try this other time.

Some update ...

After last try:

flush-queues
stop gitea
remove ./../data/queues/common folder
start gitea

with manual cron start I could update all mirrors in one row. Good. Looks like this helps. But ...

About 16-18 hours (DEFAULT_INTERVAL = 8h) later I wanted see what happens, if I start the cron again.

My actual configuration is:

[mirror]
; Default interval as a duration between each check
DEFAULT_INTERVAL = 8h
; Min interval as a duration must be > 1m
MIN_INTERVAL = 10m
LENGTH = 20000
TYPE = level

[queue.mirror]
LENGTH = 20000
WORKERS = 1

[cron.update_mirrors]
; Every day at 4AM
SCHEDULE = 0 0 4 * * *
PULL_LIMITS = 1000
PUSH_LIMITS = 1000

But "nothing" happens. Gite updates ~10 mirrors.

somera · 2022-02-18T08:16:57Z

Only ~50 mirrors are updated every day on the 04:00AM run

With the configuration which I postet in the comment above.

somera · 2022-02-24T20:24:52Z

@zeripath I opened #18895

* Restart zero worker if there is still work to do It is possible for the zero worker to timeout before all the work is finished. This may mean that work may take a long time to complete because a worker will only be induced on repushing. Also ensure that requested count is reset after pulls and push mirror sync requests and add some more trace logging to the queue push. Fix go-gitea#18607 Signed-off-by: Andrew Thornton <art27@cantab.net>

somera mentioned this issue Feb 5, 2022

[Feature] cron.update_mirrors - LIMIT_SIZE config parameter #16982

Closed

somera changed the title ~~cron.update_mirrors looks broken in Gitea 1.16.0~~ cron.update_mirrors broken in Gitea 1.16.0 Feb 5, 2022

somera mentioned this issue Feb 6, 2022

gomail: could not send email 1: gomail: invalid address "": mail: no address #18613

Closed

zeripath mentioned this issue Feb 7, 2022

Restart zero worker if there is still work to do #18658

Merged

zeripath closed this as completed in #18658 Feb 8, 2022

zeripath mentioned this issue Feb 8, 2022

Restart zero worker if there is still work to do (#18658) #18672

Merged

somera mentioned this issue Feb 25, 2022

cron.update_mirrors updates only 51 mirrors in one run #18895

Closed

go-gitea locked and limited conversation to collaborators Apr 28, 2022

cron.update_mirrors broken in Gitea 1.16.0 #18607

cron.update_mirrors broken in Gitea 1.16.0 #18607

Comments

somera commented Feb 4, 2022 • edited Loading

Gitea Version

Git Version

Operating System

How are you running Gitea?

Database

Can you reproduce the bug on the Gitea demo site?

Log Gist

Description

Screenshots

somera commented Feb 4, 2022

somera commented Feb 4, 2022

somera commented Feb 4, 2022

somera commented Feb 4, 2022

somera commented Feb 5, 2022

somera commented Feb 5, 2022

somera commented Feb 5, 2022 • edited Loading

somera commented Feb 6, 2022 • edited Loading

lunny commented Feb 6, 2022

somera commented Feb 6, 2022

somera commented Feb 6, 2022 • edited Loading

zeripath commented Feb 6, 2022

somera commented Feb 6, 2022 • edited Loading

somera commented Feb 6, 2022 • edited Loading

zeripath commented Feb 6, 2022

somera commented Feb 6, 2022 • edited Loading

zeripath commented Feb 6, 2022

somera commented Feb 6, 2022 • edited Loading

zeripath commented Feb 6, 2022

somera commented Feb 6, 2022 • edited Loading

zeripath commented Feb 6, 2022

somera commented Feb 6, 2022 • edited Loading

zeripath commented Feb 6, 2022

somera commented Feb 6, 2022

somera commented Feb 6, 2022

zeripath commented Feb 6, 2022

somera commented Feb 6, 2022

zeripath commented Feb 7, 2022 • edited Loading

zeripath commented Feb 7, 2022

zeripath commented Feb 7, 2022

somera commented Feb 7, 2022 • edited Loading

somera commented Feb 7, 2022

lunny commented Feb 7, 2022

somera commented Feb 7, 2022

somera commented Feb 11, 2022

zeripath commented Feb 11, 2022

somera commented Feb 11, 2022

zeripath commented Feb 11, 2022

somera commented Feb 11, 2022

somera commented Feb 11, 2022

somera commented Feb 11, 2022 • edited Loading

somera commented Feb 11, 2022

zeripath commented Feb 11, 2022

somera commented Feb 12, 2022

somera commented Feb 18, 2022 • edited Loading

somera commented Feb 24, 2022 • edited Loading

somera commented Feb 4, 2022 •

edited

Loading

somera commented Feb 5, 2022 •

edited

Loading

somera commented Feb 6, 2022 •

edited

Loading

somera commented Feb 6, 2022 •

edited

Loading

somera commented Feb 6, 2022 •

edited

Loading

somera commented Feb 6, 2022 •

edited

Loading

somera commented Feb 6, 2022 •

edited

Loading

somera commented Feb 6, 2022 •

edited

Loading

somera commented Feb 6, 2022 •

edited

Loading

somera commented Feb 6, 2022 •

edited

Loading

zeripath commented Feb 7, 2022 •

edited

Loading

somera commented Feb 7, 2022 •

edited

Loading

somera commented Feb 11, 2022 •

edited

Loading

somera commented Feb 18, 2022 •

edited

Loading

somera commented Feb 24, 2022 •

edited

Loading