Make the Mirror Queue a queue

[zeripath]

For some reason the mirror syncing queue never got converted to the more modern queue format. This PR converts this to a modern queue.

There is also a bug in the from the repo-archive queue PR - the assumption was made that uniqueness could be enforced with by checking equality in a map in channel unique queues - however this only works for primitive types - which was the initial intention but is an imperfect. This is fixed by marshalling the data and placing the martialled data in the unique map instead.

The documentation is also updated to add information about the deprecated configuration values.

Signed-off-by: Andrew Thornton art27@cantab.net

[wxiaoguang]

I think we can introduce a func generateDataTaskKey(data interface{}) string to generate the keys for data. It's easier for developers to change/improve the key generating method in future.

ps: just a little concern, should we limit the generated key's length? eg: if a key is too long, we can use key[:lengthLimit] + "..." + sha256hex(data) instead.

[zeripath]

Json marshalability is tightly expected by the queues so it's not really replaceable. Unless you were thinking of something like allowing pushed Data to express some interface(s):

type UniqueKeyed interface {
   UniqueKey() string
}

type Conversion interface {
  ToQueue() ([]byte, error)
  FromQueue([]byte) error
}

type defaultConverter struct {
  Data
}

func (d defaultConverter) ToQueue() ([]byte, error) {
  return json.Marshal(d.Data)
}

func (d defaultConverter) FromQueue(bs []byte) error {
  return json.Unmarshal(bs, d.Data)
}

func AsConversion(data Data) Conversion {
   if conversion, ok := data.(Conversion); ok {
     return conversion
   }
   return defaultConverter{Data: data}
}

// with defaultUniqueKeyed structs and AsUniqueKeyed() similarly.

Then wired in as appropriate throughout the queues.

---

As this isn't exposed to users I think the size isn't too much of an issue - developers should think carefully about the size of things that they push in to any queue though and we as maintainers should be aware that these things shouldn't be too large.

---

In some ways as there is no current requirement for this - deciding to do this now we would be choosing an interface and API that would bind our hands but if you really think its needed then the above would be how I would do it.

[wxiaoguang]

What I mean isn't that complex. I was thinking about:

func generateDataTaskKey(data interface{}) (string, err) {
    bs, err := json.Marshal(data)
    if err != nil {
        return err
    }
    return string(bs)
}

...

dataTaskKey, err := generateDataTaskKey(data)
...
delete(queue.table, dataTaskKey))
...
if _, ok := q.table[dataTaskKey]; ok {
...
}
...
q.table[dataTaskKey] = true
...

[delvh]

I agree with part a, the common function.
I am unsure about part b, I would rather say no as name clashes can then happen more easily.

[zeripath]

This is not the only unique queue implementation - if we were to change the keying from json marshalling here in the ChannelUniqueQueue we'd have to change it everywhere. This PR is simply a bugfix ensuring that the keying of the ChannelUniqueQueue is the same as the other queues .

If we want to make it possible to change the keying algorithm then we need to do something like above and wire it in correctly otherwise someone will simply change one queue type and think it's fine.

Whilst DRY is a good thing - if you make a helper you're saying this implementation can be changed easily when it absolutely cannot be.

[delvh]

I still would like this helper function, no matter whether that mechanism is easy to replace or not.
Then I'll approve the PR.

[zeripath]

I honestly do not see where this would be helpful.

Please feel free to send me a PR with this helper function because I cannot see where it would be useful.

[delvh]

I agree with part a, the common function.
I am unsure about part b, I would rather say no as name clashes can then happen more easily.

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (main@b9a2f26). Click here to learn what that means.
The diff coverage is 21.42%.

@@           Coverage Diff           @@
##             main   #17326   +/-   ##
=======================================
  Coverage        ?   45.19%           
=======================================
  Files           ?      784           
  Lines           ?    88227           
  Branches        ?        0           
=======================================
  Hits            ?    39870           
  Misses          ?    41881           
  Partials        ?     6476           

Impacted Files	Coverage Δ
modules/queue/unique_queue_channel.go	2.17% <0.00%> (ø)
modules/setting/mailer.go	52.63% <ø> (ø)
modules/setting/repository.go	58.49% <ø> (ø)
services/mirror/mirror.go	9.45% <5.55%> (ø)
modules/setting/queue.go	67.77% <61.90%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9a2f26...78b00fe. Read the comment docs.

[zeripath]

make lgtm work