new Unique option API

[hibiken]

Several feedbacks on the Unique option (#369, #376, and #407) suggest that we need to allow users to specify more than TTL of the uniqueness lock. Some users need to also specify:

• Whether to release or keep the lock when the task is successfully processed.
• The unique key used for the uniqueness lock.

The Unique option is based on the idea that the enqueued task creates a uniqueness lock with a given TTL and it'll be guaranteed to be unique within that time period.

Current semantics and its problems

Currently the uniqueness key is generated automatically by

• Queue name
• Task Type
• Payload

But some users (see #407) would like to specify their own unique key.

Currently the uniqueness lock is released when the task gets successfully processed, but some users (see #369) would like for the lock to stay until the TTL expires.

Suggested Ideas

The Unique option should take these as parameters:

• TTL (required)
• Unique Key (optional)
• Whether to clear the uniqueness lock on task completion (optional)

Since we are still pre v1, we are okay with making breaking changes as long as we document the change in the release notes and changelog.

Here's some of the suggested APIs for the new Unique option that I have so far.

// Idea #1: Introduce UniqueParams struct and take that as parameter of Unique function
asynq.Unique(asynq.UniqueParams{
    TTL: 5 * time.Hour,
    Key: "somekey",
    ClearOnSuccess: true,
})

// Idea #2: struct Unique implements Option interface
asynq.Unique{
    TTL: 5 * time.Hour,
    Key: "somekey",
    ClearOnSuccess: true,
}

// Idea #3: Chained builder pattern
asynq.Unique(5 * time.Hour).WithKey("somekey").ClearOnSuccess()

// Idea #4: With optional varargs pattern with a custom type
asynq.Unique(5 * time.Hour) // with default key and behavior on success
asynq.Unique(5 * time.Hour, asynqUniqueKey("somekey")) // with custom key and default behavior
asynq.Unique(5 * time.Hour, asynq.UniqueKey("somekey"), asynq.ClearOnSuccess()) // with custom key and behavior


// Idea #5: with optional varargs pattern with empty interface
asynq.Unique(5 * time.Hour) // with default key and behavior on success
asynq.Unique(5 * time.Hour, "somekey") // with custom key and default behavior
asynq.Unique(5 * time.Hour, "somekey",  false)) // with custom key and behavior

If anyone has suggestions or feedback on the API, please let me know :)

[gmhafiz]

I think not idea 3 and 4 for the reason being that I like being explicit with the required parameters to Unique(). Both builder pattern and optional args suffer from discoverability too.

For idea 2, I am unsure if we want to switch to a struct instead of a method.

Idea 1 is the best for me. Slightly more verbose but is clear about the params that can be accepted. One issue is we do not know which ones are required and optional from the UniqueParams{} struct so perhaps a check is needed in the Unique() method as well as documentation on UniqueParams{}.

[thejosk]

I agree, idea 1 looks neat and also is clear regarding the params.

[treuherz]

I prefer number 1 of these options. 2 makes it quite different from other options, 4 and 5 both feel a bit weird to me. 3 I think would be fine but is definitely less simple.

Would you be able to clarify the intended behaviour of Key being unset but ClearOnSucess being true? Would that be the same as however it checks for uniqueness currently?

If having context helps, the use case I'd find this useful for is that our scheduling process deliberately duplicates enqeueuing processes in order to ensure continuity if it goes down, or to ensure previous inputs are picked up after a change. This functionality would be really helpful for removing duplicate jobs.