Add tags to tasks, schedules and jobs

[agronholm]

Things to check first

• I have searched the existing issues and didn't find my feature already requested there

Feature description

Tasks, schedules and jobs could have tags assigned to them. Schedulers could then pick only jobs that have compatible tags.

Use case

This will greatly help in heterogeneous environments where you need certain hardware or a certain operating system to run a particular job.

[agronholm]

@NikBiks You can make your case here for a simpler solution.

[NixBiks]

Not sure I understand your thinking around using < or > in the tags. Should tags simply be unique strings?

The moment you create a job, task or a schedule, you can also add a list of tags for it. And when you start a worker you can provide a list of tags to consume from. We can jump on a call sometime if you want (I'm in Copenhagen so same time zone I think) - not today though

[agronholm]

Suppose a job runs code that requires a specific Python version, or some other metric. It's very impractical to have to enable every valid value in the scheduler where valid tag values can be a numeric range.

[agronholm]

Imagine an AI related job requiring a certain amount of VRAM. Only schedulers hosted on machines that have at least that much VRAM available should pick up those jobs.

[agronholm]

The closest implementation that I can think of is the node affinity system in Kubernetes.

[agronholm]

I guess the term "tag" would be somewhat misleading in this case then.

[NixBiks]

I think that gets complex real fast. I'd definitely advocate for a simple string - at least as the first step. The other feature could raise all kinds of trouble I think.

If not a simple string solution then you might want to rename to worker_consume_conditions or similar.

[agronholm]

Completed via f36a398.