TAS: reduce friction by defaulting the PodSet annotations

[mimowo]

What would you like to be added:

Allow scheduling workloads without TAS annotations (kueue.x-k8s.io/podset-required-topology or kueue.x-k8s.io/podset-preferred-topology) on every PodSet.

The idea is to default the annotations, but we need to decide where is the configuration.

Why is this needed:

Currently, users of TAS need to set annotations on every PodTemplate. While this gives control, it also creates friction and room for error, because users may forget to set the annotation. Also, most users want to use the "preferred" mode, and they don't need to control the topology level.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc (KEP update)
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[mimowo]

cc @mwielgus @mwysokin @tenzen-y

[mwysokin]

Maybe ResourceFlavor? Since it's usually related to the machine type and it can benefit only some of the machine types? Maybe the default could be kubernetes.io/hostname since it seems to give the best compact placement?

[tenzen-y]

Allow scheduling workloads without TAS annotations (kueue.x-k8s.io/podset-required-topology or kueue.x-k8s.io/podset-preferred-topology) on every PodSet.

This sounds like making TAS a default scheduling policy.
IIUC, TAS currently tries to pack Pods into the same and high-used nodes as much as possible (mostAllocated).
However, the policy sometimes does not meet inference and other workloads.

What about adding validations? I mean, if Jobs have TAS annotations in .metadata.annotations, we reject the Job and return appropriate places where they should put TAS annotations.