[k8s] Attach custom metadata to objects created by SkyPilot #3333
Conversation
…o k8s_custom_metadata # Conflicts: # sky/provision/kubernetes/network.py
…o k8s_custom_metadata
There was a problem hiding this comment.
Thanks for the quick fix @romilbhardwaj! Left a question : )
| custom_metadata: | ||
| labels: | ||
| mylabel: myvalue | ||
| annotations: | ||
| myannotation: myvalue |
There was a problem hiding this comment.
Are both of the labels and annotations required? We are using instance_tags: for the other clouds to allow user specify custom tags for all the VMs created. Would it be possible to have a similar argument for k8s pods?
There was a problem hiding this comment.
Both are not required, they can be different or even just one can be present. Labels and annotations serve different purposes - labels are used for filtering, selecting and scheduling, while annotations are metadata that can be consumed by third party tools (e.g., prometheus.io/scrape annotations tells prometheus which objects to scrape).
Given this, users may want to set labels and/or annotations, this config gives them the flexibility to do so.
There was a problem hiding this comment.
Makes sense! Maybe in the future, we can provide also provide instance_tags as an alias for specifying both labels and annotations with the same set of key, values.
| # Add custom metadata from config | ||
| merge_custom_metadata(content['service_spec']['metadata']) |
There was a problem hiding this comment.
Q: If we have multiple users sharing the same jump pod, and they have different metadata field setup in their ~/.sky/config.yaml. What will happen?
There was a problem hiding this comment.
Great question - the metadata specified during the first creation of the ssh jump pod will be used throughout it's life. Other users sharing the jump pod will not have their metadata overwrite the existing pod and service's metadata.
| @@ -1059,7 +1068,48 @@ def get_endpoint_debug_message() -> str: | |||
| debug_cmd=debug_cmd) | |||
|
|
|||
|
|
|||
| def combine_pod_config_fields(config_yaml_path: str) -> None: | |||
| def merge_dicts(source: Dict[Any, Any], destination: Dict[Any, Any]): | |||
There was a problem hiding this comment.
make this function private? Seems only used within the same file?
There was a problem hiding this comment.
It's also used in authentication.py, where we use merge_dicts when setting up the secrets..
| custom_metadata: | ||
| labels: | ||
| mylabel: myvalue | ||
| annotations: | ||
| myannotation: myvalue |
There was a problem hiding this comment.
Makes sense! Maybe in the future, we can provide also provide instance_tags as an alias for specifying both labels and annotations with the same set of key, values.
Closes #3317. Adds the ability to attach custom metadata to all objects created by SkyPilot. Useful for accounting and management of SkyPilot specific resources.
Tested (run the relevant ones):
bash format.shkubectl describe ...with bothingressandloadbalancerport modes.pytest tests/test_smoke.py --kubernetes