Controlling disk size for spot controller

[xelibrion]

I'm trying to kick off model training on Azure, and running into issue where skypilot is unable to launch a controller node.
I've added adisk_size parameter, but it does not seem to be helping, as it looks like it only controls disk size for a worker node.
For spot controller, skypilot has chosen Standard_D8s_v5 instance type, and it seems the default 50GB disk is not satisfying some Azure constraints.

config:

resources:
  cloud: azure
  accelerators: A100:1  # 1x NVIDIA A100 GPU
  disk_size: 150

Exception Details:	(OperationNotAllowed) The specified disk size 50 GB is smaller than the size of the corresponding disk in the VM image: 150 GB. This is not allowed. Please choose equal or greater size or do not specify an explicit size.
	Code: OperationNotAllowed
	Message: The specified disk size 50 GB is smaller than the size of the corresponding disk in the VM image: 150 GB. This is not allowed. Please choose equal or greater size or do not specify an explicit size.
	Target: osDisk.diskSizeGB

Could anyone suggest a workaround here?

[concretevitamin]

Hey @xelibrion - you could modify it in this controller template: https://github.com/skypilot-org/skypilot/blob/master/sky/templates/spot-controller.yaml.j2#L5

For it to take effect, please install from source.

TODO for us: add this to docs & make it easier.

[concretevitamin]

This is now documented in https://skypilot.readthedocs.io/en/latest/examples/spot-jobs.html#customizing-spot-controller-resources. Please feel free to reopen this if that doc page isn't clear for this use case.