[autoscaler] External Node Provider example fails

[richardliaw]

Using Ray from master. Trying out the external node provider in YAML as mentioned in the docs introduced from #1703...

Describe the problem

Example yaml to try:

# An unique identifier for the head node and workers of this cluster.
cluster_name: minimal

# The maximum number of workers nodes to launch in addition to the head
# node. This takes precedence over min_workers. min_workers default to 0.
max_workers: 0

# Cloud-provider specific configuration.
provider:
    type: external
    module: mypackage.myclass

# How Ray will authenticate with newly launched nodes.
auth:
    ssh_user: ubuntu

Source code / logs

Traceback (most recent call last):
  File "/Users/rliaw/miniconda3/envs/ray/bin/ray", line 11, in <module>
    load_entry_point('ray', 'console_scripts', 'ray')()
  File "/Users/rliaw/Research/riselab/ray/python/ray/scripts/scripts.py", line 317, in main
    return cli()
  File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/rliaw/Research/riselab/ray/python/ray/scripts/scripts.py", line 291, in create_or_update
    cluster_config_file, min_workers, max_workers, no_restart, yes)
  File "/Users/rliaw/Research/riselab/ray/python/ray/autoscaler/commands.py", line 33, in create_or_update_cluster
    config = fillout_defaults(config)
  File "/Users/rliaw/Research/riselab/ray/python/ray/autoscaler/autoscaler.py", line 535, in fillout_defaults
    defaults = get_default_config(config["provider"])
  File "/Users/rliaw/Research/riselab/ray/python/ray/autoscaler/node_provider.py", line 77, in get_default_config
    "Unsupported node provider: {}".format(provider_config["type"]))
NotImplementedError: Unsupported node provider: external

[robertnishihara]

cc @ericl @barrachri

[ericl]

It looks like a config loading function also needs to be defined for the external module. Though, I'm not sure what the exact requirements for a external config would be.

Note that there seem to be two functions for initializing configs: in the AWS case there's load_aws_config() -> default_config, which is merged with the user yaml, and then further updated by bootstrap_aws(config) -> new_config to fill in user-account specific configurations.

The simplest thing to do would just be to make both of these functions no-ops for the external node provider case and just require the cluster YAML to include all the necessary required fields.

[barrachri]

@richardliaw can you check #1743?

Should fix this issue.