GPU resources not released after killing actor

[ericl]

The following crashes with

Exception: Could not find a node with enough GPUs or other resources to create this actor. The local scheduler information is [ {'ClientType': 'local_scheduler', 'Deleted': False, 'DBClientID': '31dc437d6df69857fea7a9eb6f04004421039e18', 'AuxAddress': '127.0.0.1:37853', 'NumCPUs': 32.0, 'NumGPUs': 1.0, 'LocalSchedulerSocketName': '/tmp/scheduler9534802'}].

import ray
import sys
import time

@ray.remote(num_gpus=1)
class Actor(object):
  def __init__(self):
    pass

ray.init(num_gpus=1)
a = Actor.remote()
a.__ray_terminate__.remote(a._ray_actor_id.id())

time.sleep(5)
a = Actor.remote()  # crashes with not enough gpus

cc @stephanie-wang @robertnishihara

[robertnishihara]

That's very surprising. I can reproduce it. However, this kind of thing should be covered by our tests.. I'll look into it.

[haoyangz]

Is this fixed? I still get this error when running the above.

ray version: 0.3.1

[haoyangz]

Update: it will only work for me with time.sleep(20). It seems it takes about 15 seconds to disconnect. Is this expected?

[robertnishihara]

This should be fixed (and really shouldn't take 15 seconds).

There are some bigger changes in the pipeline to handle actor creation more like the way we handle regular task placement, and when that happens the problem should definitely go away.

[haoyangz]

I had this error when I was tuning neural network hyperparameters using Ray Tune with Hyperband. I later switched to an older machine with only two GPUs, and it didn't give me error when running the above snippets (disconnected after a few seconds). But I still got error when running Ray Tune. So in the end, only part of the intended hyper-param trials are actually tested.

Any suggestions to debug this?

Will these changes be included in the next version?

[richardliaw]

@haoyangz can you post the error and the minimal script you used?

[haoyangz]

@richardliaw Thanks. The following simple script produces the error for me:

import torch
import random
from os.path import join

from torch.autograd import Variable
import ray
from ray.tune import Trainable, TrainingResult, register_trainable, \
    run_experiments
from ray.tune.hyperband import HyperBandScheduler

class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        h_relu = self.linear1(x).clamp(min=0)
        y_pred = self.linear2(h_relu)
        return y_pred


N, D_in, H, D_out = 64, 1000, 100, 10

class MyTrainableClass(Trainable):
    def _setup(self):
        self.x = Variable(torch.randn(N, D_in).cuda())
        self.y = Variable(torch.randn(N, D_out).cuda(), requires_grad=False)
        self.model = TwoLayerNet(D_in, self.config['H'], D_out).cuda()
        self.criterion = torch.nn.MSELoss(size_average=False)
        self.optimizer = torch.optim.SGD(self.model.parameters(), lr=1e-4)

    def _train(self):
        y_pred = self.model(self.x)
        loss = self.criterion(y_pred, self.y)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return TrainingResult(mean_loss=loss.data[0], timesteps_this_iter=1)

    def _save(self, checkpoint_dir):
        path = join(checkpoint_dir, "checkpoint.pt")
        torch.save(self.model.state_dict(), path)
        return path

    def _restore(self, checkpoint_path):
        self.model = TwoLayerNet(D_in, self.config['H'], D_out).cuda()
        self.model.load_state_dict(torch.load(checkpoint_path))


register_trainable("my_class", MyTrainableClass)
ray.init()
hyperband = HyperBandScheduler(
    time_attr="timesteps_total", reward_attr="episode_reward_mean",
    max_t=100)

run_experiments({
    "hyperband_test": {
        "run": "my_class",
        "stop": {"training_iteration": 99999},
        "repeat": 20,
        "resources": {"cpu": 0, "gpu": 1},
        "config": {
            "H": lambda spec: 10 + int(90 * random.random()),
        },
    }
}, scheduler=hyperband)

[haoyangz]

@richardliaw I checked that with the default FIFO scheduler, there is no error.

[haoyangz]

Well with enough trials, it seems FIFO will also have the error but much less frequently.

Another similar error observed under FIFO (maybe also under hyperband, not sure exactly):

Traceback (most recent call last):
  File "/cluster/zeng/code/research/software/miniconda/lib/python2.7/site-packages/ray-0.3.1-py2.7-linux-x86_64.egg/ray/worker.py", line 762, in _process_task
    *arguments)
  File "/cluster/zeng/code/research/software/miniconda/lib/python2.7/site-packages/ray-0.3.1-py2.7-linux-x86_64.egg/ray/actor.py", line 209, in actor_method_executor
    method_returns = method(actor, *args)
  File "/cluster/zeng/code/research/software/miniconda/lib/python2.7/site-packages/ray-0.3.1-py2.7-linux-x86_64.egg/ray/tune/trainable.py", line 89, in __init__
    self._setup()
  File "/cluster/zeng/code/research/ray_tune_wrapper/mymodel.py", line 51, in _setup
    self.net = Net(self.config).cuda()
  File "/cluster/zeng/code/research/software/miniconda/lib/python2.7/site-packages/torch/nn/modules/module.py", line 216, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/cluster/zeng/code/research/software/miniconda/lib/python2.7/site-packages/torch/nn/modules/module.py", line 146, in _apply
    module._apply(fn)
  File "/cluster/zeng/code/research/software/miniconda/lib/python2.7/site-packages/torch/nn/modules/module.py", line 152, in _apply
    param.data = fn(param.data)
  File "/cluster/zeng/code/research/software/miniconda/lib/python2.7/site-packages/torch/nn/modules/module.py", line 216, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/cluster/zeng/code/research/software/miniconda/lib/python2.7/site-packages/torch/_utils.py", line 69, in _cuda
    return new_type(self.size()).copy_(self, async)
  File "/cluster/zeng/code/research/software/miniconda/lib/python2.7/site-packages/torch/cuda/__init__.py", line 358, in _lazy_new
    _lazy_init()
  File "/cluster/zeng/code/research/software/miniconda/lib/python2.7/site-packages/torch/cuda/__init__.py", line 121, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /opt/conda/conda-bld/pytorch_1512378422383/work/torch/lib/THC/THCGeneral.c:70

[richardliaw]

OK I'll try to reproduce this later today

[richardliaw]

OK unfortunately, I was unable to reproduce this error. Can you post the entire stdout for running the above script? I tried running the given script on a p3.2xlarge on EC2, using pytorch_p36 and pytorch_p27 and using a wheel built off of master.

Here are the wheels that I used:

python3.6
python2.7

Perhaps try these out?

PS, the above script has a typo - you'll need to change the HyperBand instantiation to:

hyperband = HyperBandScheduler(
    time_attr="timesteps_total", reward_attr="mean_loss",
    max_t=100)

And perhaps change the return of the training result, since hyperband expects monotonically increasing metrics:

return TrainingResult(mean_loss=-loss.data[0], timesteps_this_iter=1)  #note that it is now negative