Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CNTK Crash: No computation node mapping exists for Variable Placeholder #3858

Open
maybeLee opened this issue Sep 3, 2021 · 0 comments
Open

Comments

@maybeLee
Copy link

maybeLee commented Sep 3, 2021

When I am using keras with CNTK backend to load and predict on a LSTM model, CNTK crash with the following bug trace:

About to throw exception 'Function 'Composite(Combine): Input('lstm_1_copy_LA_copy_LA_copy_LA_copy_CP_input', [#], [49 x 1]) -> Output('Plus1481_Output_0', [#], [1])': No computation node mapping exists for Variable Placeholder('Placeholder1594', [#, toSequence_Minus537_Output_0], [100]).'
Traceback (most recent call last):
  File "exp1/job0/scripts/generation/script_prediction.py", line 121, in <module>
    _get_prediction(bk=bk, model_path=flags.model_path, batch_size=batch_size)
  File "exp1/job0/scripts/generation/script_prediction.py", line 42, in _get_prediction
    pred = model.predict(x,batch_size=batch_size)
  File "lib/python3.6/site-packages/keras/engine/training.py", line 1169, in predict
    steps=steps)
  File "lib/python3.6/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop
    batch_outs = f(ins_batch)
  File "lib/python3.6/site-packages/keras/backend/cntk_backend.py", line 2016, in __call__
    output_values = self.metrics_func.eval(input_dict, as_numpy=False)
  File "cntk/cntk/bindings/python/cntk/ops/functions.py", line 733, in eval
    _, output_map = self.forward(arguments, outputs, device=device, as_numpy=as_numpy)
  File "cntk/cntk/bindings/python/cntk/internal/swig_helper.py", line 69, in wrapper
    result = f(*args, **kwds)
  File "cntk/cntk/bindings/python/cntk/ops/functions.py", line 867, in forward
    keep_for_backward)
  File "cntk/cntk/bindings/python/cntk/cntk_py.py", line 1980, in _forward
    return _cntk_py.Function__forward(self, *args)
RuntimeError: Function 'Composite(Combine): Input('lstm_1_copy_LA_copy_LA_copy_LA_copy_CP_input', [#], [49 x 1]) -> Output('Plus1481_Output_0', [#], [1])': No computation node mapping exists for Variable Placeholder('Placeholder1594', [#, toSequence_Minus537_Output_0], [100]).

Steps to reproduce:
Please access the json configuration of the used lstm model from:
https://drive.google.com/file/d/1BwXQEpnmutqW1sQOUhTP1e6StlqMh9uU/view?usp=sharing
save the configuration file as model.json and then use the following script to reproduce bug:

import os
import argparse
import sys
import warnings
import configparser
import redis
import pickle


if __name__ == "__main__":
    parse = argparse.ArgumentParser()
    parse.add_argument("--backend", type=str, help="specify the backend")
    flags, _ = parse.parse_known_args(sys.argv[1:])

    bk = flags.backend
    os.environ["KERAS_BACKEND"] = bk
    os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
    warnings.filterwarnings("ignore", category=DeprecationWarning)
    warnings.filterwarnings("ignore", category=UserWarning)
    warnings.filterwarnings("ignore", category=FutureWarning)
    warnings.filterwarnings("ignore")

    if bk == "cntk":
        from cntk.device import try_set_default_device, gpu
        try_set_default_device(gpu(0))
    import numpy as np
    import keras
    model_path = "model.json"
    print("start loading the model")
    model = keras.models.model_from_json(open(model_path, "r").read())
    model.summary()
    input_shape = model.layers[0].input_shape
    input_shape = list(input_shape)
    input_shape[0] = 3
    input_shape = tuple(input_shape)
    x = np.random.rand(*input_shape)
    pred = model.predict(x)
    json = model.to_json()
    print(pred)
    print("successfully get the prediction result")
    del model

Note that when I use the different backend of Keras such as TensorFlow and Theano, this model can be successfully loaded and predicted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant