Please Help me about this #104

ilhamprayudha · 2018-07-18T02:39:39Z

1.is there any limit on data when preprocessing ?
I use the dataset with 15438 when in preprocessing it only produces 12988

what can be changed on h.params to solve OOM on wavenet training

cobr123 · 2018-07-18T05:38:01Z

in h.params
clip_mels_length = True, #For cases of OOM (Not really recommended, working on a workaround)
max_mel_frames = 900, #Only relevant when clip_mels_length = True

ilhamprayudha · 2018-07-18T07:10:40Z

why train_step is in default, whether the result of loss is good if train step above default or under default ?

I've tried using ljspeech dataset, why only 1 sentence is generated in wavenet?

how to use the model?

Rayhane-mamah · 2018-08-04T17:24:04Z

Hello @ilhamprayudha, please refer to the README for information on how to use the model (both during training and synthesis).

like @cobr123 said, "max_mel_frames" is responsible for limiting data by length so you don't get OOM errors during training. Test out different max_mel_frames values for what suits you best. You can also change "outputs_per_step" to omit OOM errors by making it bigger (I don't recommend going bigger than 3). It is also recommended to keep batch_size=32.

Can you give clear explanation about:
"why only 1 sentence is generated in wavenet?" thanks.

ilhamprayudha · 2018-08-07T00:21:32Z

How about "We also present timings for
sample generation, demonstrating more than 1000× speed-up relative to original WaveNet." In paper paralel wavenet.

I generate take along time

In paper paralel wavenet is using raw fotmat not mulaw format, so what format is in this code?

Rayhane-mamah · 2018-08-07T00:25:42Z

@ilhamprayudha I provided an answer here.

This repo speed: 1 second of audio is generated in 62 seconds.

ilhamprayudha · 2018-08-07T01:28:59Z

@Rayhane-mamah

how about best format in your code is raw or mu-law, and What is the difference to tacotron-2

in your new repo is add preprocessing_wavenet why ? is the gta used better than synthesized tacotron-2

i try your new repo and in preprocess is
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
return f(*args, **kwds)

Rayhane-mamah · 2018-08-07T09:55:18Z

Raw wavenet gives an overall better audio quality than the mulaw-quanitize but it is slower to converge.

wavenet_preprocess was added as a way to use Wavenet in a standalone fashion in case one is not interested in the Tacotron part.

Please make sure you have the same numpy version as in the requirements file.

ilhamprayudha · 2018-08-10T06:44:30Z

my numpy is same, even though such proceedings continue

this is also what happens when executing

`Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
[[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/nlplab/PKL/Tacotron-2/tacotron/feeder.py", line 166, in _enqueue_next_train_group
self._session.run(self._enqueue_op, feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
[[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

Caused by op 'datafeeder/input_queue_enqueue', defined at:
File "train.py", line 127, in
main()
File "train.py", line 121, in main
train(args, log_dir, hparams)
File "train.py", line 51, in train
checkpoint = tacotron_train(args, log_dir, hparams)
File "/home/nlplab/PKL/Tacotron-2/tacotron/train.py", line 305, in tacotron_train
return train(log_dir, args, hparams)
File "/home/nlplab/PKL/Tacotron-2/tacotron/train.py", line 123, in train
feeder = Feeder(coord, input_path, hparams)
File "/home/nlplab/PKL/Tacotron-2/tacotron/feeder.py", line 85, in init
self._enqueue_op = queue.enqueue(self._placeholders)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 346, in enqueue
self._queue_ref, vals, name=scope)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3977, in queue_enqueue_v2
timeout_ms=timeout_ms, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

CancelledError (see above for traceback): Enqueue operation was cancelled
[[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

Exception in thread background:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
[[Node: datafeeder/eval_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/eval_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/nlplab/PKL/Tacotron-2/tacotron/feeder.py", line 174, in _enqueue_next_test_group
self._session.run(self._eval_enqueue_op, feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
[[Node: datafeeder/eval_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/eval_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

Caused by op 'datafeeder/eval_queue_enqueue', defined at:
File "train.py", line 127, in
main()
File "train.py", line 121, in main
train(args, log_dir, hparams)
File "train.py", line 51, in train
checkpoint = tacotron_train(args, log_dir, hparams)
File "/home/nlplab/PKL/Tacotron-2/tacotron/train.py", line 305, in tacotron_train
return train(log_dir, args, hparams)
File "/home/nlplab/PKL/Tacotron-2/tacotron/train.py", line 123, in train
feeder = Feeder(coord, input_path, hparams)
File "/home/nlplab/PKL/Tacotron-2/tacotron/feeder.py", line 97, in init
self._eval_enqueue_op = eval_queue.enqueue(self._placeholders)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 346, in enqueue
self._queue_ref, vals, name=scope)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3977, in queue_enqueue_v2
timeout_ms=timeout_ms, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

CancelledError (see above for traceback): Enqueue operation was cancelled
[[Node: datafeeder/eval_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/eval_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

`
why did that happen

Rayhane-mamah · 2018-08-12T05:23:09Z

@ilhamprayudha, according to this issue, it appears that your numpy warning can be ignored safely, as for the enqueue canceling, it happens when a bug occurs during training and the feeder drops all queues. The log you provided doesn't hold the main reason of the crash though so I can't really say what's going on with your run. (useful crash log is usually the first lines of the crash log).

Try maybe running the training and redirecting all "stdout" to a log file so that we can get the entire crash log then send it to me. thanks :)

shartoo · 2018-08-27T01:03:11Z

I got exactly the same error as @ilhamprayudha described above using model Tacotron-2 with my own dataset,how to fix the probem?Here is detail log

[2018-08-27 03:22:24.026]  Exiting due to exception: assertion failed: [] [Condition x == y did not hold element-wise:] [x (model_1/inference/strided_slice_5:0) = ] [14100] [y (model_1/inference/strided_slice_6:0) = ] [12925]
	 [[Node: model_1/inference/assert_equal/Assert/Assert = Assert[T=[DT_STRING, DT_STRING, DT_STRING, DT_INT32, DT_STRING, DT_INT32], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](model_1/inference/assert_equal/Equal/_27, model_1/loss/assert_equal_2/Assert/Assert/data_0, model_1/loss/assert_equal_2/Assert/Assert/data_1, model_1/inference/assert_equal/Assert/Assert/data_2, model_1/inference/strided_slice_5/_29, model_1/inference/assert_equal/Assert/Assert/data_4, model_1/inference/strided_slice_1/_31)]]

Caused by op 'model_1/inference/assert_equal/Assert/Assert', defined at:
  File "I:/newwork/Tacotron-2-mandarin-new/train.py", line 133, in <module>
    main()
  File "I:/newwork/Tacotron-2-mandarin-new/train.py", line 127, in main
    train(args, log_dir, hparams)
  File "I:/newwork/Tacotron-2-mandarin-new/train.py", line 79, in train
    checkpoint = wavenet_train(args, log_dir, hparams, input_path)
  File "I:\newwork\Tacotron-2-mandarin-new\wavenet_vocoder\train.py", line 252, in wavenet_train
    return train(log_dir, args, hparams, input_path)
  File "I:\newwork\Tacotron-2-mandarin-new\wavenet_vocoder\train.py", line 176, in train
    model, stats = model_train_mode(args, feeder, hparams, global_step)
  File "I:\newwork\Tacotron-2-mandarin-new\wavenet_vocoder\train.py", line 124, in model_train_mode
    feeder.input_lengths, x=feeder.inputs)
  File "I:\newwork\Tacotron-2-mandarin-new\wavenet_vocoder\models\wavenet.py", line 176, in initialize
    y_hat = self.step(x, c, g, softmax=False) #softmax is automatically computed inside softmax_cross_entropy if needed
  File "I:\newwork\Tacotron-2-mandarin-new\wavenet_vocoder\models\wavenet.py", line 474, in step
    with tf.control_dependencies([tf.assert_equal(tf.shape(c)[-1], tf.shape(x)[-1])]):
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\check_ops.py", line 405, in assert_equal
    return control_flow_ops.Assert(condition, data, summarize=summarize)
  File "C:\Python36\lib\site-packages\tensorflow\python\util\tf_should_use.py", line 118, in wrapped
    return _add_should_use_warning(fn(*args, **kwargs))
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 172, in Assert
    return gen_logging_ops._assert(condition, data, summarize, name="Assert")
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\gen_logging_ops.py", line 51, in _assert
    name=name)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
    op_def=op_def)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): assertion failed: [] [Condition x == y did not hold element-wise:] [x (model_1/inference/strided_slice_5:0) = ] [14100] [y (model_1/inference/strided_slice_6:0) = ] [12925]
	 [[Node: model_1/inference/assert_equal/Assert/Assert = Assert[T=[DT_STRING, DT_STRING, DT_STRING, DT_INT32, DT_STRING, DT_INT32], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](model_1/inference/assert_equal/Equal/_27, model_1/loss/assert_equal_2/Assert/Assert/data_0, model_1/loss/assert_equal_2/Assert/Assert/data_1, model_1/inference/assert_equal/Assert/Assert/data_2, model_1/inference/strided_slice_5/_29, model_1/inference/assert_equal/Assert/Assert/data_4, model_1/inference/strided_slice_1/_31)]]

shartoo · 2018-08-27T01:43:47Z

My log in console is

Traceback (most recent call last):
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call
    return fn(*args)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
	 [[Node: datafeeder/eval_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/eval_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "I:\newwork\Tacotron-2-mandarin-new\tacotron\feeder.py", line 175, in _enqueue_next_test_group
    self._session.run(self._eval_enqueue_op, feed_dict=feed_dict)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 900, in run
    run_metadata_ptr)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run
    run_metadata)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
	 [[Node: datafeeder/eval_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/eval_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

Caused by op 'datafeeder/eval_queue_enqueue', defined at:
  File "I:/newwork/Tacotron-2-mandarin-new/train.py", line 133, in <module>
    main()
  File "I:/newwork/Tacotron-2-mandarin-new/train.py", line 127, in main
    train(args, log_dir, hparams)
  File "I:/newwork/Tacotron-2-mandarin-new/train.py", line 51, in train
    checkpoint = tacotron_train(args, log_dir, hparams)
  File "I:\newwork\Tacotron-2-mandarin-new\tacotron\train.py", line 310, in tacotron_train
    return train(log_dir, args, hparams)
  File "I:\newwork\Tacotron-2-mandarin-new\tacotron\train.py", line 128, in train
    feeder = Feeder(coord, input_path, hparams)
  File "I:\newwork\Tacotron-2-mandarin-new\tacotron\feeder.py", line 98, in __init__
    self._eval_enqueue_op = eval_queue.enqueue(self._placeholders)
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\data_flow_ops.py", line 346, in enqueue
    self._queue_ref, vals, name=scope)
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 4373, in queue_enqueue_v2
    timeout_ms=timeout_ms, name=name)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
    op_def=op_def)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

CancelledError (see above for traceback): Enqueue operation was cancelled
	 [[Node: datafeeder/eval_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/eval_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]


Exception in thread background:
Traceback (most recent call last):
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call
    return fn(*args)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
	 [[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "I:\newwork\Tacotron-2-mandarin-new\tacotron\feeder.py", line 167, in _enqueue_next_train_group
    self._session.run(self._enqueue_op, feed_dict=feed_dict)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 900, in run
    run_metadata_ptr)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run
    run_metadata)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
	 [[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

Caused by op 'datafeeder/input_queue_enqueue', defined at:
  File "I:/newwork/Tacotron-2-mandarin-new/train.py", line 133, in <module>
    main()
  File "I:/newwork/Tacotron-2-mandarin-new/train.py", line 127, in main
    train(args, log_dir, hparams)
  File "I:/newwork/Tacotron-2-mandarin-new/train.py", line 51, in train
    checkpoint = tacotron_train(args, log_dir, hparams)
  File "I:\newwork\Tacotron-2-mandarin-new\tacotron\train.py", line 310, in tacotron_train
    return train(log_dir, args, hparams)
  File "I:\newwork\Tacotron-2-mandarin-new\tacotron\train.py", line 128, in train
    feeder = Feeder(coord, input_path, hparams)
  File "I:\newwork\Tacotron-2-mandarin-new\tacotron\feeder.py", line 86, in __init__
    self._enqueue_op = queue.enqueue(self._placeholders)
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\data_flow_ops.py", line 346, in enqueue
    self._queue_ref, vals, name=scope)
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 4373, in queue_enqueue_v2
    timeout_ms=timeout_ms, name=name)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
    op_def=op_def)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

CancelledError (see above for traceback): Enqueue operation was cancelled
	 [[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

shartoo · 2018-08-27T09:56:48Z

I'm training a fresh new model without using pre-trained model.

shartoo · 2018-09-26T06:09:17Z

Found the solution here relationshio of hop size and scale prod. Thank you !

ilhamprayudha changed the title ~~datatraining reduce~~ datatraining reduce & OOM wavenet Jul 18, 2018

ilhamprayudha changed the title ~~datatraining reduce & OOM wavenet~~ Please Help me about this Jul 18, 2018

gloriouskilka mentioned this issue Oct 15, 2018

GTX1070 and 8G memory cause memory dump #239

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Please Help me about this #104

Please Help me about this #104

ilhamprayudha commented Jul 18, 2018 •

edited

Loading

cobr123 commented Jul 18, 2018

ilhamprayudha commented Jul 18, 2018

Rayhane-mamah commented Aug 4, 2018

ilhamprayudha commented Aug 7, 2018 •

edited

Loading

Rayhane-mamah commented Aug 7, 2018

ilhamprayudha commented Aug 7, 2018 •

edited

Loading

Rayhane-mamah commented Aug 7, 2018

ilhamprayudha commented Aug 10, 2018 •

edited

Loading

Rayhane-mamah commented Aug 12, 2018

shartoo commented Aug 27, 2018 •

edited

Loading

shartoo commented Aug 27, 2018

shartoo commented Aug 27, 2018

shartoo commented Sep 26, 2018

Please Help me about this #104

Please Help me about this #104

Comments

ilhamprayudha commented Jul 18, 2018 • edited Loading

cobr123 commented Jul 18, 2018

ilhamprayudha commented Jul 18, 2018

Rayhane-mamah commented Aug 4, 2018

ilhamprayudha commented Aug 7, 2018 • edited Loading

Rayhane-mamah commented Aug 7, 2018

ilhamprayudha commented Aug 7, 2018 • edited Loading

Rayhane-mamah commented Aug 7, 2018

ilhamprayudha commented Aug 10, 2018 • edited Loading

Rayhane-mamah commented Aug 12, 2018

shartoo commented Aug 27, 2018 • edited Loading

shartoo commented Aug 27, 2018

shartoo commented Aug 27, 2018

shartoo commented Sep 26, 2018

ilhamprayudha commented Jul 18, 2018 •

edited

Loading

ilhamprayudha commented Aug 7, 2018 •

edited

Loading

ilhamprayudha commented Aug 7, 2018 •

edited

Loading

ilhamprayudha commented Aug 10, 2018 •

edited

Loading

shartoo commented Aug 27, 2018 •

edited

Loading