Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Training error in main.py #1

Closed
polyrhythmatic opened this issue Sep 13, 2016 · 29 comments
Closed

Training error in main.py #1

polyrhythmatic opened this issue Sep 13, 2016 · 29 comments

Comments

@polyrhythmatic
Copy link

polyrhythmatic commented Sep 13, 2016 •
edited
Loading

Getting the following error when I try to train the network - any idea what this is?

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:924] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 12.00GiB
Free memory: 11.53GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:806] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
  File "main.py", line 129, in <module>
    main()
  File "main.py", line 83, in main
    loss = net.loss(audio_batch)
  File "/home/seth/Development/tensorflow-wavenet/wavenet.py", line 97, in loss
    raw_output = self._create_network(encoded)
  File "/home/seth/Development/tensorflow-wavenet/wavenet.py", line 67, in _create_network
    dilation=dilation)
  File "/home/seth/Development/tensorflow-wavenet/wavenet.py", line 23, in _create_dilation_layer
    name="conv_f")
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 168, in atrous_conv2d
    in_height = int(value_shape[1])
TypeError: __int__ returned non-int (type NoneType)
@ibab
Copy link
Owner

ibab commented Sep 13, 2016

That's strange. Which version of TensorFlow are you using?
It might also be a Python 2 problem (I'm developing on Python 3).

@polyrhythmatic
Copy link
Author

I was wondering if that was the case. I just tried using python 3.5 with tensorflow 0.10.0 and got the following error:


I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 12.00GiB
Free memory: 11.51GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_shape.py", line 563, in merge_with
    self.assert_same_rank(other)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_shape.py", line 609, in assert_same_rank
    "Shapes %s and %s must have the same rank" % (self, other))
ValueError: Shapes (?, ?, ?, 256) and (?, 256) must have the same rank

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 129, in <module>
    main()
  File "main.py", line 83, in main
    loss = net.loss(audio_batch)
  File "/home/seth/Development/tensorflow-wavenet/wavenet.py", line 105, in loss
    loss = tf.nn.softmax_cross_entropy_with_logits(raw_output, tf.reshape(shifted, [-1, self.channels]))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 491, in softmax_cross_entropy_with_logits
    precise_logits, labels, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 1427, in _softmax_cross_entropy_with_logits
    features=features, labels=labels, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2319, in create_op
    set_shapes_for_outputs(ret)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1711, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 603, in _SoftmaxCrossEntropyWithLogitsShape
    input_shape = logits_shape.merge_with(labels_shape).with_rank(2)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_shape.py", line 570, in merge_with
    (self, other))
ValueError: Shapes (?, ?, ?, 256) and (?, 256) are not compatible

Im running ubuntu 16.04 if that matters.

@adamduracz
Copy link

Getting a similar error to @polyrhythmatic with Python 2.7 and TF 0.10.0:

> python main.py
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.dylib locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GT 650M
major: 3 minor: 0 memoryClockRate (GHz) 0.9
pciBusID 0000:01:00.0
Total memory: 1023.69MiB
Free memory: 53.51MiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 650M, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
  File "main.py", line 129, in <module>
    main()
  File "main.py", line 83, in main
    loss = net.loss(audio_batch)
  File "/Users/adam/work/python/tensorflow-wavenet/wavenet.py", line 105, in loss
    loss = tf.nn.softmax_cross_entropy_with_logits(raw_output, tf.reshape(shifted, [-1, self.channels]))
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 491, in softmax_cross_entropy_with_logits
    precise_logits, labels, name=name)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1427, in _softmax_cross_entropy_with_logits
    features=features, labels=labels, name=name)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2319, in create_op
    set_shapes_for_outputs(ret)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1711, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 603, in _SoftmaxCrossEntropyWithLogitsShape
    input_shape = logits_shape.merge_with(labels_shape).with_rank(2)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/tensor_shape.py", line 570, in merge_with
    (self, other))
ValueError: Shapes (?, ?, ?, 256) and (?, 256) are not compatible

@polyrhythmatic
Copy link
Author

So I don't really have any idea what I'm doing, but it seems to me that maybe the issue is with line 105 in wavenet and some sort of shape mismatch between the arguments to tf.nn.softmax_cross_entropy_with_logits

with no arguments to

python main.py

I get

    "Shapes %s and %s must have the same rank" % (self, other))
ValueError: Shapes (?, ?, ?, 256) and (?, 256) must have the same rank

but when I provide a channel argument of 3

python main.py --channels 3

One of the shapes is more defined

    "Shapes %s and %s must have the same rank" % (self, other))
ValueError: Shapes (3, ?, ?, 256) and (?, 256) must have the same rank

Not sure if this makes any difference.

@ibab
Copy link
Owner

ibab commented Sep 14, 2016

I'll try to fix this tomorrow.
It might be because I've written the network using the current git master of TensorFlow, or there might be a bug.

@lelayf
Copy link

lelayf commented Sep 14, 2016

I have the same error here TF r0.10 on Mac OS X and python 3.5.2 in virtualenv.

@finlay-liu
Copy link

I get the same error with @adamduracz...

@ibab
Copy link
Owner

ibab commented Sep 14, 2016

The shape error resulted from a small API difference in TF 0.10 and the current git TF.
The fix is in e67b137.
I'll keep developing on TF 0.10 from now :)

I'm not sure what's causing the first error in this issue, so I'm keeping it open.

@nonstop99
Copy link

Hi, I was getting the same error as above, I tried the latest files from the git but now i'm getting..

osboxes@osboxes:~/Documents/wavenet/wavenet$ sudo python main.py Traceback (most recent call last): File "main.py", line 129, in <module> main() File "main.py", line 119, in main summary, loss_value, _ = sess.run([summaries, loss, optim]) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 710, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 908, in _run feed_dict_string, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 958, in _do_run target_list, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 978, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_3_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0) [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]] Caused by op u'create_inputs/padding_fifo_queue_DequeueMany', defined at: File "main.py", line 129, in <module> main() File "main.py", line 78, in main audio_batch, _ = queue.dequeue_many(args.batch_size) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 435, in dequeue_many self._queue_ref, n=n, component_types=self._dtypes, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 867, in _queue_dequeue_many timeout_ms=timeout_ms, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2317, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1239, in __init__ self._traceback = _extract_stack()

@adamduracz
Copy link

@ibab e67b137 indeed resolved the issue I posted, but that was hiding another one:

> python main.py
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.dylib locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GT 650M
major: 3 minor: 0 memoryClockRate (GHz) 0.9
pciBusID 0000:01:00.0
Total memory: 1023.69MiB
Free memory: 402.64MiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 650M, pci bus id: 0000:01:00.0)
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
Traceback (most recent call last):
  File "main.py", line 129, in <module>
    main()
  File "main.py", line 119, in main
    summary, loss_value, _ = sess.run([summaries, loss, optim])
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 710, in run
    run_metadata_ptr)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 908, in _run
    feed_dict_string, options, run_metadata)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 958, in _do_run
    target_list, options, run_metadata)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 978, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
Caused by op u'create_inputs/padding_fifo_queue_DequeueMany', defined at:
  File "main.py", line 129, in <module>
    main()
  File "main.py", line 78, in main
    audio_batch, _ = queue.dequeue_many(args.batch_size)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 435, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 867, in _queue_dequeue_many
    timeout_ms=timeout_ms, name=name)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2317, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1239, in __init__
    self._traceback = _extract_stack()

@nonstop99
Copy link

nonstop99 commented Sep 14, 2016 •
edited
Loading

I'm using a virtulbox image so I can make that available if it helps?

@ibab
Copy link
Owner

ibab commented Sep 14, 2016

That looks like it's not finding the data directory.
I'll make sure to add a check so that there's a better error message in this case.

@nonstop99
Copy link

Ok is there anything we need to do to fix it?

btw, I tried using Python 3.5 but got the same error

@polyrhythmatic
Copy link
Author

@ibab thanks for all your work on this! I've tried specifying the data directory and still can't seem to get anywhere with it. Any recommendations?

@ibab
Copy link
Owner

ibab commented Sep 14, 2016

I've added a bit of code that makes sure that audio files have been found and prints the pattern that they are globbed with.

@nonstop99
Copy link

@ibab

here's the output...

` sudo python3.5 train.py
Found 4706 audio files
Traceback (most recent call last):
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 965, in _do_call
return fn(*args)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 947, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors.py", line 450, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 151, in
main()
File "train.py", line 136, in main
summary, loss_value, _ = sess.run([summaries, loss, optim])
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 710, in run
run_metadata_ptr)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 908, in _run
feed_dict_string, options, run_metadata)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 958, in _do_run
target_list, options, run_metadata)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 978, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
Caused by op 'create_inputs/padding_fifo_queue_DequeueMany', defined at:
File "train.py", line 151, in
main()
File "train.py", line 90, in main
audio_batch, _ = queue.dequeue_many(args.batch_size)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/ops/data_flow_ops.py", line 435, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 867, in _queue_dequeue_many
timeout_ms=timeout_ms, name=name)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
op_def=op_def)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2317, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1239, in init
self._traceback = _extract_stack()`

@finlay-liu
Copy link

Thx. I fetch the latest code any solve my problem, now i can train the model in tf.
I put the VCTK-Corpus folder in the root of the code, it runs ok. @polyrhythmatic

@polyrhythmatic
Copy link
Author

polyrhythmatic commented Sep 14, 2016 •
edited
Loading

@finlay-liu thanks - already have the files in root and still getting an error.

@ibab I'm on the latest commit and still having issues 3e4da39

running

python3 train.py

I get the following output

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 12.00GiB
Free memory: 11.52GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
Found 44257 audio files
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 965, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 947, in _run_fn
    status, run_metadata)
  File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors.py", line 450, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 151, in <module>
    main()
  File "train.py", line 136, in main
    summary, loss_value, _ = sess.run([summaries, loss, optim])
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 710, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 908, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 958, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 978, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
Caused by op 'create_inputs/padding_fifo_queue_DequeueMany', defined at:
  File "train.py", line 151, in <module>
    main()
  File "train.py", line 90, in main
    audio_batch, _ = queue.dequeue_many(args.batch_size)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 435, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 867, in _queue_dequeue_many
    timeout_ms=timeout_ms, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2317, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1239, in __init__
    self._traceback = _extract_stack()

Any idea what this could be?

@finlay-liu
Copy link

@polyrhythmatic i use py2 it's ok, and i think it maybe the the version problem.

@nonstop99
Copy link

@finlay-liu I've tried with py2.7 and py3.5 but i'm getting the same results as @polyrhythmatic

@polyrhythmatic
Copy link
Author

@finlay-liu thanks but I have tried python 2.7 as well and it does not work.

@ibab
Copy link
Owner

ibab commented Sep 14, 2016

It just occurred to me that this could be caused by ffmpeg not being installed.
Could you check whether it's installed and in your PATH?
I'll add a note to the README.md that it is required.

@spiantino
Copy link

brew install ffmpeg fixed it for me (on OS X), but now I'm running out of memory. I suppose 6 GB is not enough (I have a 980 Ti)

@ibab
Copy link
Owner

ibab commented Sep 14, 2016 •
edited
Loading

@spiantino: The dilated convolution uses an excessively large amount of memory at the moment.
It's on my list of things that need to be fixed for the implementation is "complete".

If you still want to test the network, you can reduce the number of layers in the train.py script.

@spiantino
Copy link

Had to go all the way down to dilations = [1, 2, 4] but loss seems to be decreasing. Probably not going to learn very much, but we'll see

@ibab
Copy link
Owner

ibab commented Sep 14, 2016

I've added a memory-related issue here: #4

@ibab
Copy link
Owner

ibab commented Sep 14, 2016

I think I can close this issue now, as it seems that it was caused by missing the ffmpeg library.
If you are still having problems that match the ones above, feel free to comment here and I will reopen the issue.

@ibab ibab closed this as completed Sep 14, 2016
@adamduracz
Copy link

@ibab training works for me after brew install ffmpeg. Thanks!

@polyrhythmatic
Copy link
Author

polyrhythmatic commented Sep 15, 2016 •
edited
Loading

Thanks - this worked a charm yesterday! Just had to update cudnn to 5 today and now I'm up and training (with python 2.7 as well). I had assumed that tensorflow installed ffmpeg by default. Great work :)

@jyegerlehner jyegerlehner mentioned this issue Sep 21, 2016
Closed
akademi4eg pushed a commit that referenced this issue Jan 27, 2017
@koz4k
Merge pull request #1 from ibab/koz4k-padding-valid
1044c50 
Resolving conflicts for PR 191 in ibab/tensorflow-wavenet repo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@ibab @lelayf @adamduracz @spiantino @polyrhythmatic @finlay-liu @nonstop99