Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Segmentation fault (core dumped)" while trying to run Syntaxnet on GPU #1440

Closed
sgrvinod opened this issue May 4, 2017 · 7 comments
Closed

"Segmentation fault (core dumped)" while trying to run Syntaxnet on GPU #1440

sgrvinod opened this issue May 4, 2017 · 7 comments
Assignees
@markomernick

Comments

@sgrvinod
Copy link

sgrvinod commented May 4, 2017 •
edited
Loading

Hi, I'm new to tensorflow, and I've been trying to get syntaxnet to work on GPU.

I've tried the suggestions in #762 and #248, but these 6 tests still fail during compilation:

INFO: Elapsed time: 1005.136s, Critical Path: 720.82s
//syntaxnet:arc_standard_transitions_test                                PASSED in 0.0s
//syntaxnet:binary_segment_state_test                                    PASSED in 0.0s
//syntaxnet:binary_segment_transitions_test                              PASSED in 0.1s
//syntaxnet:char_ngram_string_extractor_test                             PASSED in 0.0s
//syntaxnet:char_properties_test                                         PASSED in 0.1s
//syntaxnet:char_shift_transitions_test                                  PASSED in 0.1s
//syntaxnet:head_transitions_test                                        PASSED in 0.1s
//syntaxnet:label_transitions_test                                       PASSED in 0.1s
//syntaxnet:morphology_label_set_test                                    PASSED in 0.0s
//syntaxnet:once_transitions_test                                        PASSED in 0.1s
//syntaxnet:parser_features_test                                         PASSED in 0.0s
//syntaxnet:segmenter_utils_test                                         PASSED in 0.0s
//syntaxnet:sentence_features_test                                       PASSED in 0.1s
//syntaxnet:shared_store_test                                            PASSED in 0.1s
//syntaxnet:tagger_transitions_test                                      PASSED in 0.0s
//syntaxnet/util:check_test                                              PASSED in 1.9s
//syntaxnet/util:registry_test                                           PASSED in 1.7s
//syntaxnet:whole_sentence_features_test                                 PASSED in 0.0s
//util/utf8:unicodetext_unittest                                         PASSED in 0.0s
//syntaxnet:beam_reader_ops_test                                         FAILED in 1 out of 2 in 19.3s
  /home/sgrvinod/.cache/bazel/_bazel_root/6a2ae9a751b7d16f58fb35c0b19b00c3/execroot/syntaxnet/bazel-out/local_linux-opt/testlogs/syntaxnet/beam_reader_ops_test/test.log
//syntaxnet:graph_builder_test                                           FAILED in 1 out of 2 in 57.3s
  /home/sgrvinod/.cache/bazel/_bazel_root/6a2ae9a751b7d16f58fb35c0b19b00c3/execroot/syntaxnet/bazel-out/local_linux-opt/testlogs/syntaxnet/graph_builder_test/test.log
//syntaxnet:lexicon_builder_test                                         FAILED in 1 out of 2 in 8.2s
  /home/sgrvinod/.cache/bazel/_bazel_root/6a2ae9a751b7d16f58fb35c0b19b00c3/execroot/syntaxnet/bazel-out/local_linux-opt/testlogs/syntaxnet/lexicon_builder_test/test.log
//syntaxnet:parser_trainer_test                                          FAILED in 1 out of 2 in 54.4s
  /home/sgrvinod/.cache/bazel/_bazel_root/6a2ae9a751b7d16f58fb35c0b19b00c3/execroot/syntaxnet/bazel-out/local_linux-opt/testlogs/syntaxnet/parser_trainer_test/test.log
//syntaxnet:reader_ops_test                                              FAILED in 1 out of 2 in 9.8s
  /home/sgrvinod/.cache/bazel/_bazel_root/6a2ae9a751b7d16f58fb35c0b19b00c3/execroot/syntaxnet/bazel-out/local_linux-opt/testlogs/syntaxnet/reader_ops_test/test.log
//syntaxnet:text_formats_test                                            FAILED in 1 out of 2 in 8.2s
  /home/sgrvinod/.cache/bazel/_bazel_root/6a2ae9a751b7d16f58fb35c0b19b00c3/execroot/syntaxnet/bazel-out/local_linux-opt/testlogs/syntaxnet/text_formats_test/test.log

Executed 25 out of 25 tests: 19 tests pass and 6 fail locally.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

The log files don't seem to have any errors, except for Segmentation fault (core dumped) at the end of each file. An example:

----------------------------------------------------------------------
Ran 6 tests in 17.048s

OK (skipped=1) # this line can vary based on the number "skipped"
external/bazel_tools/tools/test/test-setup.sh: line 159:  1161 Segmentation fault      (core dumped) "${TEST_PATH}" "$@"

(I can paste any of the complete log files here, if it helps.)

If I try to run echo 'Bob brought the pizza to Alice.' | syntaxnet/demo.sh, it does parse it, but I also get the same error:

Input: Bob brought the pizza to Alice .
Parse:
brought VBD ROOT
 +-- Bob NNP nsubj
 +-- pizza NN dobj
 |   +-- the DT det
 +-- to IN prep
 |   +-- Alice NNP pobj
 +-- . . punct
syntaxnet/demo.sh: line 56:  4346 Segmentation fault      (core dumped) $PARSER_EVAL --input=$INPUT_FORMAT --output=stdout-conll --hidden_layer_sizes=64 --arg_prefix=brain_tagger --graph_builder=structured --task_context=$MODEL_DIR/context.pbtxt --model_path=$MODEL_DIR/tagger-params --slim_model --batch_size=1024 --alsologtostderr
      4347                       (core dumped) | $PARSER_EVAL --input=stdin-conll --output=stdout-conll --hidden_layer_sizes=512,512 --arg_prefix=brain_parser --graph_builder=structured --task_context=$MODEL_DIR/context.pbtxt --model_path=$MODEL_DIR/parser-params --slim_model --batch_size=1024 --alsologtostderr
      4348                       (core dumped) | bazel-bin/syntaxnet/conll2tree --task_context=$MODEL_DIR/context.pbtxt --alsologtostderr

System details:

Ubuntu 14.04
tensorflow/tensorflow-gpu 1.1.0
bazel 0.4.5
protobuf 3.3.0
gcc 4.9.4
cuda 8.0
cudnn 5

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26                 Driver Version: 375.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    Off  | 0000:02:00.0     Off |                  N/A |
| 23%   39C    P8     9W / 250W |      1MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN X (Pascal)    Off  | 0000:03:00.0     Off |                  N/A |
| 23%   37C    P8    10W / 250W |      1MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN X (Pascal)    Off  | 0000:04:00.0     Off |                  N/A |
| 23%   37C    P8     8W / 250W |      1MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  TITAN X (Pascal)    Off  | 0000:05:00.0      On |                  N/A |
| 26%   45C    P8    15W / 250W |    380MiB / 12188MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    3      1204    G   /usr/lib/xorg/Xorg                             199MiB |
|    3      2753    G   compiz                                         115MiB |
|    3      2923    G   ...el-token=F3B47FAA7A984F7C6D4069D4A1AD5C54    62MiB |
+-----------------------------------------------------------------------------+

Everything works fine with just the CPU.

Any help would be appreciated, thanks.
@markomernick
Copy link
Member

Attaching one of the log files would help us a great deal. Thanks!

@markomernick markomernick self-assigned this May 4, 2017
@sgrvinod
Copy link
Author

sgrvinod commented May 4, 2017

For the beam_reader_ops_test:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
2017-05-04 02:03:21.238384: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-05-04 02:03:21.238401: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-05-04 02:03:21.238404: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-05-04 02:03:21.238406: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-05-04 02:03:21.238408: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-05-04 02:03:22.187218: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: 
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:02:00.0
Total memory: 11.90GiB
Free memory: 11.30GiB
2017-05-04 02:03:22.187290: W external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:485] creating context when one is currently active; existing: 0x299ee70
2017-05-04 02:03:23.876106: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 1 with properties: 
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:03:00.0
Total memory: 11.90GiB
Free memory: 11.30GiB
2017-05-04 02:03:23.876240: W external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:485] creating context when one is currently active; existing: 0x29a27f0
2017-05-04 02:03:25.694646: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 2 with properties: 
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:04:00.0
Total memory: 11.90GiB
Free memory: 11.30GiB
2017-05-04 02:03:25.694775: W external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:485] creating context when one is currently active; existing: 0x29a6650
2017-05-04 02:03:27.100399: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 3 with properties: 
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:05:00.0
Total memory: 11.90GiB
Free memory: 10.91GiB
2017-05-04 02:03:27.154755: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 1 2 3 
2017-05-04 02:03:27.154770: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y Y Y Y 
2017-05-04 02:03:27.154773: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 1:   Y Y Y Y 
2017-05-04 02:03:27.154775: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 2:   Y Y Y Y 
2017-05-04 02:03:27.154777: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 3:   Y Y Y Y 
2017-05-04 02:03:27.154785: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:27.154789: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:27.154792: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:27.154794: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:27.213910: I syntaxnet/lexicon_builder.cc:162] Term maps collected over 964 tokens from 54 documents
2017-05-04 02:03:27.215281: I syntaxnet/term_frequency_map.cc:137] Saved 547 terms to /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:27.215635: I syntaxnet/term_frequency_map.cc:137] Saved 529 terms to /tmp/beam_reader_ops_test/lcword-map.
2017-05-04 02:03:27.215703: I syntaxnet/term_frequency_map.cc:137] Saved 36 terms to /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:27.215753: I syntaxnet/term_frequency_map.cc:137] Saved 36 terms to /tmp/beam_reader_ops_test/category-map.
2017-05-04 02:03:27.215799: I syntaxnet/term_frequency_map.cc:137] Saved 39 terms to /tmp/beam_reader_ops_test/label-map.
2017-05-04 02:03:27.215856: I syntaxnet/term_frequency_map.cc:137] Saved 57 terms to /tmp/beam_reader_ops_test/char-map.
2017-05-04 02:03:27.216647: I syntaxnet/term_frequency_map.cc:137] Saved 1680 terms to /tmp/beam_reader_ops_test/char-ngram-map.
2017-05-04 02:03:27.222445: I syntaxnet/term_frequency_map.cc:101] Loaded 39 terms from /tmp/beam_reader_ops_test/label-map.
2017-05-04 02:03:27.222621: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:27.222651: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:27.222655: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:27.232534: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:27.234030: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:27.234760: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:27.234780: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:27.234783: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:27.234785: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:27.244110: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:27.244126: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:27.244129: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:27.244131: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:27.252953: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:27.252970: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:27.252972: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:27.264174: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:27.265708: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
/home/sgrvinod/.cache/bazel/_bazel_root/6a2ae9a751b7d16f58fb35c0b19b00c3/execroot/syntaxnet/bazel-out/local_linux-opt/bin/syntaxnet/beam_reader_ops_test.runfiles/org_tensorflow/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2017-05-04 02:03:29.969558: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:29.969573: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:29.969576: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:29.975825: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:29.976317: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
.2017-05-04 02:03:30.009586: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:30.009600: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:30.009603: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:30.009605: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:30.030866: I syntaxnet/lexicon_builder.cc:162] Term maps collected over 964 tokens from 54 documents
2017-05-04 02:03:30.031103: I syntaxnet/term_frequency_map.cc:137] Saved 547 terms to /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:30.031309: I syntaxnet/term_frequency_map.cc:137] Saved 529 terms to /tmp/beam_reader_ops_test/lcword-map.
2017-05-04 02:03:30.031356: I syntaxnet/term_frequency_map.cc:137] Saved 36 terms to /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:30.031391: I syntaxnet/term_frequency_map.cc:137] Saved 36 terms to /tmp/beam_reader_ops_test/category-map.
2017-05-04 02:03:30.031430: I syntaxnet/term_frequency_map.cc:137] Saved 39 terms to /tmp/beam_reader_ops_test/label-map.
2017-05-04 02:03:30.031469: I syntaxnet/term_frequency_map.cc:137] Saved 57 terms to /tmp/beam_reader_ops_test/char-map.
2017-05-04 02:03:30.031946: I syntaxnet/term_frequency_map.cc:137] Saved 1680 terms to /tmp/beam_reader_ops_test/char-ngram-map.
2017-05-04 02:03:30.034164: I syntaxnet/term_frequency_map.cc:101] Loaded 39 terms from /tmp/beam_reader_ops_test/label-map.
2017-05-04 02:03:30.034288: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:30.034297: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:30.034300: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:30.040673: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:30.041172: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:30.041722: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:30.041748: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:30.041758: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:30.041767: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:30.045597: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:30.045623: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:30.045633: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:30.045649: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:30.051496: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:30.051506: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:30.051509: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:30.058104: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:30.058597: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:32.358517: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:32.358534: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:32.358537: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:32.364742: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:32.365237: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
.2017-05-04 02:03:32.585272: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:32.585299: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:32.585306: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:32.585312: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:32.634638: I syntaxnet/lexicon_builder.cc:162] Term maps collected over 964 tokens from 54 documents
2017-05-04 02:03:32.635140: I syntaxnet/term_frequency_map.cc:137] Saved 547 terms to /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:32.635556: I syntaxnet/term_frequency_map.cc:137] Saved 529 terms to /tmp/beam_reader_ops_test/lcword-map.
2017-05-04 02:03:32.635646: I syntaxnet/term_frequency_map.cc:137] Saved 36 terms to /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:32.635718: I syntaxnet/term_frequency_map.cc:137] Saved 36 terms to /tmp/beam_reader_ops_test/category-map.
2017-05-04 02:03:32.635785: I syntaxnet/term_frequency_map.cc:137] Saved 39 terms to /tmp/beam_reader_ops_test/label-map.
2017-05-04 02:03:32.635856: I syntaxnet/term_frequency_map.cc:137] Saved 57 terms to /tmp/beam_reader_ops_test/char-map.
2017-05-04 02:03:32.636813: I syntaxnet/term_frequency_map.cc:137] Saved 1680 terms to /tmp/beam_reader_ops_test/char-ngram-map.
2017-05-04 02:03:32.641172: I syntaxnet/term_frequency_map.cc:101] Loaded 39 terms from /tmp/beam_reader_ops_test/label-map.
2017-05-04 02:03:32.641588: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:32.641607: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:32.641611: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:32.655724: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:32.657076: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:32.657684: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:32.657699: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:32.657703: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:32.657706: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:32.660477: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:32.660492: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:32.660496: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:32.660500: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:32.664503: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:32.664523: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:32.664527: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:32.674717: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:32.675302: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:33.626452: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:33.626472: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:33.626475: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:33.635570: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:33.636299: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:33.730439: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:33.730470: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:33.730477: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:33.730483: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:33.734865: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:33.734890: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:33.734896: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:33.734901: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:33.741132: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:33.741160: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:33.741166: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:33.756657: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:33.757757: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:34.702334: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:34.702352: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:34.702356: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:34.711395: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:34.712103: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:34.805902: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:34.805928: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:34.805933: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:34.805937: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:34.809062: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:34.809079: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:34.809084: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:34.809087: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:34.813751: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:34.813767: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:34.813771: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:34.825774: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:34.826631: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:35.530059: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:35.530082: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:35.530087: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:35.542615: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:35.543581: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:35.614455: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:35.614483: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:35.614489: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:35.614494: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:35.618110: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:35.618131: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:35.618137: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:35.618143: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:35.623880: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:35.623899: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:35.623903: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:35.633162: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:35.633877: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:36.457880: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:36.457905: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:36.457911: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:36.472388: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:36.473482: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
.2017-05-04 02:03:36.557737: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:36.557757: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:36.557762: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:36.557766: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:36.593237: I syntaxnet/lexicon_builder.cc:162] Term maps collected over 964 tokens from 54 documents
2017-05-04 02:03:36.593625: I syntaxnet/term_frequency_map.cc:137] Saved 547 terms to /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:36.593935: I syntaxnet/term_frequency_map.cc:137] Saved 529 terms to /tmp/beam_reader_ops_test/lcword-map.
2017-05-04 02:03:36.594004: I syntaxnet/term_frequency_map.cc:137] Saved 36 terms to /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:36.594064: I syntaxnet/term_frequency_map.cc:137] Saved 36 terms to /tmp/beam_reader_ops_test/category-map.
2017-05-04 02:03:36.594122: I syntaxnet/term_frequency_map.cc:137] Saved 39 terms to /tmp/beam_reader_ops_test/label-map.
2017-05-04 02:03:36.594178: I syntaxnet/term_frequency_map.cc:137] Saved 57 terms to /tmp/beam_reader_ops_test/char-map.
2017-05-04 02:03:36.594907: I syntaxnet/term_frequency_map.cc:137] Saved 1680 terms to /tmp/beam_reader_ops_test/char-ngram-map.
2017-05-04 02:03:36.598411: I syntaxnet/term_frequency_map.cc:101] Loaded 39 terms from /tmp/beam_reader_ops_test/label-map.
2017-05-04 02:03:36.598574: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:36.598584: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:36.598587: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:36.607941: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:36.608675: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:36.609162: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:36.609176: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:36.609180: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:36.609183: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:36.611715: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:36.611732: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:36.611738: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:36.611741: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:36.615299: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:36.615310: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:36.615314: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:36.624263: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:36.624946: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:37.430229: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:37.430243: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:37.430246: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:37.436364: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:37.436860: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
.2017-05-04 02:03:37.461051: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:37.461076: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:37.461083: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:37.461115: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:37.503880: I syntaxnet/lexicon_builder.cc:162] Term maps collected over 964 tokens from 54 documents
2017-05-04 02:03:37.504334: I syntaxnet/term_frequency_map.cc:137] Saved 547 terms to /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:37.504699: I syntaxnet/term_frequency_map.cc:137] Saved 529 terms to /tmp/beam_reader_ops_test/lcword-map.
2017-05-04 02:03:37.504784: I syntaxnet/term_frequency_map.cc:137] Saved 36 terms to /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:37.504853: I syntaxnet/term_frequency_map.cc:137] Saved 36 terms to /tmp/beam_reader_ops_test/category-map.
2017-05-04 02:03:37.504924: I syntaxnet/term_frequency_map.cc:137] Saved 39 terms to /tmp/beam_reader_ops_test/label-map.
2017-05-04 02:03:37.504998: I syntaxnet/term_frequency_map.cc:137] Saved 57 terms to /tmp/beam_reader_ops_test/char-map.
2017-05-04 02:03:37.505833: I syntaxnet/term_frequency_map.cc:137] Saved 1680 terms to /tmp/beam_reader_ops_test/char-ngram-map.
2017-05-04 02:03:37.510682: I syntaxnet/term_frequency_map.cc:101] Loaded 39 terms from /tmp/beam_reader_ops_test/label-map.
2017-05-04 02:03:37.510887: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:37.510918: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:37.510928: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:37.526684: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:37.527833: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:37.528492: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:37.528511: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:37.528517: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:37.528521: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:37.532116: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-05-04 02:03:37.532136: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
2017-05-04 02:03:37.532141: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:04:00.0)
2017-05-04 02:03:37.532145: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:3) -> (device: 3, name: TITAN X (Pascal), pci bus id: 0000:05:00.0)
2017-05-04 02:03:37.537169: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:37.537203: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:37.537211: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:37.552472: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:37.553593: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
2017-05-04 02:03:38.254027: I syntaxnet/embedding_feature_extractor.cc:35] Features: input.token.word input(1).token.word input(2).token.word stack.token.word stack(1).token.word stack(2).token.word;input.tag input(1).tag input(2).tag stack.tag stack(1).tag stack(2).tag;stack.child(1).label stack.child(1).sibling(-1).label stack.child(-1).label stack.child(-1).sibling(1).label
2017-05-04 02:03:38.254043: I syntaxnet/embedding_feature_extractor.cc:36] Embedding names: words;tags;labels
2017-05-04 02:03:38.254046: I syntaxnet/embedding_feature_extractor.cc:37] Embedding dims: 8;8;8
2017-05-04 02:03:38.260145: I syntaxnet/term_frequency_map.cc:101] Loaded 547 terms from /tmp/beam_reader_ops_test/word-map.
2017-05-04 02:03:38.260631: I syntaxnet/term_frequency_map.cc:101] Loaded 36 terms from /tmp/beam_reader_ops_test/tag-map.
.s
----------------------------------------------------------------------
Ran 6 tests in 17.048s

OK (skipped=1)
external/bazel_tools/tools/test/test-setup.sh: line 159:  1161 Segmentation fault      (core dumped) "${TEST_PATH}" "$@"

@thomas-imt
Copy link

I have a very similar (the same?) error:

external/bazel_tools/tools/test/test-setup.sh: line 105: 19914 Segmentation fault "${EXE}" "$@"

System:

  • cuda 8.0
  • cudcnn 6
  • bazel 0.4.2
  • GPU: 2x Titan X (Pascal)

Failing cases (all with a segfault in test-setup.sh)

  • //util/utf8:unicodetext_unittest
  • //dragnn/python:bulk_component_test
  • //dragnn/python:digraph_ops_test
  • //dragnn/python:graph_builder_test
  • //dragnn/python:lexicon_test
  • //dragnn/python:network_units_test
  • //dragnn/python:sentence_io_test
  • //dragnn/python:spec_builder_test
  • //dragnn/tools:model_trainer_test
  • //examples/dragnn:test_run_all_tutorials
  • //syntaxnet:beam_reader_ops_test
  • //syntaxnet:graph_builder_test
  • //syntaxnet:lexicon_builder_test
  • //syntaxnet:parser_trainer_test
  • //syntaxnet:reader_ops_test
  • //syntaxnet:text_formats_test

@PauloCirino
Copy link

Don't know if this is it but I think tensorflow uses cudcnn 5.1

@lifeiteng
Copy link

try:
sudo nvidia-smi -c 0
bazel test --jobs 1 ..... (--jobs 1 is important, parallel tests will exhaust GPU memory)

@tensorflowbutler
Copy link
Member

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.

@amahendrakar
Copy link

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@markomernick markomernick

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@lifeiteng @thomas-imt @PauloCirino @sgrvinod @markomernick @tensorflowbutler @amahendrakar