invalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [37007] rhs shape= [12003] #15
Comments
|
@lovejasmine Have you load the correct checkpoint? Or can you try on a new folder without loading an existing checkpoint? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(venv) xxx590cc37a5:conv_seq2seq xxx.xxx$
(venv) xxx590cc37a5:conv_seq2seq xxx.xxx$ python -m bin.train --config_paths="example_configs/conv_seq2seq.yml,example_configs/train_seq2seq.yml,example_configs/text_metrics_bpe.yml" \
INFO:tensorflow:Loading config from /Users/xxx.xxx/xxx_prj/conv_seq2seq/example_configs/conv_seq2seq.yml
INFO:tensorflow:Loading config from /Users/xxx.xxx/xxx_prj/conv_seq2seq/example_configs/train_seq2seq.yml
INFO:tensorflow:Loading config from /Users/xxx.xxx/xxx_prj/conv_seq2seq/example_configs/text_metrics_bpe.yml
INFO:tensorflow:Final Config:
bucxxxts: 10,20,30,40
default_params:
hooks:
params: {every_n_steps: 1000}
metrics:
model: ConvSeq2Seq
model_params:
decoder.class: seq2seq.decoders.ConvDecoderFairseq
decoder.params: {cnn.kwidths: '3,3,3', cnn.layers: 3, cnn.nhids: '256,256,256'}
embedding.dim: 256
encoder.class: seq2seq.encoders.ConvEncoderFairseq
encoder.params: {cnn.kwidths: '3,3,3,3', cnn.layers: 4, cnn.nhids: '256,256,256,256'}
optimizer.clip_gradients: 0.1
optimizer.learning_rate: 0.25
optimizer.name: Momentum
optimizer.params: {momentum: 0.99, use_nesterov: true}
source.max_seq_len: 50
source.reverse: false
target.max_seq_len: 50
WARNING:tensorflow:Ignoring config flag: default_params
INFO:tensorflow:Setting save_checkpoints_secs to 600
INFO:tensorflow:Creating ParallelTextInputPipelineFairseq in mode=train
INFO:tensorflow:
ParallelTextInputPipelineFairseq:
!!python/unicode 'num_epochs': null
!!python/unicode 'shuffle': true
!!python/unicode 'source_delimiter': !!python/unicode ' '
!!python/unicode 'source_files': [t, r, a, i, n, ., t, o, k, ., c, l, e, a, n, .,
b, p, e, ., '3', '2', '0', '0', '0', ., e, n]
!!python/unicode 'target_delimiter': !!python/unicode ' '
!!python/unicode 'target_files': [t, r, a, i, n, ., t, o, k, ., c, l, e, a, n, .,
b, p, e, ., '3', '2', '0', '0', '0', ., d, e]
INFO:tensorflow:Creating ParallelTextInputPipelineFairseq in mode=eval
INFO:tensorflow:
ParallelTextInputPipelineFairseq:
!!python/unicode 'num_epochs': 1
!!python/unicode 'shuffle': false
!!python/unicode 'source_delimiter': !!python/unicode ' '
!!python/unicode 'source_files': [n, e, w, s, t, e, s, t, '2', '0', '1', '3', .,
t, o, k, ., b, p, e, ., '3', '2', '0', '0', '0', ., e, n]
!!python/unicode 'target_delimiter': !!python/unicode ' '
!!python/unicode 'target_files': [n, e, w, s, t, e, s, t, '2', '0', '1', '3', .,
t, o, k, ., b, p, e, ., '3', '2', '0', '0', '0', ., d, e]
INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_xxxep_checkpoint_max': 5, '_tf_random_seed': None, '_task_type': None, '_environment': 'local', '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11b951e90>, '_tf_config': gpu_options {
per_process_gpu_memory_fraction: 1.0
}
, '_task_id': 0, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_evaluation_master': '', '_xxxep_checkpoint_every_n_hours': 4, '_master': ''}
INFO:tensorflow:Creating PrintModelAnalysisHook in mode=train
INFO:tensorflow:
PrintModelAnalysisHook: {}
INFO:tensorflow:Creating MetadataCaptureHook in mode=train
INFO:tensorflow:
MetadataCaptureHook: {!!python/unicode 'step': 10}
INFO:tensorflow:Creating SyncReplicasOptimizerHook in mode=train
INFO:tensorflow:
SyncReplicasOptimizerHook: {}
INFO:tensorflow:Creating TrainSampleHook in mode=train
INFO:tensorflow:
TrainSampleHook: {!!python/unicode 'every_n_secs': null, !!python/unicode 'every_n_steps': 1000,
!!python/unicode 'source_delimiter': !!python/unicode ' ', !!python/unicode 'target_delimiter': !!python/unicode ' '}
INFO:tensorflow:Creating LogPerplexityMetricSpec in mode=eval
INFO:tensorflow:
LogPerplexityMetricSpec: {}
INFO:tensorflow:Training model for 5000 steps
INFO:tensorflow:Creating ConvSeq2Seq in mode=train
INFO:tensorflow:
ConvSeq2Seq:
!!python/unicode 'decoder.class': !!python/unicode 'seq2seq.decoders.ConvDecoderFairseq'
!!python/unicode 'decoder.params': {cnn.kwidths: '3,3,3', cnn.layers: 3, cnn.nhids: '256,256,256'}
!!python/unicode 'embedding.dim': 256
!!python/unicode 'embedding.init_scale': 0.04
!!python/unicode 'embedding.share': false
!!python/unicode 'encoder.class': !!python/unicode 'seq2seq.encoders.ConvEncoderFairseq'
!!python/unicode 'encoder.params': {cnn.kwidths: '3,3,3,3', cnn.layers: 4, cnn.nhids: '256,256,256,256'}
!!python/unicode 'inference.beam_search.beam_width': 0
!!python/unicode 'inference.beam_search.choose_successors_fn': !!python/unicode 'choose_top_k'
!!python/unicode 'inference.beam_search.length_penalty_weight': 1.0
!!python/unicode 'optimizer.clip_embed_gradients': 5
!!python/unicode 'optimizer.clip_gradients': 0.1
!!python/unicode 'optimizer.learning_rate': 0.25
!!python/unicode 'optimizer.lr_decay_rate': 0.9
!!python/unicode 'optimizer.lr_decay_steps': 5000
!!python/unicode 'optimizer.lr_decay_type': !!python/unicode 'exponential_decay'
!!python/unicode 'optimizer.lr_min_learning_rate': 1.0e-05
!!python/unicode 'optimizer.lr_staircase': true
!!python/unicode 'optimizer.lr_start_decay_at': 0
!!python/unicode 'optimizer.lr_stop_decay_at': 2147483647
!!python/unicode 'optimizer.name': !!python/unicode 'Momentum'
!!python/unicode 'optimizer.params': {!!python/unicode 'momentum': 0.99, !!python/unicode 'use_nesterov': true}
!!python/unicode 'optimizer.sync_replicas': 0
!!python/unicode 'optimizer.sync_replicas_to_aggregate': 0
!!python/unicode 'position_embeddings.num_positions': 100
!!python/unicode 'source.max_seq_len': 50
!!python/unicode 'source.reverse': false
!!python/unicode 'target.max_seq_len': 50
!!python/unicode 'vocab_source': !!python/unicode 'vocab.bpe.32000'
!!python/unicode 'vocab_target': !!python/unicode 'vocab.bpe.32000'
INFO:tensorflow:Creating vocabulary lookup table of size 37007
INFO:tensorflow:Creating vocabulary lookup table of size 37007
INFO:tensorflow:Creating ConvEncoderFairseq in mode=train
INFO:tensorflow:
ConvEncoderFairseq: {cnn.kwidth_default: 3, cnn.kwidths: '3,3,3,3', cnn.layers: 4,
cnn.nhid_default: 256, cnn.nhids: '256,256,256,256', embedding_dropout_xxxep_prob: 0.9,
nhid_dropout_xxxep_prob: 0.9, position_embeddings.combiner_fn: tensorflow.add, position_embeddings.enable: true}
INFO:tensorflow:Creating ConvDecoderFairseq in mode=train
INFO:tensorflow:
ConvDecoderFairseq: {!!python/unicode 'cnn.kwidth_default': 3, !!python/unicode 'cnn.kwidths': !!python/unicode '3,3,3',
!!python/unicode 'cnn.layers': 3, !!python/unicode 'cnn.nhid_default': 256, !!python/unicode 'cnn.nhids': !!python/unicode '256,256,256',
!!python/unicode 'embedding_dropout_xxxep_prob': 0.9, !!python/unicode 'max_decode_length': 49,
!!python/unicode 'nhid_dropout_xxxep_prob': 0.9, !!python/unicode 'nout_embed': 256,
!!python/unicode 'out_dropout_xxxep_prob': 0.9, !!python/unicode 'position_embeddings.combiner_fn': !!python/unicode 'tensorflow.add',
!!python/unicode 'position_embeddings.enable': true}
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/V/read:0", shape=(256, 256), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/V:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/g/read:0", shape=(256,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/g:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/b/read:0", shape=(256,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/b:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/V/read:0", shape=(3, 256, 512), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/V:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/g/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/g:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/b/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/b:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/V/read:0", shape=(3, 256, 512), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/V:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/g/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/g:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/b/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/b:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/V/read:0", shape=(3, 256, 512), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/V:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/g/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/g:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/b/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/b:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/V/read:0", shape=(3, 256, 512), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/V:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/g/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/g:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/b/read:0", shape=(512,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/b:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/V/read:0", shape=(256, 256), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/V:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/g/read:0", shape=(256,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/g:0
INFO:tensorflow:tensor Tensor("model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/b/read:0", shape=(256,), dtype=float32), name is model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/b:0
INFO:tensorflow:Create CheckpointSaverHook.
47 ops no flops stats due to incomplete shapes. Consider passing run_meta to use run_time shapes.
Parsing GraphDef...
Parsing RunMetadata...
Parsing OpLog...
Preparing Views...
INFO:tensorflow:_TFProfRoot (--/31.97m params)
model/conv_seq2seq/Variable (0/0 params)
model/conv_seq2seq/decode/W (37007x256, 9.47m/9.47m params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_out/V (256x256, 65.54k/65.54k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_out/b (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_out/g (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_query/V (256x256, 65.54k/65.54k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_query/b (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_0/linear_mapping_att_query/g (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_out/V (256x256, 65.54k/65.54k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_out/b (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_out/g (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_query/V (256x256, 65.54k/65.54k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_query/b (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_1/linear_mapping_att_query/g (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_out/V (256x256, 65.54k/65.54k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_out/b (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_out/g (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_query/V (256x256, 65.54k/65.54k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_query/b (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/attention_layer_2/linear_mapping_att_query/g (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_0/V (3x256x512, 393.22k/393.22k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_0/b (512, 512/512 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_0/g (512, 512/512 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_1/V (3x256x512, 393.22k/393.22k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_1/b (512, 512/512 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_1/g (512, 512/512 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_2/V (3x256x512, 393.22k/393.22k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_2/b (512, 512/512 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/conv_layer_2/g (512, 512/512 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/linear_mapping_before_cnn/V (256x256, 65.54k/65.54k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/linear_mapping_before_cnn/b (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/decoder_cnn/linear_mapping_before_cnn/g (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/linear_mapping_after_cnn/V (256x256, 65.54k/65.54k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/linear_mapping_after_cnn/b (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/linear_mapping_after_cnn/g (256, 256/256 params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/V (256x37007, 9.47m/9.47m params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/b (37007, 37.01k/37.01k params)
model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/g (37007, 37.01k/37.01k params)
model/conv_seq2seq/decode/pos (100x256, 25.60k/25.60k params)
model/conv_seq2seq/encode/W (37007x256, 9.47m/9.47m params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/V (3x256x512, 393.22k/393.22k params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/b (512, 512/512 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_0/g (512, 512/512 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/V (3x256x512, 393.22k/393.22k params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/b (512, 512/512 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_1/g (512, 512/512 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/V (3x256x512, 393.22k/393.22k params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/b (512, 512/512 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_2/g (512, 512/512 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/V (3x256x512, 393.22k/393.22k params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/b (512, 512/512 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/conv_layer_3/g (512, 512/512 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/V (256x256, 65.54k/65.54k params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/b (256, 256/256 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_after_cnn/g (256, 256/256 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/V (256x256, 65.54k/65.54k params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/b (256, 256/256 params)
model/conv_seq2seq/encode/conv_encoder/encoder_cnn/linear_mapping_before_cnn/g (256, 256/256 params)
model/conv_seq2seq/encode/pos (100x256, 25.60k/25.60k params)
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/Users/xxx.xxx/xxx_prj/conv_seq2seq/bin/train.py", line 277, in
tf.app.run()
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/Users/xxx.xxx/xxx_prj/conv_seq2seq/bin/train.py", line 272, in main
schedule=FLAGS.schedule)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 106, in run
return task()
File "seq2seq/contrib/experiment.py", line 104, in continuous_train_and_eval
monitors=self._train_monitors)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 280, in new_func
return func(*args, **kwargs)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 426, in fit
loss = self._train_model(input_fn=input_fn, hooks=hooks)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 981, in _train_model
config=self.config.tf_config) as mon_sess:
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 315, in MonitoredTrainingSession
return MonitoredSession(session_creator=session_creator, hooks=all_hooks)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 601, in init
session_creator, hooks, should_recover=True)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 434, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 767, in init
_WrappedSession.init(self, self._create_session())
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 772, in _create_session
return self._sess_creator.create_session()
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 494, in create_session
self.tf_sess = self._session_creator.create_session()
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 375, in create_session
init_fn=self._scaffold.init_fn)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 256, in prepare_session
config=config)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 188, in _restore_checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1428, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [37007] rhs shape= [12003]
[[Node: save/Assign_96 = Assign[T=DT_FLOAT, _class=["loc:@model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/b"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/b, save/RestoreV2_96)]]
Caused by op u'save/Assign_96', defined at:
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/Users/xxx.xxx/xxx_prj/conv_seq2seq/bin/train.py", line 277, in
tf.app.run()
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/Users/xxx.xxx/xxx_prj/conv_seq2seq/bin/train.py", line 272, in main
schedule=FLAGS.schedule)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 106, in run
return task()
File "seq2seq/contrib/experiment.py", line 104, in continuous_train_and_eval
monitors=self._train_monitors)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 280, in new_func
return func(*args, **kwargs)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 426, in fit
loss = self._train_model(input_fn=input_fn, hooks=hooks)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 981, in _train_model
config=self.config.tf_config) as mon_sess:
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 315, in MonitoredTrainingSession
return MonitoredSession(session_creator=session_creator, hooks=all_hooks)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 601, in init
session_creator, hooks, should_recover=True)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 434, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 767, in init
_WrappedSession.init(self, self._create_session())
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 772, in _create_session
return self._sess_creator.create_session()
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 494, in create_session
self.tf_sess = self._session_creator.create_session()
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 366, in create_session
self._scaffold.finalize()
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 183, in finalize
self._saver.build()
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1070, in build
restore_sequentially=self._restore_sequentially)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 671, in build
restore_sequentially, reshape)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 445, in _AddShardedRestoreOps
name="restore_shard"))
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 414, in _AddRestoreOps
assign_ops.append(saveable.restore(tensors, shapes))
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 155, in restore
self.op.get_shape().is_fully_defined())
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py", line 47, in assign
use_locking=use_locking, name=name)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/Users/xxx.xxx/xxx_prj/ve_tf1.0_py2/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1226, in init
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [37007] rhs shape= [12003]
[[Node: save/Assign_96 = Assign[T=DT_FLOAT, _class=["loc:@model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/b"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](model/conv_seq2seq/decode/conv_decoder_fairseq/decoder/softmax/logits_before_softmax/b, save/RestoreV2_96)]]
The text was updated successfully, but these errors were encountered: