Data directory assertion failure

[cnewton]

I am getting a WARNING:tensorflow:Estimator's model_fn ... includes params argument, but params are not passed to Estimator. warning followed by a assert data_dir (from https://github.com/cmusphinx/g2p-seq2seq/blob/master/g2p_seq2seq/g2p_problem.py#L238) failure when training with the following command (on current master branch):

$ g2p-seq2seq --train cmudict.dict --model_dir g2p-model-nostress --cleanup --reinit

Package versions used (also reproducible with latest tensorflow / tensor2tensor):

tensor2tensor == 1.6.6
tensorflow == 1.8.0
tensorflow-gpu == 1.8.0

Error log below:

INFO:tensorflow:Importing user module g2p_seq2seq from path /usr/local/lib/python2.7/dist-packages/g2p_seq2seq-6.2.2a0-py2.7.egg
INFO:tensorflow:Overriding hparams in transformer_base with eval_drop_long_sequences=1,batch_size=4096,num_hidden_layers=3,hidden_size=256,filter_size=512,num_heads=4,length_bucket_step=1.5,max_length=30,min_length_bucket=6
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor-1.6.6-py2.7.egg/tensor2tensor/utils/trainer_lib.py:165: __init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.
Instructions for updating:
When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.
INFO:tensorflow:schedule=continuous_train_and_eval
INFO:tensorflow:worker_gpu=1
INFO:tensorflow:sync=False
WARNING:tensorflow:Schedule=continuous_train_and_eval. Assuming that training is running on a single machine.
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:ps_devices: ['gpu:0']
INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 1, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f527c2cdad0>, '_keep_checkpoint_every_n_hours': 1, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 2000, '_environment': 'local', '_master': '', '_model_dir': 'g2p-model-nostress', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x7f527c2cde50>, '_save_summary_steps': 100}
WARNING:tensorflow:Estimator's model_fn (<function wrapping_model_fn at 0x7f527c203e60>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:ValidationMonitor only works with --schedule=train_and_evaluate
INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 1, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f527c2cdad0>, '_keep_checkpoint_every_n_hours': 1, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 2000, '_environment': 'local', '_master': '', '_model_dir': 'g2p-model-nostress', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x7f527c2cde50>, '_save_summary_steps': 100}
WARNING:tensorflow:Estimator's model_fn (<function wrapping_model_fn at 0x7f527c203ed8>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after 600 secs (eval_spec.throttle_secs) or training is finished.
Traceback (most recent call last):
  File "/usr/local/bin/g2p-seq2seq", line 11, in <module>
    load_entry_point('g2p-seq2seq==6.2.2a0', 'console_scripts', 'g2p-seq2seq')()
  File "/usr/local/lib/python2.7/dist-packages/g2p_seq2seq-6.2.2a0-py2.7.egg/g2p_seq2seq/app.py", line 113, in main
    g2p_model.train()
  File "/usr/local/lib/python2.7/dist-packages/g2p_seq2seq-6.2.2a0-py2.7.egg/g2p_seq2seq/g2p.py", line 259, in train
    execute_schedule(self.exp, self.params)
  File "/usr/local/lib/python2.7/dist-packages/g2p_seq2seq-6.2.2a0-py2.7.egg/g2p_seq2seq/g2p.py", line 589, in execute_schedule
    getattr(exp, params.schedule)()
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor-1.6.6-py2.7.egg/tensor2tensor/utils/trainer_lib.py", line 289, in continuous_train_and_eval
    self._eval_spec)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 439, in train_and_evaluate
    executor.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 518, in run
    self.run_local()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 650, in run_local
    hooks=train_hooks)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 363, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 853, in _train_model_default
    input_fn, model_fn_lib.ModeKeys.TRAIN))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 691, in _get_features_and_labels_from_input_fn
    result = self._call_input_fn(input_fn, mode)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 798, in _call_input_fn
    return input_fn(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor-1.6.6-py2.7.egg/tensor2tensor/data_generators/problem.py", line 690, in estimator_input_fn
    dataset_kwargs=dataset_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor-1.6.6-py2.7.egg/tensor2tensor/data_generators/problem.py", line 782, in input_fn
    dataset = self.dataset(**dataset_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/g2p_seq2seq-6.2.2a0-py2.7.egg/g2p_seq2seq/g2p_problem.py", line 238, in dataset
    assert data_dir
AssertionError

[cnewton]

There's an assumption in params.py that data_path will always contain a directory name:

self.data_dir_name = os.path.dirname(data_path)

If data_path is like I have above (cmudict.dict), then dirname will of course return empty string. Giving a path works fine:

$ g2p-seq2seq --train ~/g2p-seq2seq/cmudict.dict --model_dir g2p-model-nostress --cleanup --reinit

Can this be fixed or the setup directions be clarified to avoid the confusion? Right now README.md states the training example without a path:

  g2p-seq2seq --train train_dictionary.dic --model_dir model_folder_path