Pretraining: batch_size must fit number of experiences in demo file?

[Dastyn]

Hi!

It seems that when training with pretraining, the batch_size in the yaml config file must be equal to the number of experiences recorded in the corresponding .demo file.

If not equal, then an exception is raised:

Setting up 2 worker threads for Enlighten.
  Thread -> id: 7fb9e2ffd700 -> priority: 1 
  Thread -> id: 7fb9e27fc700 -> priority: 1 
##utp:{"type":"MemoryLeaks","version":2,"phase":"Immediate","time":1581079838307,"processId":26838,"allocatedMemory":4079,"memoryLabels":[{"Default":40},{"Permanent":40},{"NewDelete":112},{"Thread":128},{"Manager":1680},{"GfxDevice":64},{"Physics":32},{"Serialization":40},{"String":311},{"GI":296},{"VR":1992},{"Subsystems":-656}]}
INFO:mlagents_envs:Environment shut down with return code 0.
Traceback (most recent call last):
  File "/home/unity/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/home/unity/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/unity/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: logits and labels must be broadcastable: logits_size=[579,2] labels_size=[573,2]
	 [[{{node softmax_cross_entropy_with_logits_5}}]]
	 [[Mean_3/_395]]
  (1) Invalid argument: logits and labels must be broadcastable: logits_size=[579,2] labels_size=[573,2]
	 [[{{node softmax_cross_entropy_with_logits_5}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/unity/.local/bin/mlagents-learn", line 11, in <module>
    sys.exit(main())
  File "/home/unity/.local/lib/python3.6/site-packages/mlagents/trainers/learn.py", line 478, in main
    run_training(0, run_seed, options, Queue())
  File "/home/unity/.local/lib/python3.6/site-packages/mlagents/trainers/learn.py", line 316, in run_training
    tc.start_learning(env_manager)
  File "/home/unity/.local/lib/python3.6/site-packages/mlagents/trainers/trainer_controller.py", line 234, in start_learning
    n_steps = self.advance(env_manager)
  File "/home/unity/.local/lib/python3.6/site-packages/mlagents_envs/timers.py", line 262, in wrapped
    return func(*args, **kwargs)
  File "/home/unity/.local/lib/python3.6/site-packages/mlagents/trainers/trainer_controller.py", line 321, in advance
    trainer.update_policy()
  File "/home/unity/.local/lib/python3.6/site-packages/mlagents/trainers/ppo/trainer.py", line 204, in update_policy
    buffer.make_mini_batch(l, l + batch_size), n_sequences
  File "/home/unity/.local/lib/python3.6/site-packages/mlagents_envs/timers.py", line 262, in wrapped
    return func(*args, **kwargs)
  File "/home/unity/.local/lib/python3.6/site-packages/mlagents/trainers/ppo/policy.py", line 184, in update
    update_vals = self._execute_model(feed_dict, self.update_dict)
  File "/home/unity/.local/lib/python3.6/site-packages/mlagents/trainers/tf_policy.py", line 165, in _execute_model
    network_out = self.sess.run(list(out_dict.values()), feed_dict=feed_dict)
  File "/home/unity/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/home/unity/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/unity/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/home/unity/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: logits and labels must be broadcastable: logits_size=[579,2] labels_size=[573,2]
	 [[node softmax_cross_entropy_with_logits_5 (defined at /lib/python3.6/site-packages/mlagents/trainers/ppo/models.py:295) ]]
	 [[Mean_3/_395]]
  (1) Invalid argument: logits and labels must be broadcastable: logits_size=[579,2] labels_size=[573,2]
	 [[node softmax_cross_entropy_with_logits_5 (defined at /lib/python3.6/site-packages/mlagents/trainers/ppo/models.py:295) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'softmax_cross_entropy_with_logits_5':
  File "/bin/mlagents-learn", line 11, in <module>
    sys.exit(main())
  File "/lib/python3.6/site-packages/mlagents/trainers/learn.py", line 478, in main
    run_training(0, run_seed, options, Queue())
  File "/lib/python3.6/site-packages/mlagents/trainers/learn.py", line 316, in run_training
    tc.start_learning(env_manager)
  File "/lib/python3.6/site-packages/mlagents/trainers/trainer_controller.py", line 213, in start_learning
    env_manager.external_brains[name_behavior_id]
  File "/lib/python3.6/site-packages/mlagents/trainers/ppo/trainer.py", line 239, in create_policy
    self.load,
  File "/lib/python3.6/site-packages/mlagents/trainers/ppo/policy.py", line 48, in __init__
    brain, trainer_params, reward_signal_configs, is_training, load, seed
  File "/lib/python3.6/site-packages/mlagents/trainers/ppo/policy.py", line 98, in create_model
    trainer_params.get("vis_encode_type", "simple")
  File "/lib/python3.6/site-packages/mlagents/trainers/ppo/models.py", line 61, in __init__
    self.create_dc_actor_critic(h_size, num_layers, vis_encode_type)
  File "/lib/python3.6/site-packages/mlagents/trainers/ppo/models.py", line 295, in create_dc_actor_critic
    for i in range(len(self.act_size))
  File "/lib/python3.6/site-packages/mlagents/trainers/ppo/models.py", line 295, in <listcomp>
    for i in range(len(self.act_size))
  File "/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 3151, in softmax_cross_entropy_with_logits_v2_helper
    precise_logits, labels, name=name)
  File "/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 10970, in softmax_cross_entropy_with_logits
    name=name)
  File "/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Typically, the demo file actually contained 573 experiences, but in the yaml config file, the batch_size = 579.

• OS: Linux Ubuntu 18.4
• ML-Agents version: v0.13.1
• TensorFlow version: 1.14

[ervteng]

Hi @Dastyn, thanks for reporting this, I've logged it with internal ID
MLA-616.

Basically there needs to be more than batch_size number of experiences in the demo file so that you could train at least one batch of demos. But the error message is pretty cryptic.

[ervteng]

This bug has been fixed in the latest master and will be released shortly. Closing this issue for now.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.