"Dimensions must be equal" Error with HMC&SGLD

[mathetake]

Hi. I am new to here and stuck with some problems with HMC&SGLD.

I've tried to develop a 4-layer neural network with Edward's HMC&SGLD inference.

My code for SGLD is the following;

def four_layer_nn(x,W_1,W_2,W_3,b_1,b_2):
    h = tf.tanh(tf.matmul(x, W_1) + b_1)
    h = tf.tanh(tf.matmul(h, W_2) + b_2)
    h = tf.matmul(h, W_3) 
    return tf.reshape(h, [-1])

N = X_train.shape[0]  # data points
D = X_train.shape[1] # feature
M = 50000 #batch size

W_1 = Normal(mu=tf.zeros([D, 20]), sigma=tf.ones([D, 20])*100)
W_2 = Normal(mu=tf.zeros([20, 15]), sigma=tf.ones([20, 15])*100)
W_3 = Normal(mu=tf.zeros([15, 1]), sigma=tf.ones([15, 1])*100)
b_1 = Normal(mu=tf.zeros(20), sigma=tf.ones(20)*100)
b_2 = Normal(mu=tf.zeros(15), sigma=tf.ones(15)*100)

#Empirical Variable for SGLD
qW_1 = Empirical(params=tf.Variable(tf.random_normal([M,D,20])))
qW_2 = Empirical(params=tf.Variable(tf.random_normal([M,20,15])))
qW_3 = Empirical(params=tf.Variable(tf.random_normal([M,15,1])))
qb_1 = Empirical(params=tf.Variable(tf.random_normal([M,20])))
qb_2 = Empirical(params=tf.Variable(tf.random_normal([M,15])))

x_ph  = tf.placeholder(tf.float32,[M,D])
y = Bernoulli(logits=four_layer_nn(x_ph, W_1, W_2, W_3, b_1, b_2))
y_ph = tf.placeholder(tf.float32,[M])
inference = ed.SGLD({W_1: qW_1, b_1: qb_1, W_2: qW_2 , b_2: qb_2, W_3: qW_3 }, data={y: y_ph})
inference.initialize(n_iter=n_epoch*int(float(N)/M), n_print=100)
...(Batch training codes follow.)

When I run this code, the following error occurs:

Traceback (most recent call last):
  File "dnn_SGLD.py", line 85, in <module>
    inference.initialize(n_iter=n_epoch*int(float(N)/M), n_print=100)
  File "/Users/komedatake/stat_model/higgs_boson/src/edward/edward/inferences/sgld.py", line 53, in initialize
    return super(SGLD, self).initialize(*args, **kwargs)
  File "/Users/komedatake/stat_model/higgs_boson/src/edward/edward/inferences/monte_carlo.py", line 101, in initialize
    self.train = self.build_update()
  File "/Users/komedatake/stat_model/higgs_boson/src/edward/edward/inferences/sgld.py", line 75, in build_update
    sample[z] = old_sample[z] + 0.5 * learning_rate * grad_log_p + \
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 751, in binary_op_wrapper
    return func(x, y, name=name)
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 71, in add
    result = _op_def_lib.apply_op("Add", x=x, y=y, name=name)
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
    op_def=op_def)
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2382, in create_op
    set_shapes_for_outputs(ret)
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1783, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 596, in call_cpp_shape_fn
    raise ValueError(err.message)
ValueError: Dimensions must be equal, but are 20 and 15

If I properly replace SGLD and Empirical by KLqp and Normal, there's no problem and the inference begins. The weird thing is that the numbers "20 and 15" in ValueError sometimes change into "15 and 7", or "15 and 20".

The same kind of error also occurred with HMC:

Traceback (most recent call last):
  File "dnn_SGLD.py", line 85, in <module>
    inference.initialize(n_iter=n_epoch*int(float(N)/M), n_print=100)
  File "/Users/komedatake/stat_model/higgs_boson/src/edward/edward/inferences/hmc.py", line 58, in initialize
    return super(HMC, self).initialize(*args, **kwargs)
  File "/Users/komedatake/stat_model/higgs_boson/src/edward/edward/inferences/monte_carlo.py", line 101, in initialize
    self.train = self.build_update()
  File "/Users/komedatake/stat_model/higgs_boson/src/edward/edward/inferences/hmc.py", line 81, in build_update
    self.step_size, self._log_joint)
  File "/Users/komedatake/stat_model/higgs_boson/src/edward/edward/inferences/hmc.py", line 167, in leapfrog
    r_new[key] += 0.5 * step_size * grad_log_joint[i]
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 751, in binary_op_wrapper
    return func(x, y, name=name)
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 71, in add
    result = _op_def_lib.apply_op("Add", x=x, y=y, name=name)
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
    op_def=op_def)
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2382, in create_op
    set_shapes_for_outputs(ret)
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1783, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/Users/komedatake/anaconda/envs/myenv/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 596, in call_cpp_shape_fn
    raise ValueError(err.message)
ValueError: Dimensions must be equal, but are 20 and 7

I already tried to run some examples of SGLD and HMC, and they worked out correctly.

How can I solve this problem? Any comments would be appreciated.

Thank you.

[dustinvtran]

i found/resolved the problem in #413; adding some unit tests before i merge it.

[dustinvtran]

fixed in #413