Unable to rebuild saved FM model

[jbixler24]

Hello, I am using the FM algorithm for film recommendation for a project assignment. I trained the first fold of my dataset and saved the model using fm.save(path='modeling/model', model_name='fm_model_0_wo_inf', manual=True, inference_only=False). I then merged my old DataInfo object with the second fold of my dataset to continue training my model using merge_behavior=True and reset the TF default graph as noted in the model retrain example file. However, when I then try to reload my saved model, I get the following ValueError:

ValueError: Argument fetch = name: "init" op: "NoOp" input: "^embedding[/user_linear_var/Assign](http://localhost:8888/user_linear_var/Assign)" input: "^embedding[/item_linear_var/Assign](http://localhost:8888/item_linear_var/Assign)" input: "^embedding[/user_embeds_var/Assign](http://localhost:8888/user_embeds_var/Assign)" input: "^embedding[/item_embeds_var/Assign](http://localhost:8888/item_embeds_var/Assign)" input: "^embedding[/sparse_linear_var/Assign](http://localhost:8888/sparse_linear_var/Assign)" input: "^embedding[/sparse_embeds_var/Assign](http://localhost:8888/sparse_embeds_var/Assign)" input: "^embedding[/dense_linear_var/Assign](http://localhost:8888/dense_linear_var/Assign)" input: "^embedding[/dense_embeds_var/Assign](http://localhost:8888/dense_embeds_var/Assign)" input: "^dense[/kernel/Assign](http://localhost:8888/kernel/Assign)" input: "^dense[/bias/Assign](http://localhost:8888/bias/Assign)" input: "^batch_normalization[/gamma/Assign](http://localhost:8888/gamma/Assign)" input: "^batch_normalization[/beta/Assign](http://localhost:8888/beta/Assign)" input: "^batch_normalization[/moving_mean/Assign](http://localhost:8888/moving_mean/Assign)" input: "^batch_normalization[/moving_variance/Assign](http://localhost:8888/moving_variance/Assign)" input: "^dense_1[/kernel/Assign](http://localhost:8888/kernel/Assign)" input: "^dense_1[/bias/Assign](http://localhost:8888/bias/Assign)" input: "^beta1_power[/Assign](http://localhost:8888/Assign)" input: "^beta2_power[/Assign](http://localhost:8888/Assign)" input: "^embedding[/user_linear_var/Adam/Assign](http://localhost:8888/user_linear_var/Adam/Assign)" input: "^embedding[/user_linear_var/Adam_1/Assign](http://localhost:8888/user_linear_var/Adam_1/Assign)" input: "^embedding[/item_linear_var/Adam/Assign](http://localhost:8888/item_linear_var/Adam/Assign)" input: "^embedding[/item_linear_var/Adam_1/Assign](http://localhost:8888/item_linear_var/Adam_1/Assign)" input: "^embedding[/user_embeds_var/Adam/Assign](http://localhost:8888/user_embeds_var/Adam/Assign)" input: "^embedding[/user_embeds_var/Adam_1/Assign](http://localhost:8888/user_embeds_var/Adam_1/Assign)" input: "^embedding[/item_embeds_var/Adam/Assign](http://localhost:8888/item_embeds_var/Adam/Assign)" input: "^embedding[/item_embeds_var/Adam_1/Assign](http://localhost:8888/item_embeds_var/Adam_1/Assign)" input: "^embedding[/sparse_linear_var/Adam/Assign](http://localhost:8888/sparse_linear_var/Adam/Assign)" input: "^embedding[/sparse_linear_var/Adam_1/Assign](http://localhost:8888/sparse_linear_var/Adam_1/Assign)" input: "^embedding[/sparse_embeds_var/Adam/Assign](http://localhost:8888/sparse_embeds_var/Adam/Assign)" input: "^embedding[/sparse_embeds_var/Adam_1/Assign](http://localhost:8888/sparse_embeds_var/Adam_1/Assign)" input: "^embedding[/dense_linear_var/Adam/Assign](http://localhost:8888/dense_linear_var/Adam/Assign)" input: "^embedding[/dense_linear_var/Adam_1/Assign](http://localhost:8888/dense_linear_var/Adam_1/Assign)" input: "^embedding[/dense_embeds_var/Adam/Assign](http://localhost:8888/dense_embeds_var/Adam/Assign)" input: "^embedding[/dense_embeds_var/Adam_1/Assign](http://localhost:8888/dense_embeds_var/Adam_1/Assign)" input: "^dense[/kernel/Adam/Assign](http://localhost:8888/kernel/Adam/Assign)" input: "^dense[/kernel/Adam_1/Assign](http://localhost:8888/kernel/Adam_1/Assign)" input: "^dense[/bias/Adam/Assign](http://localhost:8888/bias/Adam/Assign)" input: "^dense[/bias/Adam_1/Assign](http://localhost:8888/bias/Adam_1/Assign)" input: "^batch_normalization[/gamma/Adam/Assign](http://localhost:8888/gamma/Adam/Assign)" input: "^batch_normalization[/gamma/Adam_1/Assign](http://localhost:8888/gamma/Adam_1/Assign)" input: "^batch_normalization[/beta/Adam/Assign](http://localhost:8888/beta/Adam/Assign)" input: "^batch_normalization[/beta/Adam_1/Assign](http://localhost:8888/beta/Adam_1/Assign)" input: "^dense_1[/kernel/Adam/Assign](http://localhost:8888/kernel/Adam/Assign)" input: "^dense_1[/kernel/Adam_1/Assign](http://localhost:8888/kernel/Adam_1/Assign)" input: "^dense_1[/bias/Adam/Assign](http://localhost:8888/bias/Adam/Assign)" input: "^dense_1[/bias/Adam_1/Assign](http://localhost:8888/bias/Adam_1/Assign)" cannot be interpreted as a Tensor. (Operation name: "init" op: "NoOp" input: "^embedding[/user_linear_var/Assign](http://localhost:8888/user_linear_var/Assign)" input: "^embedding[/item_linear_var/Assign](http://localhost:8888/item_linear_var/Assign)" input: "^embedding[/user_embeds_var/Assign](http://localhost:8888/user_embeds_var/Assign)" input: "^embedding[/item_embeds_var/Assign](http://localhost:8888/item_embeds_var/Assign)" input: "^embedding[/sparse_linear_var/Assign](http://localhost:8888/sparse_linear_var/Assign)" input: "^embedding[/sparse_embeds_var/Assign](http://localhost:8888/sparse_embeds_var/Assign)" input: "^embedding[/dense_linear_var/Assign](http://localhost:8888/dense_linear_var/Assign)" input: "^embedding[/dense_embeds_var/Assign](http://localhost:8888/dense_embeds_var/Assign)" input: "^dense[/kernel/Assign](http://localhost:8888/kernel/Assign)" input: "^dense[/bias/Assign](http://localhost:8888/bias/Assign)" input: "^batch_normalization[/gamma/Assign](http://localhost:8888/gamma/Assign)" input: "^batch_normalization[/beta/Assign](http://localhost:8888/beta/Assign)" input: "^batch_normalization[/moving_mean/Assign](http://localhost:8888/moving_mean/Assign)" input: "^batch_normalization[/moving_variance/Assign](http://localhost:8888/moving_variance/Assign)" input: "^dense_1[/kernel/Assign](http://localhost:8888/kernel/Assign)" input: "^dense_1[/bias/Assign](http://localhost:8888/bias/Assign)" input: "^beta1_power[/Assign](http://localhost:8888/Assign)" input: "^beta2_power[/Assign](http://localhost:8888/Assign)" input: "^embedding[/user_linear_var/Adam/Assign](http://localhost:8888/user_linear_var/Adam/Assign)" input: "^embedding[/user_linear_var/Adam_1/Assign](http://localhost:8888/user_linear_var/Adam_1/Assign)" input: "^embedding[/item_linear_var/Adam/Assign](http://localhost:8888/item_linear_var/Adam/Assign)" input: "^embedding[/item_linear_var/Adam_1/Assign](http://localhost:8888/item_linear_var/Adam_1/Assign)" input: "^embedding[/user_embeds_var/Adam/Assign](http://localhost:8888/user_embeds_var/Adam/Assign)" input: "^embedding[/user_embeds_var/Adam_1/Assign](http://localhost:8888/user_embeds_var/Adam_1/Assign)" input: "^embedding[/item_embeds_var/Adam/Assign](http://localhost:8888/item_embeds_var/Adam/Assign)" input: "^embedding[/item_embeds_var/Adam_1/Assign](http://localhost:8888/item_embeds_var/Adam_1/Assign)" input: "^embedding[/sparse_linear_var/Adam/Assign](http://localhost:8888/sparse_linear_var/Adam/Assign)" input: "^embedding[/sparse_linear_var/Adam_1/Assign](http://localhost:8888/sparse_linear_var/Adam_1/Assign)" input: "^embedding[/sparse_embeds_var/Adam/Assign](http://localhost:8888/sparse_embeds_var/Adam/Assign)" input: "^embedding[/sparse_embeds_var/Adam_1/Assign](http://localhost:8888/sparse_embeds_var/Adam_1/Assign)" input: "^embedding[/dense_linear_var/Adam/Assign](http://localhost:8888/dense_linear_var/Adam/Assign)" input: "^embedding[/dense_linear_var/Adam_1/Assign](http://localhost:8888/dense_linear_var/Adam_1/Assign)" input: "^embedding[/dense_embeds_var/Adam/Assign](http://localhost:8888/dense_embeds_var/Adam/Assign)" input: "^embedding[/dense_embeds_var/Adam_1/Assign](http://localhost:8888/dense_embeds_var/Adam_1/Assign)" input: "^dense[/kernel/Adam/Assign](http://localhost:8888/kernel/Adam/Assign)" input: "^dense[/kernel/Adam_1/Assign](http://localhost:8888/kernel/Adam_1/Assign)" input: "^dense[/bias/Adam/Assign](http://localhost:8888/bias/Adam/Assign)" input: "^dense[/bias/Adam_1/Assign](http://localhost:8888/bias/Adam_1/Assign)" input: "^batch_normalization[/gamma/Adam/Assign](http://localhost:8888/gamma/Adam/Assign)" input: "^batch_normalization[/gamma/Adam_1/Assign](http://localhost:8888/gamma/Adam_1/Assign)" input: "^batch_normalization[/beta/Adam/Assign](http://localhost:8888/beta/Adam/Assign)" input: "^batch_normalization[/beta/Adam_1/Assign](http://localhost:8888/beta/Adam_1/Assign)" input: "^dense_1[/kernel/Adam/Assign](http://localhost:8888/kernel/Adam/Assign)" input: "^dense_1[/kernel/Adam_1/Assign](http://localhost:8888/kernel/Adam_1/Assign)" input: "^dense_1[/bias/Adam/Assign](http://localhost:8888/bias/Adam/Assign)" input: "^dense_1[/bias/Adam_1/Assign](http://localhost:8888/bias/Adam_1/Assign)" is not an element of this graph.)

Below is my code for reference:

train_dataset, train_datainfo = DatasetFeat.build_trainset(train,
user_col=user_cols,
item_col=movie_cols,
sparse_col=sparse_cols,
dense_col=dense_cols)
valid_dataset = DatasetFeat.build_evalset(valid)

fm = FM(task='ranking',
data_info=train_datainfo,
reg=10e-4,
seed=42,
)

fm.fit(
train_dataset,
neg_sampling=True,
verbose=2,
shuffle=False,
eval_data=valid_dataset,
metrics=['precision', 'recall', 'ndcg', 'roc_auc'],
)

fm.save(path='modeling/model', model_name='fm_model_0', manual=True, inference_only=False)

_, n_data_info = DatasetFeat.merge_trainset(
new_train, train_datainfo, merge_behavior=True)

new_fm = FM(task='ranking',
data_info=n_data_info,
reg=10e-4,
seed=42
)

tf.compat.v1.reset_default_graph()
new_fm.rebuild_model(path='modeling/model', model_name='fm_model_0', full_assign=False)

[massquantity]

You should reset the TF default graph before constructing the new model, i.e. call tf.compat.v1.reset_default_graph() before new_fm = FM(...).