Inconsistent results on property evaluation when using eval_proppred or train

[LeoGrin]

Hello, and thanks for the great work!

I've been trying to reproduce the evaluations on cross-model property prediction. When training, I get very low MSE, but running eval_proppred afterward outputs very bad results. Looking more closely, the predictions in eval_proppred seem really bad (negative for positive target values for instance). Do you know what might be causing this? (I see that there is a unused variable bn for BatchNormalization in this file, could it be related?)

More details on what I ran

I tried to copy command from your .sh files, only sometimes reducing the number of max_epoch.

First I generated the data.

python train.py --export_data True --is_proppred True --property_type ncr --dump_path ./dump --use_skeleton True --max_input_dimension 1 --n_steps_per_epoch 3125 --max_epoch 1000 --exp_name data --exp_id ncr

I stopped the script when I got ~80K samples, and separated the data into test (last 1K samples) and train (the rest). Then I trained with frozen encoder starting from your 1d pretrained model:

python train.py --reload_model ./weights/snip-1d-normalized.pth --is_proppred True --property_type ncr --freeze_encoder True --reload_data functions,dump/data/ncr/train.prefix,dump/data/ncr/train.prefix, --normalize_y True --batch_size 16 --dump_path ./dump --max_input_dimension 1 --n_steps_per_epoch 625 --max_epoch 21 --exp_name NCR_pred --exp_id original_model --lr 1e-5 --latent_dim 512 --save_periodic 10

Then I run eval_proppred:

python eval_proppred.py --is_proppred True \
                        --property_type ncr \
                        --reload_model ./dump/NCR_pred/original_model/periodic-20.pth \
                        --reload_data functions,dump/data/ncr/test.prefix,dump/data/ncr/test.prefix,\
                        --embedder_type LinearPointEmbedder  \
                       # --max_input_dimension 1 \
                        #--normalize_y True  \
                        #--use_skeleton True

(the commented values mean I tried with and without it).

Thanks in advance for you help!