Why the validation loss is always increasing?

[imzhangyd]

❓ Questions and Help

The trainning loss and metrics seems normal during training, but the validation loss is increasing. Please see bolow information for details.
I'm curious about the reason behind this issue, and whether there might be an error during the training process. If there is an error, could you please let me know how I can correct it?

Model, Dataset, Config

I trained mask-rcnn-R50-FPN-1x model on my own dataset. There is one object on each image in my dataset.
I change some params:
PRE_NMS_TOP_N_TRAIN: 200
PRE_NMS_TOP_N_TEST: 100
POST_NMS_TOP_N_TRAIN: 200
POST_NMS_TOP_N_TEST: 100
FPN_POST_NMS_TOP_N_TRAIN: 400
FPN_POST_NMS_TOP_N_TEST: 100
IMS_PER_BATCH: 8

Loss and metrics

The trainning loss seems normal, but the validation loss is always increasing. I don't know where's the problem.

The trainning loss:

The validation loss:

The metrics seems normal: