How to pick the best epoch?

[ktobah]

Hello,

I have used your repo to train a yolact++ model on my custom dataset. After the training finished, I found some weights saved. I collected the corresponding evaluation mAPs as follow:

       |  all  |  .50  |  .55  |  .60  |  .65  |  .70  |  .75  |  .80  |  .85  |  .90  |  .95  |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 61.44 | 80.53 | 80.22 | 79.55 | 76.49 | 74.82 | 72.89 | 66.13 | 51.65 | 29.44 |  2.70 |
  mask | 61.86 | 75.29 | 74.92 | 74.30 | 74.09 | 71.49 | 70.52 | 67.08 | 58.66 | 41.70 | 10.57 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 67.98 | 85.15 | 84.73 | 83.99 | 83.54 | 80.88 | 79.63 | 72.96 | 65.73 | 37.70 |  5.48 |
  mask | 67.06 | 79.99 | 79.54 | 79.39 | 78.23 | 77.86 | 74.65 | 72.50 | 65.45 | 48.87 | 14.16 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 70.34 | 89.35 | 88.29 | 87.86 | 85.61 | 83.42 | 82.30 | 75.03 | 64.12 | 41.19 |  6.20 |
  mask | 67.08 | 81.22 | 79.09 | 78.97 | 78.18 | 77.28 | 76.29 | 71.96 | 64.42 | 49.93 | 13.49 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 69.38 | 86.42 | 86.42 | 85.20 | 83.94 | 81.91 | 80.07 | 76.30 | 66.11 | 40.34 |  7.05 |
  mask | 67.79 | 81.03 | 80.61 | 80.29 | 79.67 | 78.00 | 76.82 | 73.26 | 67.46 | 47.66 | 13.11 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 73.64 | 91.37 | 91.13 | 90.35 | 89.90 | 88.10 | 84.27 | 81.03 | 68.73 | 43.02 |  8.53 |
  mask | 67.72 | 80.57 | 80.05 | 79.44 | 78.99 | 78.53 | 76.75 | 73.24 | 66.26 | 50.64 | 12.72 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 72.92 | 92.05 | 91.76 | 88.09 | 87.57 | 86.42 | 85.28 | 80.28 | 64.42 | 44.74 |  8.56 |
  mask | 67.15 | 81.35 | 80.92 | 79.90 | 78.56 | 76.28 | 74.81 | 72.57 | 64.65 | 49.21 | 13.24 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 74.55 | 91.65 | 91.36 | 90.75 | 90.44 | 88.41 | 86.84 | 84.67 | 67.89 | 45.85 |  7.69 |
  mask | 67.81 | 80.91 | 80.45 | 80.07 | 79.30 | 78.67 | 77.01 | 73.25 | 66.93 | 47.52 | 14.03 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 73.94 | 91.58 | 91.04 | 90.53 | 89.29 | 88.49 | 86.74 | 82.15 | 66.17 | 45.53 |  7.91 |
  mask | 67.50 | 81.09 | 80.28 | 79.61 | 79.24 | 77.72 | 76.62 | 74.48 | 65.07 | 47.38 | 13.46 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 74.16 | 91.94 | 91.59 | 91.17 | 89.74 | 89.07 | 87.13 | 80.82 | 65.18 | 46.95 |  8.03 |
  mask | 68.10 | 81.16 | 80.56 | 80.00 | 79.79 | 79.29 | 76.93 | 73.42 | 66.01 | 50.03 | 13.79 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 73.08 | 92.31 | 92.04 | 91.41 | 90.18 | 89.18 | 85.48 | 73.48 | 64.46 | 44.68 |  7.58 |
  mask | 67.52 | 81.37 | 80.82 | 80.14 | 79.91 | 79.52 | 77.60 | 70.91 | 65.84 | 46.21 | 12.86 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+

1. Now, I would like to know based on, or what would be the best metric, to use to select the best weights? I am confused as sometimes the all has higher value while its mAP@0.5 is low, and other epochs has the opposite.
2. In the paper, what metric was used to select the best weight? mAP was reported, but was that mAP@0.5?
3. I could see that the mask metric surpasses the box metric in some case, and the apposite (with the mask mAP hanging around 80%), is this normal?

[xqh-code]

Hello,

I have used your repo to train a yolact++ model on my custom dataset. After the training finished, I found some weights saved. I collected the corresponding evaluation mAPs as follow:

       |  all  |  .50  |  .55  |  .60  |  .65  |  .70  |  .75  |  .80  |  .85  |  .90  |  .95  |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 61.44 | 80.53 | 80.22 | 79.55 | 76.49 | 74.82 | 72.89 | 66.13 | 51.65 | 29.44 |  2.70 |
  mask | 61.86 | 75.29 | 74.92 | 74.30 | 74.09 | 71.49 | 70.52 | 67.08 | 58.66 | 41.70 | 10.57 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 67.98 | 85.15 | 84.73 | 83.99 | 83.54 | 80.88 | 79.63 | 72.96 | 65.73 | 37.70 |  5.48 |
  mask | 67.06 | 79.99 | 79.54 | 79.39 | 78.23 | 77.86 | 74.65 | 72.50 | 65.45 | 48.87 | 14.16 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 70.34 | 89.35 | 88.29 | 87.86 | 85.61 | 83.42 | 82.30 | 75.03 | 64.12 | 41.19 |  6.20 |
  mask | 67.08 | 81.22 | 79.09 | 78.97 | 78.18 | 77.28 | 76.29 | 71.96 | 64.42 | 49.93 | 13.49 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 69.38 | 86.42 | 86.42 | 85.20 | 83.94 | 81.91 | 80.07 | 76.30 | 66.11 | 40.34 |  7.05 |
  mask | 67.79 | 81.03 | 80.61 | 80.29 | 79.67 | 78.00 | 76.82 | 73.26 | 67.46 | 47.66 | 13.11 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 73.64 | 91.37 | 91.13 | 90.35 | 89.90 | 88.10 | 84.27 | 81.03 | 68.73 | 43.02 |  8.53 |
  mask | 67.72 | 80.57 | 80.05 | 79.44 | 78.99 | 78.53 | 76.75 | 73.24 | 66.26 | 50.64 | 12.72 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 72.92 | 92.05 | 91.76 | 88.09 | 87.57 | 86.42 | 85.28 | 80.28 | 64.42 | 44.74 |  8.56 |
  mask | 67.15 | 81.35 | 80.92 | 79.90 | 78.56 | 76.28 | 74.81 | 72.57 | 64.65 | 49.21 | 13.24 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 74.55 | 91.65 | 91.36 | 90.75 | 90.44 | 88.41 | 86.84 | 84.67 | 67.89 | 45.85 |  7.69 |
  mask | 67.81 | 80.91 | 80.45 | 80.07 | 79.30 | 78.67 | 77.01 | 73.25 | 66.93 | 47.52 | 14.03 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 73.94 | 91.58 | 91.04 | 90.53 | 89.29 | 88.49 | 86.74 | 82.15 | 66.17 | 45.53 |  7.91 |
  mask | 67.50 | 81.09 | 80.28 | 79.61 | 79.24 | 77.72 | 76.62 | 74.48 | 65.07 | 47.38 | 13.46 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 74.16 | 91.94 | 91.59 | 91.17 | 89.74 | 89.07 | 87.13 | 80.82 | 65.18 | 46.95 |  8.03 |
  mask | 68.10 | 81.16 | 80.56 | 80.00 | 79.79 | 79.29 | 76.93 | 73.42 | 66.01 | 50.03 | 13.79 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
   box | 73.08 | 92.31 | 92.04 | 91.41 | 90.18 | 89.18 | 85.48 | 73.48 | 64.46 | 44.68 |  7.58 |
  mask | 67.52 | 81.37 | 80.82 | 80.14 | 79.91 | 79.52 | 77.60 | 70.91 | 65.84 | 46.21 | 12.86 |
-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+

1. Now, I would like to know based on, or what would be the best metric, to use to select the best weights? I am confused as sometimes the all has higher value while its mAP@0.5 is low, and other epochs has the opposite.
2. In the paper, what metric was used to select the best weight? mAP was reported, but was that mAP@0.5?
3. I could see that the mask metric surpasses the box metric in some case, and the apposite (with the mask mAP hanging around 80%), is this normal?

Hi, I also do my own dataset training, and the training results are similar to yours, and I would like to communicate with you about

[Jessica-hub]

Hi, How did you regenerate the evaluation mAPs?