Comparing results on EfficientNetV2_b0

[leondgarse]

EfficientnetV2 B0 using SGD + L2 regularizer + cosine lr decay training on MS1MV3 dataset

• Training script

  • EfficientnetV2 from my another project keras_cv_attention_models.

    pip install keras-cv-attention-models

  • Setting random_status=100 will use randaug with magnitude=5. The default available_ops for randaug contains only color related ones and cutout. Randaug methods like shear / tansform are removed, as they are not good for face data.

  import losses, train, models
  import tensorflow_addons as tfa
  keras.mixed_precision.set_global_policy("mixed_float16")
  
  data_basic_path = '/datasets/ms1m-retinaface-t1'
  data_path = data_basic_path + '_112x112_folders'
  eval_paths = [os.path.join(data_basic_path, ii) for ii in ['lfw.bin', 'cfp_fp.bin', 'agedb_30.bin']]
  
  from keras_cv_attention_models import efficientnet
  basic_model = efficientnet.EfficientNetV2B0(input_shape=(112, 112, 3), num_classes=0)
  basic_model = models.buildin_models(basic_model, dropout=0, emb_shape=512, output_layer='GDC', bn_epsilon=1e-4, bn_momentum=0.9, scale=True, use_bias=False)
  basic_model = models.add_l2_regularizer_2_model(basic_model, weight_decay=5e-4, apply_to_batch_normal=False)
  # basic_model.save('checkpoints/efficientnetv2-b0-imagenet-basic.h5')
  # basic_model = keras.models.load_model('checkpoints/efficientnetv2-b0-imagenet-basic.h5')
  
  tt = train.Train(data_path, eval_paths=eval_paths,
      save_path='TT_efv2_b0_swish_GDC_arc_emb512_dr0_sgd_l2_5e4_bs512_ms1m_randaug_cutout_bnm09_bne1e4_cos16_batch_float16.h5',
      basic_model=basic_model, model=None, lr_base=0.1, lr_decay=0.5, lr_decay_steps=16, lr_min=1e-4,
      batch_size=512, random_status=100, output_weight_decay=1)
  
  optimizer = keras.optimizers.SGD(learning_rate=0.1, momentum=0.9)
  sch = [
      {"loss": losses.ArcfaceLoss(scale=16), "epoch": 5, "optimizer": optimizer},
      {"loss": losses.ArcfaceLoss(scale=32), "epoch": 5},
      {"loss": losses.ArcfaceLoss(scale=64), "epoch": 40},
  ]
  tt.train(sch, 0)

• Comparing different data_augment / optimizer / loss / batch_size
  

  Method	lfw	cfp_fp	agedb_30	IJBB	IJBC
  arc, random 0, bs512	0.997167	0.975429	0.975	0.937098	0.951935
  arc, randaug_cutout, bs512	0.997667	0.974857	0.976167	0.938267	0.953623
  cosface, randaug_cutout, bs512	0.997167	0.978571	0.975833	0.937196
  arc, randaug_cutout, bs1024, lrbase 0.2	0.997667	0.977429	0.977833	0.933398	0.951066
  arc, randaug_cutout, bs512, SGD lookahead	0.997500	0.977571	0.977000	0.936806
  arc, randaug_cutout, bs512, SGD nesterov=True	0.998000	0.977857	0.976000	0.938267	0.953418

• arc, randaug_cutout, bs512 IJBB / IJBC result detail

  EFV2 B0 MS1MV3	1e-06	1e-05	0.0001	0.001	0.01	0.1	AUC
  IJBB_11	0.389094	0.868939	0.938267	0.959786	0.974294	0.984031	0.992253
  IJBC_11	0.875339	0.925858	0.953623	0.970394	0.982001	0.989262	0.994469

[leondgarse]

Go on training for another 17 epochs with lr_base=0.025

• Training script

  • SGD + Lookahead performs better this time.

  import losses, train, models
  import tensorflow_addons as tfa
  keras.mixed_precision.set_global_policy("mixed_float16")
  
  data_basic_path = '/datasets/ms1m-retinaface-t1'
  data_path = data_basic_path + '_112x112_folders'
  eval_paths = ["IJBB.bin", "IJBC.bin"] + [os.path.join(data_basic_path, ii) for ii in ['lfw.bin', 'cfp_fp.bin', 'agedb_30.bin']]
  
  # SAM
  # model = keras.models.load_model("checkpoints/TT_efv2_b0_swish_GDC_arc_emb512_dr0_sgd_l2_5e4_bs512_ms1m_randaug_cutout_bnm09_bne1e4_cos16_batch_float16.h5", custom_objects={"NormDense": models.NormDense, "ArcfaceLoss": losses.ArcfaceLoss})
  # model = models.SAMModel(model.inputs[0], model.outputs[0], rho=0.05, name=model.name)
  
  tt = train.Train(data_path, eval_paths=eval_paths,
      save_path='TT_efv2_b0_swish_GDC_arc_emb512_dr0_sgd_l2_5e4_bs512_ms1m_randaug_cutout_bnm09_bne1e4_cos16_batch_float16_E50_arc_sgd_LA.h5',
      basic_model=None, model='checkpoints/TT_efv2_b0_swish_GDC_arc_emb512_dr0_sgd_l2_5e4_bs512_ms1m_randaug_cutout_bnm09_bne1e4_cos16_batch_float16.h5',
      lr_base=0.025, lr_decay=0.5, lr_decay_steps=16, lr_min=1e-4,
      batch_size=512, random_status=100, output_weight_decay=1)
  
  optimizer = tfa.optimizers.Lookahead(keras.optimizers.SGD(learning_rate=0.1, momentum=0.9))
  tt.train({"loss": losses.ArcfaceLoss(scale=64), "epoch": 17, "optimizer": optimizer}, 0)

• Comparing different optimizer and other methods
  

  Method	lfw	cfp_fp	agedb_30	IJBB	IJBC
  randaug, lr_base 0.025	0.997500	0.975286	0.977000	0.939143	0.95439
  randaug, lr_base 1e-3	0.997500	0.975571	0.977667	0.937098
  randaug, lr_base 0.025, SGD lookahead	0.997833	0.976571	0.977333	0.940701	0.955259
  randaug, lr_base 0.025, SGD lookahead, SAM	0.997500	0.976000	0.976000	0.939143	0.954134

• randaug, lr_base 0.025, SGD lookahead IJBB / IJBC result detail

  EFV2 B0 SGD LA	1e-06	1e-05	0.0001	0.001	0.01	0.1	AUC
  IJBB_11	0.375755	0.872639	0.940701	0.959202	0.974781	0.984226	0.991931
  IJBC_11	0.870174	0.926471	0.955259	0.970394	0.982257	0.989058	0.994494