add to_static for electra

[MayYouBeProsperous]

PR types

Others

PR changes

Others

Description

• 【Hackathon 5th No.64】PaddleNLP套件模型接入动转静训练功能 community#751

为 electra 模型接入动转静

训练环境

cd model_zoo/electra
export CUDA_VISIBLE_DEVICES="0"
export DATA_DIR=./BookCorpus/

训练结果

• 动态图

  运行命令

  python -u ./run_pretrain.py \
      --model_type electra \
      --model_name_or_path electra-small \
      --input_dir $DATA_DIR \
      --output_dir ./pretrain_model/ \
      --train_batch_size 8 \
      --learning_rate 5e-4 \
      --max_seq_length 128 \
      --weight_decay 1e-2 \
      --adam_epsilon 1e-6 \
      --warmup_steps 10000 \
      --num_train_epochs 4 \
      --logging_steps 100 \
      --save_steps 10000 \
      --max_steps -1 \
      --device gpu
  

  运行结果
  

• 静态图 SOT模式
  运行命令

  export GRAPH_SIZE=0 COST_MODEL=False ENABLE_FALL_BACK=True
  python -u ./run_pretrain.py \
      --model_type electra \
      --model_name_or_path electra-small \
      --input_dir $DATA_DIR \
      --output_dir ./pretrain_model/ \
      --train_batch_size 8 \
      --learning_rate 5e-4 \
      --max_seq_length 128 \
      --weight_decay 1e-2 \
      --adam_epsilon 1e-6 \
      --warmup_steps 10000 \
      --num_train_epochs 4 \
      --logging_steps 100 \
      --save_steps 10000 \
      --max_steps -1 \
      --device gpu \
      --to_static True

  运行结果
  

• 静态图 AST模式
  运行命令

  export COST_MODEL=False ENABLE_FALL_BACK=False
  python -u ./run_pretrain.py \
      --model_type electra \
      --model_name_or_path electra-small \
      --input_dir $DATA_DIR \
      --output_dir ./pretrain_model/ \
      --train_batch_size 8 \
      --learning_rate 5e-4 \
      --max_seq_length 128 \
      --weight_decay 1e-2 \
      --adam_epsilon 1e-6 \
      --warmup_steps 10000 \
      --num_train_epochs 4 \
      --logging_steps 100 \
      --save_steps 10000 \
      --max_steps -1 \
      --device gpu \
      --to_static True

  运行结果
  

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Attention: 154 lines in your changes are missing coverage. Please review.

Comparison is base (10ff94a) 58.23% compared to head (ded8a2d) 57.85%.
Report is 61 commits behind head on develop.

❗ Current head ded8a2d differs from pull request most recent head 434d280. Consider uploading reports for the commit 434d280 to get more accurate results

Files	Patch %	Lines
paddlenlp/trainer/training_args.py	3.48%	83 Missing ⚠️
paddlenlp/transformers/llama/modeling_auto.py	8.97%	71 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #7575      +/-   ##
===========================================
- Coverage    58.23%   57.85%   -0.38%     
===========================================
  Files          579      582       +3     
  Lines        85819    86480     +661     
===========================================
+ Hits         49973    50032      +59     
- Misses       35846    36448     +602     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[2742195759]

这里只有 electra 一个模型支持了动转静吗？还是说这个是所有模型的训练入口？

[MayYouBeProsperous]

这里只有 electra 一个模型支持了动转静吗？还是说这个是所有模型的训练入口？

这里只有 electra 的，打算一个模型提交一个 PR ，这样可以吗？

[2742195759]

可以的

[2742195759]

LGTM

[w5688414]

LGTM