medBERTjp - SentencePiece v0.2

• Japanese Medical BERT model simultaneously pre-trained on both clinical references and Japanese Wikipedia via our method.
• Vocabulary: custom 32k vocabulary
  - requirements:
  - SentencePiece
  - character_coverage = 0.99995
  - model_type = unigram
  - normalization_rule_name = nmt_nfkc
  - add_dummy_prefix = false
  -remove_extra_whitespaces = true
• Pre-training:
  - BERT-Base (12-layer, 768-hidden, 12-heads)
  - trained from scratch.
  - max_seq_length = 128 tokens
  - global_batch_size = 2,048 sequences
  - learning_rate = 7e-4
  - warmup_proportion = 0.0125
  - training_steps = 200K steps
  - Use subword regularization (nbest_size=16, alpha(smoothing parameter)=0.1)

medBERTjp - MeCab-Unidic-2.3.0

• Japanese Medical BERT model simultaneously pre-trained on both clinical references and Japanese Wikipedia via our method.
• Vocabulary: custom 32k vocabulary
  - requirements:
  - fugashi
  - unidic-py
• Pre-training:
  - BERT-Base (12-layer, 768-hidden, 12-heads)
  - trained from scratch with WWM.
  - max_seq_length = 128 tokens
  - global_batch_size = 2,048 sequences
  - learning_rate = 7e-4
  - warmup_proportion = 0.0125
  - training_steps = 125K steps

medBERTjp - SentencePiece

• Japanese Medical BERT model simultaneously pre-trained on both clinical references and Japanese Wikipedia via our method.
• Vocabulary: custom 32k vocabulary
  - requirements:
  - SentencePiece
  - character_coverage = 0.99995
  - model_type = unigram
• Pre-training:
  - BERT-Base (12-layer, 768-hidden, 12-heads)
  - trained from scratch.
  - max_seq_length = 128 tokens
  - global_batch_size = 2,048 sequences
  - learning_rate = 7e-4
  - warmup_proportion = 0.0125
  - training_steps = 125K steps

medBERTjp - MeCab-IPAdic-NEologd-JMeDic

• Japanese Medical BERT model simultaneously pre-trained on both clinical references and Japanese Wikipedia via our method.
• Vocabulary: custom 32k vocabulary
  - requirements:
  - fugashi
  - mecab-ipadic-NEologd
  - J-MeDic
• Pre-training:
  - BERT-Base (12-layer, 768-hidden, 12-heads)
  - trained from scratch with WWM.
  - max_seq_length = 128 tokens
  - global_batch_size = 2,048 sequences
  - learning_rate = 7e-4
  - warmup_proportion = 0.0125
  - training_steps = 125K steps

medBERTjp - MeCab-IPAdic

• Japanese Medical BERT model simultaneously pre-trained on both clinical references and Japanese Wikipedia via our method.
• Vocabulary: custom 32k vocabulary
  - requirements:
  - fugashi
  - ipadic-py
• Pre-training:
  - BERT-Base (12-layer, 768-hidden, 12-heads)
  - trained from scratch with WWM.
  - max_seq_length = 128 tokens
  - global_batch_size = 2,048 sequences
  - learning_rate = 7e-4
  - warmup_proportion = 0.0125
  - training_steps = 125K steps