Skip to content

medBERTjp - SentencePiece
Compare
Choose a tag to compare
@sy-wada sy-wada released this 23 Oct 09:33
· 29 commits to main since this release
v0.1-sp
15631bb
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
  • Japanese Medical BERT model simultaneously pre-trained on both clinical references and Japanese Wikipedia via our method.
  • Vocabulary: custom 32k vocabulary
    - requirements:
    - SentencePiece
    - character_coverage = 0.99995
    - model_type = unigram
  • Pre-training:
    - BERT-Base (12-layer, 768-hidden, 12-heads)
    - trained from scratch.
    - max_seq_length = 128 tokens
    - global_batch_size = 2,048 sequences
    - learning_rate = 7e-4
    - warmup_proportion = 0.0125
    - training_steps = 125K steps
Assets 3
Loading