Finetuning scheduler (#115)

* notebook-based fts tutorial and example
* additional tuning literature references and several additional recommended improvements for finetuning_scheduler tutorial
* numerous expository enhancements and switch to DeBERTav3
* enhance linkcheck configuration and switch two images to be base64 encoded
* enabled setting base learning rates on a per-phase basis, updating current finetuning_scheduler_module release SHA
* clarify finetuning schedule definition configuration options and update fts module requirement ref
* updated tutorial with initial version of finetuning-scheduler dependency
* update finetuning-scheduler commit sha, simplify notebook dependency with finetuning-scheduler[examples] extra, accommodate acceleratorconnector refactor
* sync finetuning_scheduler tutorial for pl 1.6.0rc1
* update commit sha to point to fts with official pl 1.6.0 release
* require only finetuning-scheduler dependency now that it is available on pypi
* update fts documentation to point to stable version (0.1.1)
* implement workaround for cuda toolkit build difference of torch and torchvision in CI that was resulting in a segfault triggered by sentencepiece upon pytorch_lightning import
* isort skip directive required to implement sentencepiece workaround
* metadata and aesthetic image improvements
* make notebook accelerator agnostic
* reset lr_scheduler interval to epoch
* Apply suggestions from code review

Co-authored-by: thomas chaton <thomas@grid.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>

Author: 4 people

docs/source/conf.py

@@ -181,6 +181,11 @@
 # (source start file, name, description, authors, manual section).
 man_pages = [(master_doc, project, project + " Documentation", [author], 1)]
 
+# -- Options for linkcheck builder ----------------------------------------------
+# regex pattern 0: allow linking to a specific selection state in
+#  tensorboard.dev links while continuing to validate the base experiment link
+linkcheck_anchors_ignore = ["scalars.*&runSelectionState.*"]
+
 # -- Options for Texinfo output ----------------------------------------------
 
 # Grouping the document tree into Texinfo files. List of tuples

lightning_examples/finetuning-scheduler/.meta.yml

@@ -0,0 +1,19 @@
+title: Finetuning Scheduler
+author: "[Dan Dale](https://github.com/speediedan)"
+created: 2021-11-29
+updated: 2022-05-10
+license: CC BY-SA
+build: 3
+tags:
+  - finetuning
+description: |
+  This notebook introduces the [Finetuning Scheduler](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) extension
+  and demonstrates the use of it to finetune a small foundational model on the
+  [RTE](https://huggingface.co/datasets/viewer/?dataset=super_glue&config=rte) task of
+  [SuperGLUE](https://super.gluebenchmark.com/) with iterative early-stopping defined according to a user-specified
+  schedule. It uses Hugging Face's ``datasets`` and ``transformers`` libraries to retrieve the relevant benchmark data
+  and foundational model weights. The required dependencies are installed via the finetuning-scheduler ``[examples]`` extra.
+requirements:
+  - finetuning-scheduler[examples]
+accelerator:
+  - GPU

lightning_examples/finetuning-scheduler/RteBoolqModule_ft_schedule_deberta_base.yaml

@@ -0,0 +1,18 @@
+
+0:
+  params:
+  - model.classifier.bias
+  - model.classifier.weight
+  - model.pooler.dense.bias
+  - model.pooler.dense.weight
+  - model.deberta.encoder.LayerNorm.bias
+  - model.deberta.encoder.LayerNorm.weight
+  - model.deberta.encoder.rel_embeddings.weight
+  - model.deberta.encoder.layer.{0,11}.(output|attention|intermediate).*
+1:
+  params:
+  - model.deberta.embeddings.LayerNorm.bias
+  - model.deberta.embeddings.LayerNorm.weight
+2:
+  params:
+  - model.deberta.embeddings.word_embeddings.weight