Feat: Add Completion dataset type

[NanoCode012]

Proposal

Train a completion model like the base llama before fine-tuning on a specific domain. I am testing this method for fine-tuning completion model on another low-resource language (to give it knowledge about the language) before fine-tuning using instruct types (into chat).

Usage

datasets:
  - path: data/text.jsonl
    type: completion

Data format:

Row of {"text": "..."}

Misc

• I tried to write less repetitive code while following your class format. This code can definitely be shorter. Some functions just return the input
• I did not apply formatter like black

Results so far

On a 10k dataset, it does not perform quite well. It does manage to use answer in correct language, but nothing special. My training also failed several times, so I think the result is not conclusive yet.

[NanoCode012]

@winglian , I'm wondering if this PR will be ok? I use a similar one for my fine-tuning.

[winglian]

@NanoCode012 yup, this looks good to me. One thing I want to do down the line is to convert the prompters and tokenizers into a plugin type system where each one registers their own namespace for the dataset transform step. That way, only new classes need to be added and ideally no clunky conditionals to map them would be needed.

[NanoCode012]

Yes agreed. I was initially also going to move the _tokenizer function to the parent as all of them were using the same one. But, it would've been better on a separate PR.

Your idea sounds better though!