How to evaluate?

[tongxuluo]

We tried to pretrain a 421M Samba, but after pretraining, we find that you did not open source the evaluation script.

[richardburleigh]

I've hacked together Huggingface Transformers support for a model I'm training at the moment, which allows Samba to work with EleutherAI/lm-evaluation-harness

It's too messy to share at the moment, but if you're comfortable posting your model I can run whichever evals you'd like. Otherwise hopefully the Samba team will get the approval they deserve and can release the official Transformers support for this amazing architecture.

[tongxuluo]

I've hacked together Huggingface Transformers support for a model I'm training at the moment, which allows Samba to work with EleutherAI/lm-evaluation-harness

It's too messy to share at the moment, but if you're comfortable posting your model I can run whichever evals you'd like. Otherwise hopefully the Samba team will get the approval they deserve and can release the official Transformers support for this amazing architecture.

Wow, thank you very much! We have already uploaded the checkpoints on Huggingface: https://huggingface.co/wydu/Samba . We would greatly appreciate it if you could help us evaluate the checkpoints Samba-iter-200000-ckpt.pth and Samba2-iter-188000-ckpt.pth on the following tasks: lambada_openai, wikitext, piqa, sciq, winogrande, arc_challenge, arc_easy, and logiqa.

[richardburleigh]

Here you go - nice work!
Samba-iter-200000-ckpt.pth

|    Tasks     |Version|Filter|n-shot|    Metric     |   | Value |   |Stderr|
|--------------|------:|------|-----:|---------------|---|------:|---|------|
|arc_challenge |      1|none  |     0|acc            |↑  | 0.2304|±  |0.0123|
|              |       |none  |     0|acc_norm       |↑  | 0.2534|±  |0.0127|
|arc_easy      |      1|none  |     0|acc            |↑  | 0.5004|±  |0.0103|
|              |       |none  |     0|acc_norm       |↑  | 0.4377|±  |0.0102|
|lambada_openai|      1|none  |     0|acc            |↑  | 0.3641|±  |0.0067|
|              |       |none  |     0|perplexity     |↓  |32.1942|±  |1.3605|
|logiqa        |      1|none  |     0|acc            |↑  | 0.2243|±  |0.0164|
|              |       |none  |     0|acc_norm       |↑  | 0.2750|±  |0.0175|
|piqa          |      1|none  |     0|acc            |↑  | 0.6866|±  |0.0108|
|              |       |none  |     0|acc_norm       |↑  | 0.6736|±  |0.0109|
|sciq          |      1|none  |     0|acc            |↑  | 0.7870|±  |0.0130|
|              |       |none  |     0|acc_norm       |↑  | 0.7000|±  |0.0145|
|wikitext      |      2|none  |     0|bits_per_byte  |↓  | 0.9050|±  |   N/A|
|              |       |none  |     0|byte_perplexity|↓  | 1.8725|±  |   N/A|
|              |       |none  |     0|word_perplexity|↓  |28.6249|±  |   N/A|
|winogrande    |      1|none  |     0|acc            |↑  | 0.5422|±  |0.0140|

Samba2-iter-188000-ckpt.pth

|    Tasks     |Version|Filter|n-shot|    Metric     |   | Value |   |Stderr|
|--------------|------:|------|-----:|---------------|---|------:|---|------|
|arc_challenge |      1|none  |     0|acc            |↑  | 0.2133|±  |0.0120|
|              |       |none  |     0|acc_norm       |↑  | 0.2619|±  |0.0128|
|arc_easy      |      1|none  |     0|acc            |↑  | 0.5059|±  |0.0103|
|              |       |none  |     0|acc_norm       |↑  | 0.4495|±  |0.0102|
|lambada_openai|      1|none  |     0|acc            |↑  | 0.3844|±  |0.0068|
|              |       |none  |     0|perplexity     |↓  |29.8475|±  |1.3024|
|logiqa        |      1|none  |     0|acc            |↑  | 0.2166|±  |0.0162|
|              |       |none  |     0|acc_norm       |↑  | 0.2688|±  |0.0174|
|piqa          |      1|none  |     0|acc            |↑  | 0.6725|±  |0.0109|
|              |       |none  |     0|acc_norm       |↑  | 0.6795|±  |0.0109|
|sciq          |      1|none  |     0|acc            |↑  | 0.8040|±  |0.0126|
|              |       |none  |     0|acc_norm       |↑  | 0.7280|±  |0.0141|
|wikitext      |      2|none  |     0|bits_per_byte  |↓  | 0.9107|±  |   N/A|
|              |       |none  |     0|byte_perplexity|↓  | 1.8799|±  |   N/A|
|              |       |none  |     0|word_perplexity|↓  |29.2371|±  |   N/A|
|winogrande    |      1|none  |     0|acc            |↑  | 0.5091|±  |0.0141|