Respect tokenizer.ggml.add_bos_token value when tokenizing #6384
| Job | Run time |
|---|---|
| 1m 45s | |
| 2m 56s | |
| 1m 44s | |
| 2m 43s | |
| 2m 49s | |
| 1m 21s | |
| 3m 40s | |
| 1m 27s | |
| 2m 57s | |
| 1m 24s | |
| 2m 18s | |
| 3m 26s | |
| 3m 10s | |
| 20m 42s | |
| 13m 39s | |
| 14m 16s | |
| 3m 16s | |
| 4m 22s | |
| 2m 53s | |
| 3m 7s | |
| 3m 24s | |
| 4m 41s | |
| 4m 55s | |
| 3m 51s | |
| 2m 43s | |
| 0s | |
| 1h 53m 29s |