[Feature Request] Word Error Rate

[jonathanmutal]

Hi everyone,

I was wondering if you were planning to implement Word Error Rate (WER). Even though WER is mostly used to test speech systems, It wouldn't be a bad idea to have a reproducible WER on sacreBLEU. I think it shouldn't take a lot of time after having implemented TER. I would be happy to help anyway.

Best regards,
Jonathan.

[martinpopel]

I'm not aware of anyone planning to add WER, but I agree it may be useful to have it implemented in sacreBLEU. With the new modular design it would not clutter other parts of the source code and WER is simple, so it does not need any extra dependencies, so I don't see any downsides.

I would be happy to help anyway.

Great. Pull requests are welcome. Don't forget to add some tests.

[scottfleming]

Is there a reason why you wouldn't use something like the editdistance package? Hard to beat the speed in C/C++. I don't mean this tongue-in-cheek, I recognize that there are plenty of legitimate reasons why you might want an implementation in sacreBLEU that does not require an external dependency. fairseq (from Meta) basically just uses a wrapper, but admittedly it seems like more work than one would want if you're just looking for an out-of-the-box WER scorer.

[ozancaglayan]

I fear that without that extra dep (which I previously used for WER in another project) the speed will be annoyingly slow. Also, TER could benefit from that package as well.