Code for test suites

[samsontmr]

Hi! Will you be releasing the code for running the test suites?

[hans]

Hi, we're currently preparing the full pipeline to reproduce these analyses, and it should be complete within a month or so. In the meantime, if you're interested, we can share the raw test suite data — please let me know if so.

[samsontmr]

I see, would you be able to share the code for running the suites on GPT-2 as well? Otherwise I can wait for the full pipeline. Thanks!

[jakpra]

Hi! Any update on this?

All I can see in the notebooks and scripts is that result CSVs are being loaded, but not how the results were computed from one of the testsuite formats.
Most crucially I'm looking for a reader/preprocessor for the JSON format, which seems to contain the richest information. Though then again, reimplementing it from the detailed description in the paper doesn't sound too bad either. I'm just lazy :D
Maybe I'm just missing it?