The problem of reproducibility.

[huybery]

Hi,
Thank you for your excellent paper and open source code.
But I re-train the default configuration code directly, and did not reproduce excellent performance.

Here is my result:

                     easy                 medium               hard                 extra                all
count                248                  446                  174                  166                  1034
=====================   EXECUTION ACCURACY     =====================
execution            0.867                0.776                0.644                0.524                0.735

====================== EXACT MATCHING ACCURACY =====================
exact match          0.879                0.787                0.644                0.500                0.739

What did I miss? I've run three times and it's almost the nearly result.

[OhadRubin]

This is expected. SmBoP has a mean accuracy on the development set of 0.739365 with a std of 0.005093 (n=7).
Note that this is probably a consequence of using RATSQL, see here.

[huybery]

The instability of RAT training is really a tough problem.
Thank you again for your timely reply :)