Documentation for configuring stockfish to play at target ELO

[siemanko]

I am not sure if this is the best place to ask this, let me know if I should post this somewhere else. I'd like to configure stockfish 11 to play at a specific ELO rating. Based on everything I read online, I arrived at the following configuration:

Environment:

git tag: sf_11
Make command: `make build ARCH=x86-64-modern`
Processor: 2.4 GHz 8-Core Intel Core i9

Stockfish:

Threads: 1
Hash: 128
UCI_LimitStrength: true
UCI_Elo: 1350
movetime: 60s (no ponder, no infinite)

However the AI seems to be playing well below 1350 ELO (maybe ~1000). Is there something I am missing? I'd hate to be freeloading, so let me know if I can help improve documentation on that. (cc @vondele who added UCI_elo)

[vondele]

that's presumably the right settings (IIRC it was calibrated with 60s games, not 60s per move, but that would be even less strong). Are you sure movetime is 60s and not 60ms (go movetime 60 is 60ms)?

[siemanko]

Thanks for reply! Yes, I have used 60s movetime, I watched the games live and it definitely spent 60 seconds between moves. I also double-checked the code sets the right limit just now. The only caveat is that I used python-chess bindings, so maybe there's something mistake in my use of those bindings or the bindings themselves. Although they do look like pretty straightforward wrapper for sending UCI commands.

Do you have any advice on how to debug this? My ideas so far:

• compare to a black box reference AI with known fixed ELO (do you have any recommendations?) and tweak options until stockfish has 50% winrate.
• recompile stockfish to save all UCI commands to file to make sure what python-chess sends is what I'd expect.

For reference this is the game I got running at the settings mentioned above, I was told this is pretty obviously below 1350 ELO:

[Site "virtual"]
[Date "2020.07.01"]
[Round "?"]
[White "player-1"]
[Black "stockfish-1350"]
[Result "1-0"]
1. e4 d6 2. Nf3 Nf6 3. Nc3 c5 4. d4 Nc6 5. d5 Nb4 6. a3 Na6 7. Bb5+ Nd7 8. O-O
Nb8 9. Be3 h5 10. Qe2 a6 11. Bd3 b5 12. Rad1 c4 13. Bd4 e6 14. dxe6 fxe6 15. e5
Rh6 16. Be4 Nxe5 17. Bxe5 d5 18. Nxd5 exd5 19. Rxd5 Nd7 20. Re1 Re6 21. Bd4 Bb7
22. Re5 Nxe5 23. Bxe5 Bxe4 24. Qxe4 Rc8 25. h4 c3 26. bxc3 Qa5 27. Nd4 Re7 28.
Nc6 Qb6 29. Nxe7 Rd8 30. Nf5 Bc5 31. Nxg7+ Kf7 32. Qf5+ Qf6 33. Qxf6+ Kg8 34.
Ne8 Bxf2+ 35. Kxf2 Rd2+ 36. Ke3 Rd7 37. Qg5+ Kf7 38. Qg7+ Kxe8 39. Qg8+ Ke7 40.
Qg5+ Ke6 41. Bd4 Rd5 42. Qg6+ Ke7 43. Qg7+ Kd6 44. Ke4 Kc6 45. Qg8 Rd7 46. Qg6+
Kc7 47. Kd3 Rxd4+ 48. cxd4 b4 49. Qg7+ Kd8 50. Qg8+ Kd7 51. d5 bxa3 52. Qf7+ Kd8
53. Re8# 1-0 

[vondele]

it is almost impossible to judge Elo level from a single game, and the caveat is this is anchored to CCRL (see 0dbc72d) , which is not necessarily FIDE Elo. So to debug this, first have a look if indeed we're way off the scale, i.e. play a match against a CCRL engine that is ~1350 and see what the score is we get. You might find an engine in that range http://www.computerchess.org.uk/ccrl/404/ that still compiles (be sure to pick the same version as has been measured).

[siemanko]

Gotcha, ok I will run the comparisons and report the findings. Btw. I also noticed you wrote in that PR you used 8moves_v3.pgn, what does that mean? Is this some important configuration option?

[vondele]

@siemanko that's a particular testing book for the openings. The precise Elo number will depend a bit on that... hard to guess how much. Books can be found here : https://github.com/official-stockfish/books

[vondele]

@siemanko I'll close this issue, if you have further questions or data, just post it here, and in case reopen the issue.

[sgrvinod]

@vondele I apologize if this is the wrong place to ask this question, but if SF's Elo calibration was anchored to CCRL, does this mean that the time control used during calibration (2'+1" for SF16) was on their "standard" CPU (the Intel i7-4770K)?

[vondele]

no, it was having the reference engine play at the same time as SF, but anchor that reference engine to its CCRL rating.

[Disservin]

#4341 (comment)

This has been computed on the 8moves_v3.pgn with 120+1TC, so high draw rate and small Elo differences to be expected at the highest level.

btw here's the commit for the updated calibration

[sgrvinod]

Thank you!