Skip to content

Stockfish 10: Crazyhouse Self Play

Jump to bottom
QueensGambit edited this page Jun 3, 2019 · 28 revisions

Stockfish 10: Crazyhouse Self Play Dataset

UPDATE 14.01.2019:

The event for creating the Stockfish Crazyhouse dataset is over: Strength Evaluation

Thanks to crazyhorse, ObiWanBenoni, Pichau, varvarakh and queensgambit for helping generating:

121571  games total
White-Black-Draw: 80303-38796-2472
White-Black-Draw: 66.05% 31.91% 2.03%

The full dataset is available here (licensed under GPLV3):

The dataset was split in a train, validation and test set:

119571 training samples - 1000 validation samples - 1000 test samples

Afterwards the pre-trained human based CNN was used for learning from Stockfish. Before training it achieved 46.29% validation move accuracy. After 11 hours of training using a GTX 1080ti it scored 56.6%. More metrics can be found in the train log:

The weights can be downloaded here:

(You need to put the weight files in CrazyAra_X.X.X/model and remove or move the old weights somewhere else)

Goal

The goal is to create a dataset from stockfish 10 playing against itself with at least 250k crazyhouse games and at maximum 1 million crazyhouse game. The starting positions are based on one thousand unique opening positions in the lichess.org opening explorer.

For each game position stockfish will use 1 million +/- 100k nodes with a hash size of 512mb.

Motivation

  • Having an engine based perspective on human based opening helps developing crazyhouse opening theory.
  • It can help to discover stockfish's weaknesses which can later be fixed.
  • It can be used to train a neural network which tries to mimic 1 million nodes Stockfish play using only 1 node.

Components:

For running stockfish in self play mode.

1. cutechess-cli (Cute Chess Command Line Interface):

Download Cute Chess 1.0.0.

sudo apt-get install cutechess_20170720.1.0.0.1.0.0-1_amd64.deb

The cutechess-cli should be added to the environment path variable by default after installing cutechess. This way you can run cutechess-cli from any location. If you built cutechess-cli manually it should be located at cutechess/projects/cli/cutechess-cli.

2. Multi-variant stockfish 10:

Download the latest release variant_sf_10

3. Opening Suite

Download 1k_cz_lichess_startpos.pgn.

This opening book is based on the neural network weights which was trained on the lichess.org database. Consequently the opening book does correspond to the most popular human crazyhouse openings. The file 1k_cz_lichess_startpos.pgn features 1,000 unique opening positions where each opening has 5.84 +/- 2.77 plys. One ply is a half-move in chess notation.

4. Game Generator Script

Download sf_cz_game_generator.py.

Make sure you have python installed on your system.

Start the script on your command line with

python sf_cz_game_generator.py --sf_path SF_PATH --opening_book_path OPENING_BOOK_PATH

and set the paths accordingly. Give the full path for the excecutable file including .exe on windows.

(If python has trouble parsing a path which contain spaces you can try putting the path in " or relocate the executable somewhere else.)

If you put 1k_cz_lichess_startpos.pgn in the same directory as sf_cz_game_generator.py then you don't have to set OPENING_BOOK_PATH.

The games will be exported to a file sf_vs_sf_USERNAME.pgn in your local directory.

You can change the name of the file via --pgnout_path PGNOUT_PATH.

You can stop the generation any time using ctrl+c/command+c/closing terminal and continue with the same command as above. New games will be appended to the pgn-file.

Speed: Using a Intel® Core™ i5-8250U CPU @ 1.60GHz × 8 every minute 4 new games are generated. The generator script will call the cutechess-cli excecutable and will define the settings for the matches, so you don't have to start the cutechess-cli in parallel.

$ python sf_cz_game_generator.py -h

usage: sf_cz_game_generator.py [-h] [--cutechess_cli_path CUTECHESS_CLI_PATH]
                               [--sf_path SF_PATH]
                               [--opening_book_path OPENING_BOOK_PATH]
                               [--pgnout_path PGNOUT_PATH] [--threads THREADS]
                               [--hash HASH] [--variant VARIANT]

Sf self play game generator v1.0

optional arguments:
  -h, --help            show this help message and exit
  --cutechess_cli_path CUTECHESS_CLI_PATH
                        cutechess executable path (default: cutechess-cli)
  --sf_path SF_PATH     cutechess executable path (default: /home/queensgambit
                        /Programs/crazyhouse_engines/stockfish/stockfish-x86_6
                        4-modern)
  --opening_book_path OPENING_BOOK_PATH
                        opening book path (default: 1k_cz_lichess_startpos.pgn
  --pgnout_path PGNOUT_PATH
                        filepath where the games will be stored (default:
                        sf_vs_sf_queensgambit.pgn)
  --threads THREADS     number of threads for generating games (default: 7
                        number of cores-1 detected by python)
  --hash HASH           hash size in mb (default: 512)
  --variant VARIANT     define the chess variant (default:
                        crazyhouse)'3check': Three-check Chess '5check': Five-
                        check Chess 'atomic': Atomic Chess 'berolina':
                        Berolina Chess 'capablanca': Capablanca Chess
                        'caparandom': Capablanca Random Chess 'checkless':
                        Checkless Chess 'chessgi': Chessgi (Drop Chess)
                        'crazyhouse': Crazyhouse (Drop Chess) 'extinction':
                        Extinction Chess 'fischerandom': Fischer Random
                        Chess/Chess 960 'gothic': Gothic Chess 'horde': Horde
                        Chess (v2) 'janus': Janus Chess 'kinglet': Kinglet
                        Chess 'kingofthehill': King of the Hill Chess 'loop':
                        Loop Chess (Drop Chess) 'losers': Loser's Chess
                        'racingkings': Racing Kings Chess 'standard': Standard
                        Chess.

Upload:

Upload the games to the full dataset by uploading the pgn-file to google drive.

The pgn-files will be concatenated to one big dataset on:

Saturday, 5 January 2019, 20:00:00 (UTC time).

UPDATE: On 5 January 2019: 49172 games have been generated by 3 people

  • queensgambit (18458)
  • crazyhorse (16682)
  • picachu (14032)

The deadline for creating the dataset was extended to Saturday, 12 January 2019, 20:00:00 (UTC time).

Appendix:

NPS check

You can check if the game generation is working properly by creating a 1 min crazyhouse using cutechess-gui.

After launching cutechess-gui add stockfish to the engine list:

Go to Tools->Settings->Engines->+->

Tab: Basic

Name: stockfish-x86_64-modern Command: ./stockfish-x86_64-modern Working Directory: /home/queensgambit/Programs/crazyhouse_engines/stockfish

Now give proper settings in Tab: Advanced:

  • Threads=7 (depending on your system)
  • Hash=512

Start a new game Game->New-> and select CPU: stockfish-x86_64-modern, Variant: crazyhouse, Time Control: 40 moves in 1min. During the game you should see NPS statistics as well as the reached depth.

Clone this wiki locally