Test on Optuna v2.9.0/2.9.1

[fsmosca]

New releases:
https://github.com/optuna/optuna/releases/tag/v2.9.0

There are refactoring and breaking changes in TPE sampler. This sampler needs to be tested before supporting this new version into the tuner.

• TPE
• TPE, multivariate=True

[fsmosca]

Sampler: TPE

Unoptimize some piece values and then try to optimize it with this sampler. Verify the optimized values by running an engine vs engine match between the unoptimized and optimized piece values.

Optimization Command Line

python tuner.py --engine ./engines/deuterium/deuterium.exe ^
--sampler name=tpe ^
--threshold-pruner result=0.45 ^
--opening-file ./start_opening/ogpt_chess_startpos.epd --opening-format epd ^
--concurrency 6 ^
--base-time-sec 2 --inc-time-sec 0.05 ^
--trials 100 --games-per-trial 100 ^
--study-name opt_tpe_piecevalue_optunav2.9 ^
--resign-movecount 3 --resign-score 400 ^
--draw-movenumber 30 --draw-movecount 6 --draw-score 0 ^
--input-param "{'BishopValueOp': {'default': 250, 'min': 200, 'max': 400, 'step': 2}, 'RookValueEn': {'default': 400, 'min': 300, 'max': 700, 'step': 4}, 'QueenValueOp': {'default': 800, 'min': 600, 'max': 1300, 'step': 4}}"

Best param found by TPE

option.BishopValueOp=304 option.QueenValueOp=1232 option.RookValueEn=632

Unoptimized param

option.BishopValueOp=250 option.QueenValueOp=800 option.RookValueEn=400

Match result best param vs unoptimized param

TC 5s+50ms

Score of opt_param vs unopt_param: 88 - 37 - 75  [0.627] 200
...      opt_param playing White: 43 - 19 - 38  [0.620] 100
...      opt_param playing Black: 45 - 18 - 37  [0.635] 100
...      White vs Black: 61 - 64 - 75  [0.492] 200
Elo difference: 90.6 +/- 38.7, LOS: 100.0 %, DrawRatio: 37.5 %

opt_param won the match at 62.7%, so tpe in optuna v2.9.1 is ok.

[fsmosca]

Sampler: TPE, multivariate=True

Unoptimize some piece values and then try to optimize it with this sampler. Verify the optimized values by running an engine vs engine match between the unoptimized and optimized piece values.

Optimization Command Line

python tuner.py --engine ./engines/deuterium/deuterium.exe ^
--sampler name=tpe multivariate=True ^
--threshold-pruner result=0.45 ^
--opening-file ./start_opening/ogpt_chess_startpos.epd --opening-format epd ^
--concurrency 6 ^
--base-time-sec 2 --inc-time-sec 0.05 ^
--trials 100 --games-per-trial 100 ^
--study-name opt_tpe_piecevalue_optunav2.9 ^
--resign-movecount 3 --resign-score 400 ^
--draw-movenumber 30 --draw-movecount 6 --draw-score 0 ^
--input-param "{'BishopValueOp': {'default': 250, 'min': 200, 'max': 400, 'step': 2}, 'RookValueEn': {'default': 400, 'min': 300, 'max': 700, 'step': 4}, 'QueenValueOp': {'default': 800, 'min': 600, 'max': 1300, 'step': 4}}"

Best param found by TPE

option.BishopValueOp=300 option.QueenValueOp=1128 option.RookValueEn=592

Unoptimized param

option.BishopValueOp=250 option.QueenValueOp=800 option.RookValueEn=400

Match result best param vs unoptimized param

TC 5s+50ms

Score of opt_param vs unopt_param: 80 - 49 - 71  [0.578] 200
...      opt_param playing White: 46 - 19 - 35  [0.635] 100
...      opt_param playing Black: 34 - 30 - 36  [0.520] 100
...      White vs Black: 76 - 53 - 71  [0.557] 200
Elo difference: 54.3 +/- 39.0, LOS: 99.7 %, DrawRatio: 35.5 %

opt_param won the match at 57.8%, so tpe in optuna v2.9.1 is ok.