Getting JSON data for point loss batch analysis

[lemonsqueeze]

Hi,

I'm using katrain to get katago json data for a bunch of sgf files i'd like to analyze for research purposes.

It doesn't look like it's possible with the UI but i've come across analyze_rank.py in #44 which got me started for batch-analyzing games. (I'm sure someone must have written this already but couldn't find it so far. That's ok, this is fun to hack but still i feel i must be reinventing the wheel here ! pointers welcome).

I'm interested in point loss so going for a deep analysis instead of the fast one in analyze_rank.py. With max_visits = 1500 and no time limit score estimates are getting more reliable, but there are still candidates with very few visits so i'm a little worried :

{
  "move": "Q2",
  "color": "B",
  "number": 15,
  "scoreLead": -0.9664865378947745,
  "scoreSelfplay": -1.286389747568789,
  "scoreStdev": 21.541545879916775,
  "utility": -0.12590617465633064,
  "visits": 1500,
  "winrate": 0.4371709482508226,
  "pv": ["Q2", "P2", "R1", "R17", "Q17", "R16", "R14", "R15", "Q15", "S14", "M4", "C3", "D3", "C4", "
  "analysis": [
    {"move": "P2", "order": 0, "visits": 1500, "scoreLead": -0.9735200560004781, "winrate": 0.4367476
    {"move": "R17", "order": 1, "visits": 51, "scoreLead": -1.0363104219956802, "winrate": 0.43247503
    {"move": "C3", "order": 2, "visits": 48, "scoreLead": -1.0134750061425206, "winrate": 0.433308394
    {"move": "F3", "order": 3, "visits": 6, "scoreLead": 0.14420036995401714, "winrate": 0.5082638826
    {"move": "R14", "order": 4, "visits": 4, "scoreLead": -0.01672968454658985, "winrate": 0.49544872
    {"move": "C6", "order": 5, "visits": 3, "scoreLead": 0.23954929411411285, "winrate": 0.5127319216
    {"move": "O17", "order": 6, "visits": 3, "scoreLead": 0.08936926225821178, "winrate": 0.502760668
  ]
},

In this case except for the first candidate with 1500 visits i guess i'd better completely ignore the others (only interested in best move anyway).

{
  "move": "P2",
  "color": "W",
  "number": 16,
  "scoreLead": -0.9735200560004781,
  "scoreSelfplay": -1.2958255615766947,
  "scoreStdev": 21.54407626160053,
  "utility": -0.13152747776142845,
  "visits": 1500,
  "winrate": 0.43674760015979985,
  "pv": ["P2", "R1", "R17", "Q17", "R16", "R14", "R15", "Q15", "S14", "M4", "C3", "D3", "C4", "C6", "C5", "$
  "analysis": [{"move": "R1", "order": 999, "visits": 1500, "scoreLead": -0.9749042471774557, "winrate": 0.$
},

Not sure what happened here, looks like only one move was considered in the analysis ?

{
  "move": "D4",
  "color": "B",
  "number": 3,
  "scoreLead": -1.1236937358763324,
  "scoreSelfplay": -1.5667519076162746,
  "scoreStdev": 21.598872021459375,
  "utility": -0.14955704192180344,
  "visits": 1500,
  "winrate": 0.4268740027509653,
  "pv": ["D4", "Q4", "R3", "Q3", "R4", "R6", "R5", "Q5", "S6", "C3", "D3", "C4", "C6", "C5", "D5", "B6", "C$
  "analysis": [
    {"move": "Q5", "order": 999, "visits": 1500, "scoreLead": -0.337117403667626, "winrate": 0.477958785549$
    {"move": "Q4", "order": 0, "visits": 1288, "scoreLead": -1.1384520949120411, "winrate": 0.4259753763773$
    {"move": "Q3", "order": 1, "visits": 115, "scoreLead": -1.055417620899099, "winrate": 0.431187311341676$
    {"move": "R4", "order": 2, "visits": 84, "scoreLead": -1.0194371013709649, "winrate": 0.433226403283807$
    {"move": "R3", "order": 3, "visits": 12, "scoreLead": -0.8789529015559027, "winrate": 0.439822297925012$
  ]
},

If i understand correctly order: 999 is something katrain does, always the next game move,
and order: 0 is katago's best move ?

{
  "move": "Q4",
  "color": "W",
  "number": 8,
  "scoreLead": -2.712600304254664,
  "scoreSelfplay": -3.7457758005050854,
  "scoreStdev": 21.563700537716958,
  "utility": -0.33606110881551826,
  "visits": 1500,
  "winrate": 0.34259312628883,
  "pv": ["Q4", "S5", "S3", "P3", "R5", "S7", "S4", "M3", "O4", "O3", "R10"],
  "analysis": [
    {"move": "S5", "order": 0, "visits": 200, "scoreLead": -2.255016310569035, "winrate": 0.366802145939721$
    {"move": "S3", "order": 1, "visits": 1500, "scoreLead": -2.8379196376659954, "winrate": 0.3362063865049$
    {"move": "R2", "order": 2, "visits": 80, "scoreLead": -2.6986823880258743, "winrate": 0.338590271222458$
    {"move": "P3", "order": 3, "visits": 36, "scoreLead": -2.6808607977089984, "winrate": 0.339099083912248$
    {"move": "R5", "order": 4, "visits": 34, "scoreLead": -4.133533985036006, "winrate": 0.2734066377637826$
    {"move": "S4", "order": 5, "visits": 22, "scoreLead": -2.818905469888048, "winrate": 0.339218835681255,$
    {"move": "C17", "order": 6, "visits": 3, "scoreLead": -3.4279443422953286, "winrate": 0.309672510295058$
  ]
},

This one looks problematic, first candidate has only 200 visits.
(maybe he was forced to look at S3 as it's the next game move and there wasn't enough playouts left to analyze S5 properly ?)
If i remember correctly there's a Force equal visits in the UI settings, maybe i need to do something similar here ?

I can post code / sgf if that helps.
btw, i had to revert to katrain 1.3.4 to get analyze_rank.py to work, it wouldn't work with latest release. I guess that's ok as long as it's getting correct data from katago, or is it worth trying to get it working with 1.6 ?

That's a lot of questions =)
Thanks for Katrain, amazing tool !

[sanderland]

The bots repo hasn't been updated in a while, but you're right in that it has a lot of useful code. An old version should be fine for your purpose, or try to update it could be fun.
Your main confusion seems to come from this function

katrain/katrain/core/game_node.py

Line 101 in e4e140f

def update_move_analysis(self, move_analysis, move_gtp):

When the next move is played, it updates it's parent (since there is now a more accurate analysis available) - using order:999 if the move wasn't previously in the candidates.
If you want a more normal analysis, you probably have to implement your own callback function to the KataGoEngine, or hack around and strip this updating out.

[sanderland]

Also, the uneven visits is just a normal result of the alphago zero algorithm, a very sharp policy can make the exploration a tad useless. You can of course take all candidates and fire off an equal number of visits to them, or even take the top n policy moves in addition.