Support BIPOP-CMA-ES algorithm. #136

c-bata · 2020-07-25T12:22:46Z

BIPOP-CMA-ES is a multi-start CMA-ES with equal budgets for two interlaced restart strategies, one with an increasing population size and one with varying small population sizes.

Hansen N. Benchmarking a BI-Population CMA-ES on the BBOB-2009 Function Testbed. In the workshop Proceedings of the Genetic and Evolutionary Computation Conference, GECCO, pages 2389–2395. ACM, 2009.

github-actions · 2020-07-25T12:25:23Z

Benchmark result of rosenbrock function

Report ID: b7b8cccd46397ff21164bf7c48b6aac89e7eec3ed5b68f68d56671706eb15ac5
Kurobako Version: 0.1.10
Number of Solvers: 7
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Goptuna (BIPOP-CMA-ES)	0	1
Goptuna (CMA-ES)	0	1
Goptuna (IPOP-CMA-ES)	0	1
Goptuna (TPE)	0	1
Optuna (CMA-ES)	0	1
Optuna (TPE)	0	1
Random	0	1

Individual Results

(1) Problem: Rosenbrock Function

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Goptuna (TPE) (study)	4.702807 +- 4.378348	231728.954 +- 329038.327	0.056 +- 0.002
1	Optuna (TPE) (study)	3.335621 +- 4.531201	304779.288 +- 201670.007	0.794 +- 0.009
1	Optuna (CMA-ES) (study)	0.338189 +- 0.509293	262976.352 +- 188831.092	0.197 +- 0.006
1	Goptuna (BIPOP-CMA-ES) (study)	1.896292 +- 3.201906	7277.894 +- 4959.995	0.021 +- 0.001
1	Goptuna (IPOP-CMA-ES) (study)	1.896292 +- 3.201906	7277.894 +- 4959.995	0.021 +- 0.002
1	Goptuna (CMA-ES) (study)	1.896292 +- 3.201906	7277.894 +- 4959.995	0.022 +- 0.002
1	Random (study)	26.724556 +- 28.472584	264073.317 +- 311529.192	0.000 +- 0.000

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "bipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (BIPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "tpe"
    ]
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 8931843d684313fcaad663dbaa143cbb7bea514bc200c5c8593e10ad7d8d446c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Optuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "tpe"
    ]
  }
}

specification:

{
  "name": "Optuna (TPE)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 01f15f09812e2d814a26d1219a981765c157b45100698158c37abe239cea997b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/rosenbrock_problem",
    "args": []
  }
}

specification:

{
  "name": "Rosenbrock Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rosenbrock",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 448a4fa37c2c00cd2de71c65834d73154693960539a6aab5ea721d8e87cebf27

problem: Rosenbrock Function
solver: Goptuna (BIPOP-CMA-ES)
budget: 100
repeats: 5
concurrency: 1

ID: 61d0e750fd0ffa044e7c517592e06f5a752aac50eb791f8194f11ca88afc650f

problem: Rosenbrock Function
solver: Goptuna (CMA-ES)
budget: 100
repeats: 5
concurrency: 1

ID: 0ea83c6798372e5bb6cc28f133c9f43d8fc3cc2fc35bac6c736d0b173932eae5

problem: Rosenbrock Function
solver: Goptuna (IPOP-CMA-ES)
budget: 100
repeats: 5
concurrency: 1

ID: 9f7d1940842a6b2263038ddb1e94170e969f33e73dfa4fdb0f4302d1ca147ec4

problem: Rosenbrock Function
solver: Goptuna (TPE)
budget: 100
repeats: 5
concurrency: 1

ID: fac4aa5c1cb91e3a4b7b06481bd8bca69d9e363c718dae984eaf369c9d95c73a

problem: Rosenbrock Function
solver: Optuna (CMA-ES)
budget: 100
repeats: 5
concurrency: 1

ID: 2dabcbaaca241ae2723f9c504aa6f2e6fbbd25022b83f7b7b69b8aca4dec9f64

problem: Rosenbrock Function
solver: Optuna (TPE)
budget: 100
repeats: 5
concurrency: 1

ID: f853cdb8bf8b30946ab222443e8ea4105b3e4aa0dde8d0a1f14b8310b870a195

problem: Rosenbrock Function
solver: Random
budget: 100
repeats: 5
concurrency: 1

github-actions · 2020-07-25T12:28:33Z

Benchmark result of sigopt/evalset/Ackley problem

Report ID: e0ad916eeaa059d8dc752a9368f3586a5bb461fa704ceeec2381c29ebfe1a6be
Kurobako Version: 0.1.10
Number of Solvers: 6
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Goptuna (BIPOP-CMA-ES)	0	1
Goptuna (CMA-ES)	0	1
Goptuna (IPOP-CMA-ES)	0	1
Goptuna (TPE)	0	1
Optuna (TPE)	0	1
Random	0	1

Individual Results

(1) Problem: sigopt/evalset/Ackley(dim=10)

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Goptuna (TPE) (study)	5.111347 +- 0.487780	8322.563 +- 592.899	42.871 +- 6.946
1	Optuna (TPE) (study)	6.326946 +- 1.945140	8739.853 +- 1732.156	123.787 +- 31.467
1	Goptuna (BIPOP-CMA-ES) (study)	0.050740 +- 0.039694	3875.009 +- 339.454	1.440 +- 0.304
1	Goptuna (IPOP-CMA-ES) (study)	0.050740 +- 0.039694	3875.009 +- 339.454	1.291 +- 0.221
1	Goptuna (CMA-ES) (study)	0.050740 +- 0.039694	3875.009 +- 339.454	1.399 +- 0.295
1	Random (study)	15.476572 +- 0.916833	16117.231 +- 457.140	0.001 +- 0.000

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "bipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (BIPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "tpe"
    ]
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "tpe"
    ]
  }
}

specification:

{
  "name": "Optuna (TPE)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 93492282fbba0e83fba906eaf579809327a6e263940fb4fd42cc3abc9a155beb

recipe:

{
  "sigopt": {
    "name": "ACKLEY",
    "dim": 10
  }
}

specification:

{
  "name": "sigopt/evalset/Ackley(dim=10)",
  "attrs": {
    "github": "https://github.com/sigopt/evalset",
    "paper": "Dewancker, Ian, et al. \"A strategy for ranking optimization methods using multiple criteria.\" Workshop on Automatic Machine Learning. 2016.",
    "version": "kurobako_problems=0.1.7"
  },
  "params_domain": [
    {
      "name": "p0",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p1",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p2",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p3",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p4",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p5",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p6",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p7",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p8",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p9",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Objective Value",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: f0fa2ceb80845ca22ffa2479f45c1ff82f4870c1bf20f556b5155edd6387cda8

problem: sigopt/evalset/Ackley(dim=10)
solver: Goptuna (BIPOP-CMA-ES)
budget: 1000
repeats: 5
concurrency: 1

ID: 1d5f43ce7d06f0ecc20f649b9a660458e0e55148ceb2ff28705e0a123f802638

problem: sigopt/evalset/Ackley(dim=10)
solver: Goptuna (CMA-ES)
budget: 1000
repeats: 5
concurrency: 1

ID: 6f8c38733e62ecf7d89a3669fe8739e12eee2d4828c035728ff6e509aac633af

problem: sigopt/evalset/Ackley(dim=10)
solver: Goptuna (IPOP-CMA-ES)
budget: 1000
repeats: 5
concurrency: 1

ID: 66298c57166677e2f5d11b0595998342167947d2100a4d34abe76b2d73bdbd0a

problem: sigopt/evalset/Ackley(dim=10)
solver: Goptuna (TPE)
budget: 1000
repeats: 5
concurrency: 1

ID: 135c5259d0a821bd89b8eb14637fe9ff1e1028162a8a083ac07acff879672620

problem: sigopt/evalset/Ackley(dim=10)
solver: Optuna (TPE)
budget: 1000
repeats: 5
concurrency: 1

ID: ac65437313c4ab7ce99a7ecb88d98d40e124c8e950a546486d06df3b17d4e0ec

problem: sigopt/evalset/Ackley(dim=10)
solver: Random
budget: 1000
repeats: 5
concurrency: 1

c-bata added 4 commits July 25, 2020 20:58

Add BIPOP-CMA-ES example

0497302

Implement BIPOP-CMA-ES sampler

f30e017

Add kurobako solver for BIPOP-CMA-ES

99a7a06

Add BIPOP-CMA-ES in supported algorithms

e9a458d

Fix lint errors

3708656

c-bata force-pushed the bipop-cmaes branch from 7290c20 to 3708656 Compare July 25, 2020 12:41

c-bata changed the title ~~Support BIPOP-CMA-ES sampler.~~ Support BIPOP-CMA-ES algorithm. Jul 25, 2020

c-bata merged commit c13e9a4 into master Jul 25, 2020

c-bata deleted the bipop-cmaes branch July 25, 2020 12:48

c-bata mentioned this pull request May 17, 2023

Add BIPOP-CMA-ES support in CmaEsSampler optuna/optuna#4464

Merged

Support BIPOP-CMA-ES algorithm. #136

Support BIPOP-CMA-ES algorithm. #136

Conversation

c-bata commented Jul 25, 2020 • edited Loading

github-actions bot commented Jul 25, 2020 • edited Loading

Benchmark result of rosenbrock function

Table of Contents

Overall Results

Individual Results

(1) Problem: Rosenbrock Function

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

ID: 8931843d684313fcaad663dbaa143cbb7bea514bc200c5c8593e10ad7d8d446c

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

Problems

ID: 01f15f09812e2d814a26d1219a981765c157b45100698158c37abe239cea997b

Studies

ID: 448a4fa37c2c00cd2de71c65834d73154693960539a6aab5ea721d8e87cebf27

ID: 61d0e750fd0ffa044e7c517592e06f5a752aac50eb791f8194f11ca88afc650f

ID: 0ea83c6798372e5bb6cc28f133c9f43d8fc3cc2fc35bac6c736d0b173932eae5

ID: 9f7d1940842a6b2263038ddb1e94170e969f33e73dfa4fdb0f4302d1ca147ec4

ID: fac4aa5c1cb91e3a4b7b06481bd8bca69d9e363c718dae984eaf369c9d95c73a

ID: 2dabcbaaca241ae2723f9c504aa6f2e6fbbd25022b83f7b7b69b8aca4dec9f64

ID: f853cdb8bf8b30946ab222443e8ea4105b3e4aa0dde8d0a1f14b8310b870a195

github-actions bot commented Jul 25, 2020 • edited Loading

Benchmark result of sigopt/evalset/Ackley problem

Table of Contents

Overall Results

Individual Results

(1) Problem: sigopt/evalset/Ackley(dim=10)

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

Problems

ID: 93492282fbba0e83fba906eaf579809327a6e263940fb4fd42cc3abc9a155beb

Studies

ID: f0fa2ceb80845ca22ffa2479f45c1ff82f4870c1bf20f556b5155edd6387cda8

ID: 1d5f43ce7d06f0ecc20f649b9a660458e0e55148ceb2ff28705e0a123f802638

ID: 6f8c38733e62ecf7d89a3669fe8739e12eee2d4828c035728ff6e509aac633af

ID: 66298c57166677e2f5d11b0595998342167947d2100a4d34abe76b2d73bdbd0a

ID: 135c5259d0a821bd89b8eb14637fe9ff1e1028162a8a083ac07acff879672620

ID: ac65437313c4ab7ce99a7ecb88d98d40e124c8e950a546486d06df3b17d4e0ec

c-bata commented Jul 25, 2020 •

edited

Loading

github-actions bot commented Jul 25, 2020 •

edited

Loading

github-actions bot commented Jul 25, 2020 •

edited

Loading