Rastrigin benchmark. #141

c-bata · 2020-07-26T18:33:45Z

github-actions · 2020-07-26T18:36:01Z

Benchmark result of Rosenbrock function

Report ID: f4563e09f526ea5541ab48a699a5ac794c3e0f69f5917f79db63fbf978c9ad04
Kurobako Version: 0.1.10
Number of Solvers: 7
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Goptuna (BIPOP-CMA-ES)	2	1
Goptuna (CMA-ES)	2	1
Goptuna (IPOP-CMA-ES)	2	1
Goptuna (TPE)	0	0
Optuna (CMA-ES)	1	1
Optuna (TPE)	0	1
Random	0	0

Individual Results

(1) Problem: Rosenbrock Function

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Optuna (TPE) (study)	1.444340 +- 2.328256	132001.686 +- 216733.334	2.795 +- 0.515
1	Optuna (CMA-ES) (study)	0.617863 +- 1.143378	107558.525 +- 205615.855	0.667 +- 0.156
1	Goptuna (BIPOP-CMA-ES) (study)	0.836955 +- 2.786545	36906.435 +- 47729.791	0.041 +- 0.016
1	Goptuna (IPOP-CMA-ES) (study)	0.836955 +- 2.786545	36906.435 +- 47729.791	0.041 +- 0.022
1	Goptuna (CMA-ES) (study)	0.836955 +- 2.786545	36906.435 +- 47729.791	0.036 +- 0.019
4	Goptuna (TPE) (study)	1.325865 +- 1.228238	331129.200 +- 430962.258	0.162 +- 0.033
5	Random (study)	5.724383 +- 6.688278	140688.449 +- 232034.768	0.000 +- 0.000

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "bipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (BIPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "tpe"
    ]
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 8931843d684313fcaad663dbaa143cbb7bea514bc200c5c8593e10ad7d8d446c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Optuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "tpe"
    ]
  }
}

specification:

{
  "name": "Optuna (TPE)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 01f15f09812e2d814a26d1219a981765c157b45100698158c37abe239cea997b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/rosenbrock_problem",
    "args": []
  }
}

specification:

{
  "name": "Rosenbrock Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rosenbrock",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 448a4fa37c2c00cd2de71c65834d73154693960539a6aab5ea721d8e87cebf27

problem: Rosenbrock Function
solver: Goptuna (BIPOP-CMA-ES)
budget: 100
repeats: 20
concurrency: 1

ID: 61d0e750fd0ffa044e7c517592e06f5a752aac50eb791f8194f11ca88afc650f

problem: Rosenbrock Function
solver: Goptuna (CMA-ES)
budget: 100
repeats: 20
concurrency: 1

ID: 0ea83c6798372e5bb6cc28f133c9f43d8fc3cc2fc35bac6c736d0b173932eae5

problem: Rosenbrock Function
solver: Goptuna (IPOP-CMA-ES)
budget: 100
repeats: 20
concurrency: 1

ID: 9f7d1940842a6b2263038ddb1e94170e969f33e73dfa4fdb0f4302d1ca147ec4

problem: Rosenbrock Function
solver: Goptuna (TPE)
budget: 100
repeats: 20
concurrency: 1

ID: fac4aa5c1cb91e3a4b7b06481bd8bca69d9e363c718dae984eaf369c9d95c73a

problem: Rosenbrock Function
solver: Optuna (CMA-ES)
budget: 100
repeats: 20
concurrency: 1

ID: 2dabcbaaca241ae2723f9c504aa6f2e6fbbd25022b83f7b7b69b8aca4dec9f64

problem: Rosenbrock Function
solver: Optuna (TPE)
budget: 100
repeats: 20
concurrency: 1

ID: f853cdb8bf8b30946ab222443e8ea4105b3e4aa0dde8d0a1f14b8310b870a195

problem: Rosenbrock Function
solver: Random
budget: 100
repeats: 20
concurrency: 1

github-actions · 2020-07-26T19:11:49Z

Benchmark result of Rastrigin problem

Report ID: 91c092360c4c4f8501831200a29d9f73f17f2acdce7a5a2cafa033942c060b22
Kurobako Version: 0.1.10
Number of Solvers: 7
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Goptuna (BIPOP-CMA-ES)	0	1
Goptuna (CMA-ES)	0	1
Goptuna (IPOP-CMA-ES)	0	1
Goptuna (TPE)	1	1
Optuna (CMA-ES)	0	1
Optuna (TPE)	1	1
Random	0	0

Individual Results

(1) Problem: Rastrigin function (dim=2)

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Goptuna (TPE) (study)	0.058057 +- 0.056490	2201.300 +- 500.487	86.190 +- 11.713
1	Optuna (TPE) (study)	0.255531 +- 0.374109	1705.209 +- 951.156	194.462 +- 60.953
1	Optuna (CMA-ES) (study)	1.079281 +- 0.660871	3170.515 +- 1659.740	31.243 +- 3.695
1	Goptuna (BIPOP-CMA-ES) (study)	0.571118 +- 0.443425	2936.112 +- 2657.204	8.574 +- 1.190
1	Goptuna (IPOP-CMA-ES) (study)	0.461764 +- 0.396077	2759.061 +- 2252.559	8.693 +- 1.397
1	Goptuna (CMA-ES) (study)	1.189730 +- 1.362081	3442.373 +- 3291.051	8.334 +- 1.633
3	Random (study)	1.342591 +- 0.675203	5484.908 +- 2427.214	0.006 +- 0.003

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "bipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (BIPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "tpe"
    ]
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 8931843d684313fcaad663dbaa143cbb7bea514bc200c5c8593e10ad7d8d446c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Optuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "tpe"
    ]
  }
}

specification:

{
  "name": "Optuna (TPE)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 0091bc29d1a812f56db93aa64502974e93cc18283ec26b6c5c99b085b81529b8

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/rastrigin_problem",
    "args": [
      "2"
    ]
  }
}

specification:

{
  "name": "Rastrigin function (dim=2)",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rastrigin",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 8c50e86759933a0dbaf04c154ae3cebfbbf6b397a49a1db1b47ac5dae7365a89

problem: Rastrigin function (dim=2)
solver: Goptuna (BIPOP-CMA-ES)
budget: 2500
repeats: 10
concurrency: 1

ID: 5d4812251fac7d96cd6bcb683c134e553741f955e2379673731a49deb201616d

problem: Rastrigin function (dim=2)
solver: Goptuna (CMA-ES)
budget: 2500
repeats: 10
concurrency: 1

ID: af7aa97997c9e5ca6c80b9a7adc66f4979f841b5abba604f9329d12e88fd05fe

problem: Rastrigin function (dim=2)
solver: Goptuna (IPOP-CMA-ES)
budget: 2500
repeats: 10
concurrency: 1

ID: ad5dfb8e8e8259b0545508a7f9bb4794a20418603980eec1b360a0c1109a2024

problem: Rastrigin function (dim=2)
solver: Goptuna (TPE)
budget: 2500
repeats: 10
concurrency: 1

ID: f156615c68518cb4353f04dea3c364c1df46918484a4b84e612a63733075fecf

problem: Rastrigin function (dim=2)
solver: Optuna (CMA-ES)
budget: 2500
repeats: 10
concurrency: 1

ID: fa9d2c5ebf27912d80d2ed5427827541df0332cf56973bcbb8cc878af37ae53c

problem: Rastrigin function (dim=2)
solver: Optuna (TPE)
budget: 2500
repeats: 10
concurrency: 1

ID: 235bc03468a4b2491baa33c1956bb754d38a1022f20005381e0557818c7d759f

problem: Rastrigin function (dim=2)
solver: Random
budget: 2500
repeats: 10
concurrency: 1

c-bata · 2020-07-28T10:58:06Z

cmaes/optimizer.go

+	for i := 0; i < o.dim; i++ {
+		if o.c.At(i, i) <= 0 {
+			minC[i] = epsilon
+		}
+	}


It is required to add epsilon to negative values only for termination criterions.

c-bata added 2 commits July 27, 2020 03:29

Add Rastrigin benchmark

b50ca87

Run Rastrigin benchmark on GitHub Actions

7b97f03

c-bata added 2 commits July 27, 2020 03:45

Fix benchmark runner

c997d7a

fix

90f5033

c-bata added 2 commits July 28, 2020 04:29

Add solvers option for benchmark

b6b43d8

Fix IPOP-CMA-ES and BIPOP-CMA-ES

507b4e5

c-bata commented Jul 28, 2020

View reviewed changes

c-bata merged commit 47c44a8 into master Jul 28, 2020

c-bata deleted the benchmark-rastrigin branch July 28, 2020 11:13

Rastrigin benchmark. #141

Rastrigin benchmark. #141

Conversation

c-bata commented Jul 26, 2020 • edited Loading

github-actions bot commented Jul 26, 2020 • edited Loading

Benchmark result of Rosenbrock function

Table of Contents

Overall Results

Individual Results

(1) Problem: Rosenbrock Function

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

ID: 8931843d684313fcaad663dbaa143cbb7bea514bc200c5c8593e10ad7d8d446c

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

Problems

ID: 01f15f09812e2d814a26d1219a981765c157b45100698158c37abe239cea997b

Studies

ID: 448a4fa37c2c00cd2de71c65834d73154693960539a6aab5ea721d8e87cebf27

ID: 61d0e750fd0ffa044e7c517592e06f5a752aac50eb791f8194f11ca88afc650f

ID: 0ea83c6798372e5bb6cc28f133c9f43d8fc3cc2fc35bac6c736d0b173932eae5

ID: 9f7d1940842a6b2263038ddb1e94170e969f33e73dfa4fdb0f4302d1ca147ec4

ID: fac4aa5c1cb91e3a4b7b06481bd8bca69d9e363c718dae984eaf369c9d95c73a

ID: 2dabcbaaca241ae2723f9c504aa6f2e6fbbd25022b83f7b7b69b8aca4dec9f64

ID: f853cdb8bf8b30946ab222443e8ea4105b3e4aa0dde8d0a1f14b8310b870a195

github-actions bot commented Jul 26, 2020 • edited Loading

Benchmark result of Rastrigin problem

Table of Contents

Overall Results

Individual Results

(1) Problem: Rastrigin function (dim=2)

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

ID: 8931843d684313fcaad663dbaa143cbb7bea514bc200c5c8593e10ad7d8d446c

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

Problems

ID: 0091bc29d1a812f56db93aa64502974e93cc18283ec26b6c5c99b085b81529b8

Studies

ID: 8c50e86759933a0dbaf04c154ae3cebfbbf6b397a49a1db1b47ac5dae7365a89

ID: 5d4812251fac7d96cd6bcb683c134e553741f955e2379673731a49deb201616d

ID: af7aa97997c9e5ca6c80b9a7adc66f4979f841b5abba604f9329d12e88fd05fe

ID: ad5dfb8e8e8259b0545508a7f9bb4794a20418603980eec1b360a0c1109a2024

ID: f156615c68518cb4353f04dea3c364c1df46918484a4b84e612a63733075fecf

ID: fa9d2c5ebf27912d80d2ed5427827541df0332cf56973bcbb8cc878af37ae53c

ID: 235bc03468a4b2491baa33c1956bb754d38a1022f20005381e0557818c7d759f

c-bata Jul 28, 2020

Choose a reason for hiding this comment

c-bata commented Jul 26, 2020 •

edited

Loading

github-actions bot commented Jul 26, 2020 •

edited

Loading

github-actions bot commented Jul 26, 2020 •

edited

Loading