Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rastrigin benchmark. #141

Merged
merged 6 commits into from
Jul 28, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,12 @@ name: Run kurobako benchmark
on:
pull_request:
paths:
- '.github/workflows/benchmark-ackley.yml'
- '.github/workflows/benchmark-rastrigin.yml'
- '_benchmarks/runner.sh'
- 'internal/**.go'
- 'tpe/**.go'
- 'cmaes/**.go'
- 'tpe/**.go'
jobs:
benchmarks-ackley:
benchmarks-rastrigin:
name: Run kurobako benchmark
runs-on: ubuntu-latest

Expand All @@ -33,10 +32,13 @@ jobs:
chmod +x kurobako
./kurobako -h

- name: Run sigopt/evalset/Ackley Benchmark
- name: Run Rastrigin Benchmark
env:
KUROBAKO: ./kurobako
run: ./_benchmarks/runner.sh ackley ./kurobako-report.json
DIM: 2
REPEATS: 10
BUDGET: 2500
run: ./_benchmarks/runner.sh rastrigin ./kurobako-report.json
- name: Plot kurobako result
uses: c-bata/github-actions-kurobako/plot@v1
id: kurobako-plot
Expand Down Expand Up @@ -66,24 +68,29 @@ jobs:
if: ${{ env.HAS_SECRET == 1 }}
- name: Upload an image to google cloud storage
if: ${{ env.HAS_SECRET == 1 }}
run: gsutil cp ${{ steps.kurobako-plot.outputs.image-path }} gs://kurobako-reports/${{ github.repository }}/ackley-${{ github.sha }}.png
run: gsutil cp ${{ steps.kurobako-plot.outputs.image-path }} gs://kurobako-reports/${{ github.repository }}/rastrigin-${{ github.sha }}.png
- name: Comment to Pull Request
if: ${{ env.HAS_SECRET == 1 }}
uses: c-bata/github-actions-kurobako@v2
with:
report-md-path: './kurobako-report.md'
public-image-url: https://storage.googleapis.com/kurobako-reports/${{ github.repository }}/ackley-${{ github.sha }}.png
title: 'Benchmark result of sigopt/evalset/Ackley problem'
public-image-url: https://storage.googleapis.com/kurobako-reports/${{ github.repository }}/rastrigin-${{ github.sha }}.png
title: 'Benchmark result of Rastrigin problem'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- run: cp ${{ steps.kurobako-plot.outputs.image-path }} ackley.png
- run: mv ./kurobako-report.md ./kurobako-report-ackley.md
- run: cp ${{ steps.kurobako-plot.outputs.image-path }} rastrigin.png
- run: mv ./kurobako-report.md ./kurobako-report-rastrigin.md
- run: mv ./kurobako-report.json ./kurobako-report-rastrigin.json
- uses: actions/upload-artifact@v2
with:
name: kurobako-report
path: rastrigin.png
- uses: actions/upload-artifact@v2
with:
name: kurobako-report
path: ackley.png
path: kurobako-report-rastrigin.md
- uses: actions/upload-artifact@v2
with:
name: kurobako-report
path: kurobako-report-ackley.md
path: kurobako-report-rastrigin.json
8 changes: 8 additions & 0 deletions .github/workflows/benchmark-rosenbrock.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ on:
- '.github/workflows/benchmark-rosenbrock.yml'
- '_benchmarks/runner.sh'
- 'cmaes/**.go'
- 'tpe/**.go'
jobs:
benchmarks-rosenbrock:
name: Run kurobako benchmark
Expand Down Expand Up @@ -34,6 +35,8 @@ jobs:
- name: Run benchmarks of Rosenbrock function
env:
KUROBAKO: ./kurobako
REPEATS: 20
BUDGET: 100
run: ./_benchmarks/runner.sh rosenbrock ./kurobako-report.json
- name: Plot kurobako result
uses: c-bata/github-actions-kurobako/plot@v1
Expand Down Expand Up @@ -78,6 +81,7 @@ jobs:

- run: cp ${{ steps.kurobako-plot.outputs.image-path }} rosenbrock.png
- run: mv ./kurobako-report.md ./kurobako-report-rosenbrock.md
- run: mv ./kurobako-report.json ./kurobako-report-rosenbrock.json
- uses: actions/upload-artifact@v2
with:
name: kurobako-report
Expand All @@ -86,3 +90,7 @@ jobs:
with:
name: kurobako-report
path: kurobako-report-rosenbrock.md
- uses: actions/upload-artifact@v2
with:
name: kurobako-report
path: kurobako-report-rosenbrock.json
7 changes: 4 additions & 3 deletions _benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,11 @@ Usage:
Problem:
rosenbrock : https://www.sfu.ca/~ssurjano/rosen.html
himmelblau : https://en.wikipedia.org/wiki/Himmelblau%27s_function
ackley : Ackley function in https://github.com/sigopt/evalset
ackley : https://www.sfu.ca/~ssurjano/ackley.html
rastrigin : https://www.sfu.ca/~ssurjano/rastr.html
weierstrass : Weierstrass function in https://github.com/sigopt/evalset
schwefel20 : Schwefel20 function in https://github.com/sigopt/evalset
schwefel36 : Schwefel36 function in https://github.com/sigopt/evalset
schwefel20 : https://www.sfu.ca/~ssurjano/schwef.html
schwefel36 : https://www.sfu.ca/~ssurjano/schwef.html
Options:
--help, -h print this
Example:
Expand Down
69 changes: 69 additions & 0 deletions _benchmarks/rastrigin_problem/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
package main

import (
"fmt"
"math"
"os"
"strconv"

"github.com/sile/kurobako-go"
)

type rastriginProblemFactory struct {
dim int
}

func (r *rastriginProblemFactory) Specification() (*kurobako.ProblemSpec, error) {
spec := kurobako.NewProblemSpec(fmt.Sprintf("Rastrigin function (dim=%d)", r.dim))

spec.Params = make([]kurobako.Var, r.dim)
for i := 0; i < r.dim; i++ {
x := kurobako.NewVar("x" + strconv.Itoa(i+1))
x.Range = kurobako.ContinuousRange{Low: -5.12, High: 5.12}.ToRange()
spec.Params[i] = x
}

spec.Values = []kurobako.Var{kurobako.NewVar("Rastrigin")}
return &spec, nil
}

func (r *rastriginProblemFactory) CreateProblem(seed int64) (kurobako.Problem, error) {
return &rastriginProblem{}, nil
}

type rastriginProblem struct {
}

func (r *rastriginProblem) CreateEvaluator(params []float64) (kurobako.Evaluator, error) {
return &rastriginEvaluator{params: params}, nil
}

type rastriginEvaluator struct {
params []float64
}

func (r *rastriginEvaluator) Evaluate(nextStep uint64) (uint64, []float64, error) {
v := float64(10 * len(r.params))
for i := range r.params {
v += math.Pow(r.params[i], 2) - 10*math.Cos(2*math.Pi*r.params[i])
}
return 1, []float64{v}, nil
}

func main() {
var dim int
if len(os.Args) == 2 {
a := os.Args[1]
b, err := strconv.Atoi(a)
if err != nil {
panic(err)
}
dim = b
}
runner := kurobako.NewProblemRunner(&rastriginProblemFactory{
dim: dim,
})
if err := runner.Run(); err != nil {
panic(err)
}
}
95 changes: 54 additions & 41 deletions _benchmarks/runner.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ set -e
KUROBAKO=${KUROBAKO:-kurobako}
DIR=$(cd $(dirname $0); pwd)
BINDIR=$(dirname $DIR)/bin
REPEATS=${REPEATS:-5}
BUDGET=${BUDGET:-300}
SEED=${BUDGET:-1}
DIM=${DIM:-2}
SOLVERS=${SOLVERS:-all}

usage() {
cat <<EOF
Expand All @@ -14,10 +19,11 @@ Usage:
Problem:
rosenbrock : https://www.sfu.ca/~ssurjano/rosen.html
himmelblau : https://en.wikipedia.org/wiki/Himmelblau%27s_function
ackley : Ackley function in https://github.com/sigopt/evalset
ackley : https://www.sfu.ca/~ssurjano/ackley.html
rastrigin : https://www.sfu.ca/~ssurjano/rastr.html
weierstrass : Weierstrass function in https://github.com/sigopt/evalset
schwefel20 : Schwefel20 function in https://github.com/sigopt/evalset
schwefel36 : Schwefel36 function in https://github.com/sigopt/evalset
schwefel20 : https://www.sfu.ca/~ssurjano/schwef.html
schwefel36 : https://www.sfu.ca/~ssurjano/schwef.html
Options:
--help, -h print this
Example:
Expand All @@ -31,6 +37,7 @@ mkdir -p $BINDIR
go build -o ${BINDIR}/goptuna_solver ${DIR}/goptuna_solver/main.go
go build -o ${BINDIR}/himmelblau_problem ${DIR}/himmelblau_problem/main.go
go build -o ${BINDIR}/rosenbrock_problem ${DIR}/rosenbrock_problem/main.go
go build -o ${BINDIR}/rastrigin_problem ${DIR}/rastrigin_problem/main.go

RANDOM_SOLVER=$($KUROBAKO solver random)
CMA_SOLVER=$($KUROBAKO solver command ${BINDIR}/goptuna_solver cmaes)
Expand All @@ -43,57 +50,25 @@ OPTUNA_TPE_SOLVER=$($KUROBAKO solver command python ${DIR}/optuna_solver.py tpe)
case "$1" in
himmelblau)
PROBLEM=$($KUROBAKO problem command ${BINDIR}/himmelblau_problem)
$KUROBAKO studies \
--solvers $RANDOM_SOLVER $CMA_SOLVER $IPOP_CMA_SOLVER $BIPOP_CMA_SOLVER \
$OPTUNA_CMA_SOLVER $TPE_SOLVER $OPTUNA_TPE_SOLVER \
--problems $PROBLEM \
--seed 1 --repeats 5 --budget 100 \
| $KUROBAKO run --parallelism 1 > $2
;;
rosenbrock)
PROBLEM=$($KUROBAKO problem command ${BINDIR}/rosenbrock_problem)
$KUROBAKO studies \
--solvers $RANDOM_SOLVER $CMA_SOLVER $IPOP_CMA_SOLVER $BIPOP_CMA_SOLVER \
$OPTUNA_CMA_SOLVER $TPE_SOLVER $OPTUNA_TPE_SOLVER \
--problems $PROBLEM \
--seed 1 --repeats 5 --budget 100 \
| $KUROBAKO run --parallelism 1 > $2
;;
ackley)
PROBLEM=$($KUROBAKO problem sigopt --dim 10 ackley)
$KUROBAKO studies \
--solvers $RANDOM_SOLVER $CMA_SOLVER $IPOP_CMA_SOLVER $BIPOP_CMA_SOLVER \
$TPE_SOLVER $OPTUNA_TPE_SOLVER \
--problems $PROBLEM \
--seed 1 --repeats 5 --budget 1000 \
| $KUROBAKO run --parallelism 5 > $2
PROBLEM=$($KUROBAKO problem sigopt --dim $DIM ackley)
;;
rastrigin)
# "kurobako problem sigopt --dim 8 rastrigin" only accepts 8-dim.
PROBLEM=$($KUROBAKO problem command ${BINDIR}/rastrigin_problem $DIM)
;;
weierstrass)
PROBLEM=$($KUROBAKO problem sigopt --dim 10 weierstrass)
$KUROBAKO studies \
--solvers $RANDOM_SOLVER $CMA_SOLVER $IPOP_CMA_SOLVER $BIPOP_CMA_SOLVER \
$TPE_SOLVER $OPTUNA_TPE_SOLVER \
--problems $PROBLEM \
--seed 1 --repeats 5 --budget 1000 \
| $KUROBAKO run --parallelism 5 > $2
PROBLEM=$($KUROBAKO problem sigopt --dim $DIM weierstrass)
;;
schwefel20)
PROBLEM=$($KUROBAKO problem sigopt --dim 2 schwefel20)
$KUROBAKO studies \
--solvers $RANDOM_SOLVER $CMA_SOLVER $IPOP_CMA_SOLVER $BIPOP_CMA_SOLVER \
$TPE_SOLVER $OPTUNA_TPE_SOLVER \
--problems $PROBLEM \
--seed 1 --repeats 10 --budget 100 \
| $KUROBAKO run --parallelism 5 > $2
;;
schwefel36)
PROBLEM=$($KUROBAKO problem sigopt --dim 2 schwefel36)
$KUROBAKO studies \
--solvers $RANDOM_SOLVER $CMA_SOLVER $IPOP_CMA_SOLVER $BIPOP_CMA_SOLVER \
$TPE_SOLVER $OPTUNA_TPE_SOLVER \
--problems $PROBLEM \
--seed 1 --repeats 10 --budget 100 \
| $KUROBAKO run --parallelism 5 > $2
;;
help|--help|-h)
usage
Expand All @@ -105,3 +80,41 @@ case "$1" in
exit 1
;;
esac
case $SOLVERS in
all)
$KUROBAKO studies \
--solvers \
$RANDOM_SOLVER \
$CMA_SOLVER \
$IPOP_CMA_SOLVER \
$BIPOP_CMA_SOLVER \
$TPE_SOLVER \
$OPTUNA_CMA_SOLVER \
$OPTUNA_TPE_SOLVER \
--problems $PROBLEM \
--seed $SEED --repeats $REPEATS --budget $BUDGET \
| $KUROBAKO run --parallelism 7 > $2
;;
cmaes)
$KUROBAKO studies \
--solvers \
$RANDOM_SOLVER \
$CMA_SOLVER \
$IPOP_CMA_SOLVER \
$BIPOP_CMA_SOLVER \
$OPTUNA_CMA_SOLVER \
--problems $PROBLEM \
--seed $SEED --repeats $REPEATS --budget $BUDGET \
| $KUROBAKO run --parallelism 5 > $2
;;
tpe)
$KUROBAKO studies \
--solvers \
$RANDOM_SOLVER \
$TPE_SOLVER \
$OPTUNA_TPE_SOLVER \
--problems $PROBLEM \
--seed $SEED --repeats $REPEATS --budget $BUDGET \
| $KUROBAKO run --parallelism 3 > $2
;;
esac
6 changes: 5 additions & 1 deletion cmaes/optimizer.go
Original file line number Diff line number Diff line change
Expand Up @@ -391,7 +391,11 @@ func (o *Optimizer) Tell(solutions []*Solution) error {
// Avoid eigendecomposition error by arithmetic overflow
// This ensures that C has positive definite properties.
minC := make([]float64, o.dim)
floats.AddConst(epsilon, minC)
for i := 0; i < o.dim; i++ {
if o.c.At(i, i) <= 0 {
minC[i] = epsilon
}
}
Comment on lines +394 to +398
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is required to add epsilon to negative values only for termination criterions.

o.c.AddSym(o.c, mat.NewDiagDense(o.dim, minC))

// Stores 'best' and 'worst' values of the last 'funHistTerm' generations.
Expand Down