Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggest int parameter with step #98

Merged
merged 4 commits into from
Apr 10, 2020
Merged

Suggest int parameter with step #98

merged 4 commits into from
Apr 10, 2020

Conversation

c-bata
Copy link
Owner

@c-bata c-bata commented Apr 10, 2020

Comment on lines +112 to +123
// StepIntUniformDistributionName is the identifier name of IntUniformDistribution
const StepIntUniformDistributionName = "StepIntUniformDistribution"

// StepIntUniformDistribution is a uniform distribution on integers.
type StepIntUniformDistribution struct {
// High is higher endpoint of the range of the distribution (included in the range).
High int `json:"high"`
// Low is lower endpoint of the range of the distribution (included in the range).
Low int `json:"low"`
// Step is a spacing between values.
Step int `json:"step"`
}
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a new distribution type to keep backward compatibility.

@github-actions
Copy link

github-actions bot commented Apr 10, 2020

Benchmark result of sigopt/evalset/Ackley problem

plot curve image

  • Report ID: 41b809378d20555513ee9c7e20dbd2a0ef15955c04f6f51520f43731af17035f
  • Kurobako Version: 0.1.8
  • Number of Solvers: 2
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (TPE) 1 1
Random 0 0

Individual Results

(1) Problem: sigopt/evalset/Ackley(dim=5)

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Goptuna (TPE) (study) 8.775326 +- 1.811908 1231.902 +- 128.157 0.459 +- 0.040
2 Random (study) 13.719907 +- 1.685560 1502.255 +- 151.019 0.000 +- 0.000

Solvers

ID: 79277b8469b8810702825f63c5b450bba72028be027598049d9b7a5d94ccd4ce

recipe:

{
  "command": {
    "path": "./tpe_solver",
    "args": []
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 3f8337a230f8a248b2222ad48cb65d6334b244dc5793727eaf280dcf9c08b732

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.6"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: f5a73e549969e72aaf6bfb7b92b4b824f86adf193a308383e29d6c8dea7fe3fe

recipe:

{
  "sigopt": {
    "name": "ACKLEY",
    "dim": 5
  }
}

specification:

{
  "name": "sigopt/evalset/Ackley(dim=5)",
  "attrs": {
    "github": "https://github.com/sigopt/evalset",
    "paper": "Dewancker, Ian, et al. \"A strategy for ranking optimization methods using multiple criteria.\" Workshop on Automatic Machine Learning. 2016.",
    "version": "kurobako_problems=0.1.7"
  },
  "params_domain": [
    {
      "name": "p0",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p1",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p2",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p3",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p4",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Objective Value",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 56dbe97b2b0b9aa62ca35370ecf9f61ec0f5a215074f5d8ecdf70b17ce188fbb

ID: e7386be274c1bfa828b7fffcf965be0c56c47a4e18d8566cfc83310946d19c14

@github-actions
Copy link

Benchmark result of himmelblau function

plot curve image

  • Report ID: 027a724771a1b793b871db261a884e6c5ea09edbbd8420c0eabaa8bbf5be65ea
  • Kurobako Version: 0.1.4
  • Number of Solvers: 3
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (CMA-ES) 2 1
Goptuna (TPE) 0 0
Random 0 0

Individual Results

(1) Problem: Himmelblau Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Goptuna (CMA-ES) (study) 0.000001 +- 0.000001 1157.893 +- 264.877 0.075 +- 0.005
2 Random (study) 0.448275 +- 0.342809 743.186 +- 182.032 0.000 +- 0.000
2 Goptuna (TPE) (study) 0.083196 +- 0.047579 712.047 +- 397.889 0.334 +- 0.014

Solvers

ID: 55d05ec1515f9127963f4269fdf631907f78c963bff02793ff8c76b987bc277d

recipe:

{
  "command": {
    "path": "./cma_solver",
    "args": []
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 79277b8469b8810702825f63c5b450bba72028be027598049d9b7a5d94ccd4ce

recipe:

{
  "command": {
    "path": "./tpe_solver",
    "args": []
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 29b0b413e0e229de860c855bd7258ff2335c36821b0877f4c28fe96fc82afb3c

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.4"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: fa9f14b4a97956cc4d0dd6769e8548cd5a12ae28520361741d847bd67d1cc511

recipe:

{
  "command": {
    "path": "./himmelblau_problem",
    "args": []
  }
}

specification:

{
  "name": "Himmelblau Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Himmelblau",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 8fbd46ba92dfcba2c6fac15cb53a81d261f30a2a7993b48024a012dd774671b9

ID: d39fa90bc8984e8b3690e27b79ca195bce3500a8b5873d2908b98cb34d2cd554

ID: fdcb4ad678796c7c8110d1aba73c9e9a31816a93aef77b8e5c8658d67d206a53

@github-actions
Copy link

Benchmark result of rosenbrock function

plot curve image

  • Report ID: 4ac5c4941e3758ed9b6590b43b4d233ebd0c666079260d679317e644482c489b
  • Kurobako Version: 0.1.4
  • Number of Solvers: 3
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (CMA-ES) 1 1
Goptuna (TPE) 1 1
Random 0 0

Individual Results

(1) Problem: Rosenbrock Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Goptuna (CMA-ES) (study) 0.107743 +- 0.169375 33104.174 +- 44970.384 0.095 +- 0.009
1 Goptuna (TPE) (study) 0.341390 +- 0.478685 181558.957 +- 286705.584 0.396 +- 0.010
3 Random (study) 2.195504 +- 1.702321 204950.104 +- 222156.604 0.000 +- 0.000

Solvers

ID: 55d05ec1515f9127963f4269fdf631907f78c963bff02793ff8c76b987bc277d

recipe:

{
  "command": {
    "path": "./cma_solver",
    "args": []
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 79277b8469b8810702825f63c5b450bba72028be027598049d9b7a5d94ccd4ce

recipe:

{
  "command": {
    "path": "./tpe_solver",
    "args": []
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 29b0b413e0e229de860c855bd7258ff2335c36821b0877f4c28fe96fc82afb3c

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.4"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 52cccf44ea75f7b3ffac45f5590f4957c127439c3a1685179124a94a06e16b64

recipe:

{
  "command": {
    "path": "./rosenbrock_problem",
    "args": []
  }
}

specification:

{
  "name": "Rosenbrock Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rosenbrock",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 49f9ee312dfcca494dca8fb37a2b167ecaa24509a795ae24efb7f1c695ac6da9

ID: 9393401716052aeaafc25616db33049d94c0cd2884bc6becab4e0dca520536f9

ID: 0a0b6c449c4456a2f1e8d0e919c0c8e6ccaf2d274ec43075abb065facdd30861

@c-bata c-bata merged commit 5730a51 into master Apr 10, 2020
@c-bata c-bata deleted the suggest-int-with-step branch April 10, 2020 06:29
@c-bata
Copy link
Owner Author

c-bata commented Apr 10, 2020

package main

import (
	"fmt"
	"log"
	"math"

	"github.com/c-bata/goptuna"
	"github.com/c-bata/goptuna/tpe"
)

func objective(trial goptuna.Trial) (float64, error) {
	x1, _ := trial.SuggestUniform("x1", -10, 10)
	x2, _ := trial.SuggestIntWithStep("x2", -10, 10, 4)
	fmt.Println("x1:", x1, "x2:", x2)
	return math.Pow(x1-2, 2) + math.Pow(float64(x2)+5, 2), nil
}

func main() {
	study, err := goptuna.CreateStudy(
		"goptuna-example",
		goptuna.StudyOptionSampler(tpe.NewSampler()),
	)
	if err != nil {
		log.Fatal("failed to create study:", err)
	}

	if err = study.Optimize(objective, 25); err != nil {
		log.Fatal("failed to optimize:", err)
	}

	v, _ := study.GetBestValue()
	params, _ := study.GetBestParams()
	log.Printf("Best evaluation=%f (x1=%f, x2=%d)",
		v, params["x1"].(float64), params["x2"].(int))
}

TPE

(venv) $ go run _examples/simple_tpe/main.go 
x1: 8.903922985882328 x2: 6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=0 state=Complete evaluation=168.664153
x1: 3.1191253039081026 x2: -6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=1 state=Complete evaluation=2.252441
x1: -2.648255867350829 x2: -6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=2 state=Complete evaluation=22.606283
x1: -6.151227806501357 x2: -2
2020/04/10 15:31:12 [INFO] Trial finished: trialID=3 state=Complete evaluation=75.442515
x1: 7.943394262996019 x2: 2
2020/04/10 15:31:12 [INFO] Trial finished: trialID=4 state=Complete evaluation=84.323935
x1: -4.228286963890898 x2: -2
2020/04/10 15:31:12 [INFO] Trial finished: trialID=5 state=Complete evaluation=47.791559
x1: 6.995605635257469 x2: 2
2020/04/10 15:31:12 [INFO] Trial finished: trialID=6 state=Complete evaluation=73.956076
x1: 2.1816038398071225 x2: -6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=7 state=Complete evaluation=1.032980
x1: 5.493084783719606 x2: -10
2020/04/10 15:31:12 [INFO] Trial finished: trialID=8 state=Complete evaluation=37.201641
x1: 5.741479126079884 x2: 6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=9 state=Complete evaluation=134.998666
x1: -9.740032298308943 x2: -10
2020/04/10 15:31:12 [INFO] Trial finished: trialID=10 state=Complete evaluation=162.828358
x1: 2.2792673422777243 x2: -10
2020/04/10 15:31:12 [INFO] Trial finished: trialID=11 state=Complete evaluation=25.077990
x1: 2.2126458217189837 x2: -10
2020/04/10 15:31:12 [INFO] Trial finished: trialID=12 state=Complete evaluation=25.045218
x1: 0.6458624077556872 x2: -2
2020/04/10 15:31:12 [INFO] Trial finished: trialID=13 state=Complete evaluation=10.833689
x1: -3.9575192647507818 x2: -6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=14 state=Complete evaluation=36.492036
x1: 8.004427986605243 x2: -6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=15 state=Complete evaluation=37.053155
x1: -4.978326364027314 x2: -6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=16 state=Complete evaluation=49.697039
x1: 6.492831811202192 x2: -2
2020/04/10 15:31:12 [INFO] Trial finished: trialID=17 state=Complete evaluation=29.185538
x1: -4.416872137811545 x2: -6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=18 state=Complete evaluation=42.176248
x1: 0.4494618027117787 x2: -6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=19 state=Complete evaluation=3.404169
x1: 5.7883043457126835 x2: 2
2020/04/10 15:31:12 [INFO] Trial finished: trialID=20 state=Complete evaluation=63.351250
x1: -0.3975975133423404 x2: -2
2020/04/10 15:31:12 [INFO] Trial finished: trialID=21 state=Complete evaluation=14.748474
x1: 4.812367284646379 x2: -2
2020/04/10 15:31:12 [INFO] Trial finished: trialID=22 state=Complete evaluation=16.909410
x1: -4.699298433424429 x2: -6
2020/04/10 15:31:12 [INFO] Trial finished: trialID=23 state=Complete evaluation=45.880600
x1: 5.58418573643919 x2: 2
2020/04/10 15:31:12 [INFO] Trial finished: trialID=24 state=Complete evaluation=61.846387
2020/04/10 15:31:12 Best evaluation=1.032980 (x1=2.181604, x2=-6)

CMA-ES

(venv) $ go run _examples/simple_tpe/main.go 
x1: 8.903922985882328 x2: 6
2020/04/10 15:31:55 [INFO] Trial finished: trialID=0 state=Complete evaluation=168.664153
x1: 2.754448402616003 x2: 2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=1 state=Complete evaluation=49.569192
x1: 4.044133009080397 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=2 state=Complete evaluation=13.178480
x1: -3.7754691329714474 x2: 2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=3 state=Complete evaluation=82.356044
x1: -6.05170080870474 x2: 2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=4 state=Complete evaluation=113.829886
x1: -1.0337193942946883 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=5 state=Complete evaluation=18.203453
x1: 4.03338489973466 x2: 2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=6 state=Complete evaluation=53.134654
x1: 4.672902846195324 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=7 state=Complete evaluation=16.144410
x1: 0.2765486876871135 x2: 2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=8 state=Complete evaluation=51.970284
x1: -2.8620637598332856 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=9 state=Complete evaluation=32.639664
x1: -2.9692920226408988 x2: 2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=10 state=Complete evaluation=73.693863
x1: 4.1179364131180165 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=11 state=Complete evaluation=13.485655
x1: 1.4563071210423104 x2: 2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=12 state=Complete evaluation=49.295602
x1: 4.242080593974547 x2: -6
2020/04/10 15:31:55 [INFO] Trial finished: trialID=13 state=Complete evaluation=6.026925
x1: 7.163918320291986 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=14 state=Complete evaluation=35.666052
x1: 0.7729490955807687 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=15 state=Complete evaluation=10.505654
x1: -0.8307715079884153 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=16 state=Complete evaluation=17.013267
x1: 8.959392196047002 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=17 state=Complete evaluation=57.433140
x1: 7.879445184099316 x2: -6
2020/04/10 15:31:55 [INFO] Trial finished: trialID=18 state=Complete evaluation=35.567876
x1: 2.3146902576468156 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=19 state=Complete evaluation=9.099030
x1: 0.5717963153721013 x2: -6
2020/04/10 15:31:55 [INFO] Trial finished: trialID=20 state=Complete evaluation=3.039766
x1: 4.540374507703152 x2: -6
2020/04/10 15:31:55 [INFO] Trial finished: trialID=21 state=Complete evaluation=7.453503
x1: -0.7001277380146882 x2: -2
2020/04/10 15:31:55 [INFO] Trial finished: trialID=22 state=Complete evaluation=16.290690
x1: 7.296515549479219 x2: -6
2020/04/10 15:31:55 [INFO] Trial finished: trialID=23 state=Complete evaluation=29.053077
x1: 6.8267295312545055 x2: -6
2020/04/10 15:31:55 [INFO] Trial finished: trialID=24 state=Complete evaluation=24.297318
2020/04/10 15:31:55 Best evaluation=3.039766 (x1=0.571796, x2=-6)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant