Support string values for categorical parameter in Grid Suggestion

[andreyvelich]

/kind feature

User can't specify string values for Grid suggestion in Categorical parameter.
Currently, our Chocolate suggestion supports only int or float values for the list in Categorical parameter. See here: https://github.com/kubeflow/katib/blob/master/examples/v1alpha3/grid-example.yaml#L37.

User should be able to specify any values in the list.

We can convert string values from the list to integers and save them in Chocolate DB.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.97

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[StefanoFioravanzo]

Hey @andreyvelich I was testing some grid experiments and this has been my experience:

• Running a grid experiment with one int argument, one double argument and one categorical argument (with integers as strings), worked as expected. The trials were generated and the search space properly explored.
• When running a grid experiment with just one argument, of type categorical, just 2 trials get generated and then the experiment hangs. The suggestions pod produces this log Chocolate db is exhausted, increase Search Space or decrease maxTrialCount!, even though there are still some values in the list that have not been tried.

So I don't get what is the underlying behaviour, can you explain why this is happening?

[andreyvelich]

Hey @andreyvelich I was testing some grid experiments and this has been my experience:

• Running a grid experiment with one int argument, one double argument and one categorical argument (with integers as strings), worked as expected. The trials were generated and the search space properly explored.

Yes, for me it works as well. Only problem with the strings values. As I said above, we can convert these values to appropriate list of integer values and match them when we create Assignment for the Trial.

• When running a grid experiment with just one argument, of type categorical, just 2 trials get generated and then the experiment hangs. The suggestions pod produces this log Chocolate db is exhausted, increase Search Space or decrease maxTrialCount!, even though there are still some values in the list that have not been tried.

So I don't get what is the underlying behaviour, can you explain why this is happening?

How many maxTrialCount have you specified? Can you show me your search space also.

This can happen because Chocolate suggestion has generated all possible combination of parameters from the Search Space.
This PR: #1116 should fix some of these problems.

After this PR, inside suggestion sqlite3 DB, you should see all parameters that Chocolate suggestion was generated with correct Chocolate ID (ID of generated set of parameters).

[StefanoFioravanzo]

The maxTrialCount was 30 and two 2 trials were performed. And definitely there were other values to be tried. I will first test again with the new PR included and will report back. Thanks!

[StefanoFioravanzo]

@andreyvelich I tested the new image and indeed the search space is explored even with categorical arguments. Also the logging is much improved!

So, now when the search space is exhausted, but the goal is not reached and also max trial count is not reaches, the suggestions pod still outputs db is exhausted, increase Search Space or decrease maxTrialCount!. It would be ideal to have this being reflected in the status of the experiment. And it should be simple to differentiate between when the maxTrialCount has been reached and when the search space was fully explored. WDYT?

I am willing to help for this and send PR, maybe can you give me some pointers?

[andreyvelich]

/assign @andreyvelich