When resume NelderMead optimization, occur ValueError

[KanaiYuma-aist]

Describe the bug
When resume neldermead optimization, occur ValueError and stop aiaccel.

To Reproduce

1. Executed aiaccel using below config file.

generic:
  workspace: "./work"
  job_command: "python user.py"
  batch_job_timeout: 14400

resource:
  type: "abci"
  num_node: 6

ABCI:
  group: "[group]"
  job_script_preamble: "./job_script_preamble.sh"
  job_execution_options: ""

optimize:
  search_algorithm: "nelder-mead"
  goal: "minimize"
  trial_number: 100
  rand_seed: 42
  parameters:
    -
      name: "lr"
      type: "uniform_float"
      lower: 0.0001
      upper: 0.1
    -
      name: "weight_decay"
      type: "uniform_float"
      lower: 1.0e-6
      upper: 1.0e-3

2. An unexpected error (over the file capacity limit) occurred and aiaccel stopped.
3. Executed aiaccel using resume option.
4. Occured ValueError and stoped aiaccel.

Screenshots
console

log
error_NM.log

Desktop (please complete the following information):

• OS: linux (ABCI)

Additional context
sample work directory
work_sample.tar.gz

[KanaiYuma-aist]

I assume that the fyr_order: is partially an array, which is the cause of the value error.
e.g. [[1, 2], [1, 2], 1, 3, 2, 3, 3, 3, 2, 2, 2, 1, 3, 3, 2]

The above array is passed to np.int64 and a value error occurs.

aiaccel/aiaccel/optimizer/nelder_mead/sampler.py

Lines 785 to 788 in f243f92

	elif type(v) is int:
	history[key].append(
	np.int64(dict_objects['_history'][key])
	)

Not sure why it's only partially arrayed.

[KanaiYuma-aist]

When the optimization is resumed twice, fyr_order: becomes a two-dimensional array in the first resume.
During the aiacell run, optimizer.yaml in the state directory stores a part of the fyr_order: as a two-dimensional array.
Then, on the second resume, the 2-dimensional array is passed to np.int64, and a value error occurs.

aiaccel/aiaccel/optimizer/nelder_mead/sampler.py

Lines 776 to 790 in f243f92

	for v in dict_objects['_history'][key]:
	if type(v) is list:
	history[key].append(
	np.array(dict_objects['_history'][key])
	)
	elif type(v) is float:
	history[key].append(
	np.float64(dict_objects['_history'][key])
	)
	elif type(v) is int:
	history[key].append(
	np.int64(dict_objects['_history'][key])
	)
	else:
	history[key].append(dict_objects['_history'][key])

In the above code, the target to append to history[key] is dict_objects['_history'][key], which may be the reason why fyr_order: is a 2-dimensional array.
When changed the target of append to v in all three places above(np.array(v), np.float64(v), np.int64(v) ), the value error did not occur on the second resume.

@tkokada @aramoto99
I would like to confirm the specification of fyr_order:.
Is it correct in the specification that fyr_order: becomes a two-dimensional array?
Also, does changing the target of append to v cause a problem in terms of the specification?

---

最適化を2回resumeすると、1回目のresumeでfyr_order:が2次元配列になります。
aiacell実行中、stateディレクトリのoptimizer.yamlには、fyr_order:の一部が2次元配列のまま保存されます。
そして、2回目のresumeで2次元配列がnp.int64に渡されてvalue errorになっている様子です。

上記コード中で、history[key]にappendする対象がdict_objects['_history'][key]になっていることが
fyr_order:が2次元配列になっている原因と思われます。
appendする対象を上記3箇所全てvに変更した所(np.array(v), np.float64(v), np.int64(v) )、2回目のresumeでvalue errorになりませんでした。

fyr_order:の仕様について確認したいのですが、
fyr_order:が2次元配列になることは仕様上正しい挙動ですか？
また、appendする対象をvに変更することは仕様上問題が発生しますか？