About verification of the effectiveness of the method proposed in this paper

[qibao77]

Since it's a verifiable problem with a known answer, which is better, the complex chain of thought (CoT) production method proposed in this paper or the complex CoT that the model completes based on the answer directly provided?

[jymChen]

Hi @qibao77,
Thank you for your attention!

For this question, I would recommend the production method proposed in the paper, where the model independently attempts multiple times to find the correct solution. While effective, this approach can consume many computational resources or API quotas.

To address scenarios requiring extensive searches, our code extra provide the --efficient_search option in search_for_complex_reasoning_path.py. If the model reaches the maximum number of searchs without success, this option allows it to directly refine the reasoning path based on the provided answer. However, constructing a reasoning path based on the given answer may introduce biases, as the intermediate reasoning steps filled in by the model may potentially contain errors.

[qibao77]

Hi @qibao77, Thank you for your attention!

For this question, I would recommend the production method proposed in the paper, where the model independently attempts multiple times to find the correct solution. While effective, this approach can consume many computational resources or API quotas.

To address scenarios requiring extensive searches, our code extra provide the --efficient_search option in search_for_complex_reasoning_path.py. If the model reaches the maximum number of searchs without success, this option allows it to directly refine the reasoning path based on the provided answer. However, constructing a reasoning path based on the given answer may introduce biases, as the intermediate reasoning steps filled in by the model may potentially contain errors.

Thank you for your reply! You suppose that "intermediate reasoning steps filled in by the model may potentially contain errors", have you conducted some experiments to support it?