issues when run disinformation

[lumosity4tpj]

when I run the disinformation like run_specs.conf#L538-L540, I have two issues：

• capability=reiteration：raise helm.benchmark.executor.ExecutorError: HuggingFace error: num_return_sequences has to be 1, but is 5 when doing greedy search. so the settings don't match, this problem also occurs when run real_toxicity_prompts.
• capability=wedging：raise ValueError: More than one stop sequence is not supported. I find this to define stop_sequence as a list of two lengths, but this only allows one length.

When I looked at the corresponding references this this I still wondered how to solve these problems.

[lumosity4tpj]

when I run the disinformation like run_specs.conf#L538-L540, I have two issues：

• capability=reiteration：raise helm.benchmark.executor.ExecutorError: HuggingFace error: num_return_sequences has to be 1, but is 5 when doing greedy search. so the settings don't match, this problem also occurs when run real_toxicity_prompts.
• capability=wedging：raise ValueError: More than one stop sequence is not supported. I find this to define stop_sequence as a list of two lengths, but this only allows one length.

When I looked at the corresponding references this this I still wondered how to solve these problems.

• For the first point, I'm sorry that it was caused by my mistake in choosing the wrong generate mode of huggingface, so it is not a problem;
• For the second point, I think that if the length>1, it should be optimized to not pass the stop_sequence as eos_token, but should return the result when the maximum length or the eos of the original tokenizer is reached, and judge whether to truncate outside the model.generate. Is the reason we don't do this because we only want stop sentence as eos and ignore the original tokenizer's eos?
• Moreover, I found that some tokenizers will tokenize stop_sequence (length==1) tokenize into two ids, then this will report an error, which can also be solved by the above method.

If you would give me some answers, I would really appreciate it

[yifanmai]

This looks like a duplicate of #1501 which was fixed in #1892 i.e. HuggingFaceClient failing if the stop sequence is longer than one token.