temperature in kirch

[nivancat]

Hi! I had a question regarding order of the operations in your implementation of the maryland scheme. Why do you first add delta and then do temperature? Doesnt it contradict 4.2 in the original paper about effect of delta on the perplexity (see first two paragraphs)

[pierrefdz]

Hi!
The temperature is usually applied during the softmax, so the delta needs to be applied before, no? You suggest to first divide the logits by the temperature, then add the delta, then softmax?

[nivancat]

The scheme you describe breaks the 4.2 of the original paper in that it changes the perplexity

[pierrefdz]

Hi, could you be more precise?

[nivancat]

Quote from the paper Sec 4.2 page 5 (https://arxiv.org/pdf/2301.10226)

"A soft watermark has very little impact on the perplexity of
tokens with extremely high or low entropy. When the distribution produced by the language model is uniform (maximal
entropy), the randomness of the green list results in tokens
being uniformly sampled, and the perplexity remains untouched. Conversely, in the case of minimal entropy, where
all probability mass is concentrated on a single token, the
soft watermark rule has no effect and there is once again no
impact on perplexity."

[pierrefdz]

Yes, I've read this, but I don't understand your comment. Can you explain what you mean by "Doesnt it contradict 4.2 in the original paper about effect of delta on the perplexity (see first two paragraphs)"

[nivancat]

If you were to compare both constructions which one would have no impact on the final perplexity

[sdathath]

Yes, I've read this, but I don't understand your comment. Can you explain what you mean by "Doesnt it contradict 4.2 in the original paper about effect of delta on the perplexity (see first two paragraphs)"

Consider a scenario when you first watermark and then do top-k, and you measure perplexity with respect to the original model (which includes top-k as part of the model). If a non top-k token gets pushed into the top-k as a consequence of watermarking, your perplexity will be infinite.... so that theorem does not hold, and the implementation directly contradicts 4.3 in the original paper, which bounds the change in perplexity under the original unwatermarked model.

[pierrefdz]

Yes, I've read this, but I don't understand your comment. Can you explain what you mean by "Doesnt it contradict 4.2 in the original paper about effect of delta on the perplexity (see first two paragraphs)"

Consider a scenario when you first watermark and then do top-k, and you measure perplexity with respect to the original model (which includes top-k as part of the model). If a non top-k token gets pushed into the top-k as a consequence of watermarking, your perplexity will be infinite.... so that theorem does not hold, and the implementation directly contradicts 4.3 in the original paper, which bounds the change in perplexity under the original unwatermarked model.

You can look at the implementation of the original authors here

https://github.com/jwkirchenbauer/lm-watermarking

[sdathath]

Yes, I think that the implementation is inconsistent with the theory presented as far I understand (which is what the OP seems to be hinting towards).