No updating on the ref_model?

[hdvvip]

Hi, thanks for the great repo,

I wonder that in the original PPO paper, the authors update the ref_model to become the fine-tuned model after every iteration (theta_old = theta). I attached the image below for your convenience.

So, there must be a code line where ref_moel = model?
Why didn't you update your ref_model as shown in the original PPO paper?
Thank you.

[hdvvip]

Ok I understood, you used logprob of the current network as theta_old

train_stats = self.train_minibatch(logprobs[idx].unsqueeze(0), values[idx].unsqueeze(0),
rewards[idx].unsqueeze(0), queries[idx].unsqueeze(0),
responses[idx].unsqueeze(0),
torch.cat([queries[idx],responses[idx]]).unsqueeze(0))

This works similarly to update theta_old after every iteration.