How do I assess whether the accuracy before and after dedup

[prozzzzzz]

https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py
After I finish executing this file, I have two folders: deduplicated_output and removed. How do I determine which of the removed samples corresponds to which of the retained samples

[guipenedo]

you can enable the option to save cluster_ids, which will then allow you to map the kept documents to their duplicates