deduplicate based on url

[ghost]

When i set: crawler.queue({skipDuplicates:true}), (which is supposed to be true by default.)
According to the doc: " The request is considered to be the same if URL, userAgent, device, and extraHeaders are strictly the same.",
is there a way to deduplicate based only on the URL?
I only need to export URLs of a certain domain.

[Minyar2004]

same issue for me

[kulikalov]

Hey @Devhercule! Could you elaborate a little?

userAgent, device, and extraHeaders are not changing randomly, so the only thing that is new on every page is the URL. So, I don't see issues here

[kulikalov]

closing due to inactivity