Add feature to exclude the same http response

[0xAwali]

while scanning wide scope there are a lot of urls have the same base code so they have the same http response so it's time to let httpx exclude these urls based on comparing every http response with others

[ibitebyt3s]

The anew tool might be a good fit for this only as a standalone program.

Wanted to suggest using its code in this project at first but I can't remember 100% if this was true since it happend about 2 years ago but anyway.

When using httpx with the -tech-detect option there will be times when a technology isn't detected in the same host, thus making anew useless in this case.

Lets say that in the first run it detected [php, Wordpress] and in the second run it detected only [php]. Using anew to check for duplicates will still keep duplicate hosts if their technologies detected are diffrent. Same could happen with other CLI options like -ip

[ehsandeep]

More ideas - https://twitter.com/har1sec/status/1600445181115109377

when -path is used, make a request to / to get number of lines and words and use this as a baseline to filter response (quick custom 404 filter)

[dogancanbakir]

How about something based on vision clusters? This can be deduplication of records in a cluster based on a threshold.

@Mzack9999 fyi

[Xitro01]

Just saw this getting included in httpx. Only thing I'm afraid of, is that it is going to exclude a lot of potential targets.
How are these duplicates checked? Currently I do word count, but also check if it's the same IP, then the chances are much slimmer that you miss something.

[dogancanbakir]

@Xitro01 We use simhash to do deduplication, and it's optional. See: https://github.com/projectdiscovery/httpx/pull/1951/files

[Xitro01]

@Xitro01 We use simhash to do deduplication, and it's optional. See: https://github.com/projectdiscovery/httpx/pull/1951/files

Thanks for the response, yes this is probably going to skip a lot of stuff that might be interesting.
Imo the IP addresses should be compared first, before doing response/page comparison.

[dogancanbakir]

@Xitro01 Thanks for the suggestion. Would you be willing to create a new issue to share your ideas? It could be a good addition to what we added.

[Xitro01]

@Xitro01 Thanks for the suggestion. Would you be willing to create a new issue to share your ideas? It could be a good addition to what we added.

#2014