Releases: ameenmaali/urldedupe
Releases · ameenmaali/urldedupe
v1.0.4 - Add --no-extensions and --query-strings-only flags
Addition of 2 additional switches for further filtering down
results:
- --no-extensions|-ne will allow you to discard any results that
have extensions (.js, .jpg, .exe) - --query-strings-only|-qs will allow you to discard any results
that do not have query strings
v1.0.3 - Dedupe similar URLs and performance improvements
v1.0.3 Updates
- Several performance and code style improvements thanks to @larskraemer!
- Added
-s|--similar
flag to detect and dedupe similar looking URLs, such as:
https://site.com/api/users/123
https://site.com/api/users/222
https://site.com/api/users/412/profile
https://site.com/users/photos/photo.jpg
https://site.com/users/photos/myPhoto.jpg
https://site.com/users/photos/photo.png
Becomes:
https://site.com/api/users/123
https://site.com/api/users/412/profile
https://site.com/users/photos/photo.jpg
v1.0.2 - Performance upgrades
Performance upgrades to decode method, removing unnecessary vector and final loop, and more.
v.1.0.1 - Use non-regex parsing by default
Adds a new CLI flag for regex parsing, which was previously by default. This is very slow for large data sets, and should only be used when absolute 100% accuracy is needed. In most cases, non-regex parsing is just as accurate and significantly quicker.
v1.0.0 - Initial Release
Create LICENSE