-
Notifications
You must be signed in to change notification settings - Fork 868
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error page classifier #1245
Error page classifier #1245
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm - commented out training logic to reduce embed occupation. We will need to move it the utils repository and automate periodic training via GH action
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dogancanbakir This is an interesting feature, can you add a small document about what this feature means to the user and its working as well when best to use?
Also, we can write information to filtered_error_page.json
of what is being filtered when -fep
option is used for further review/testing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Merge conflict
- Let's check if we can move the bayesian classifier agnostic implementation to
utils/mlutils
and keep here only the loading with weights (with commented out code) - Optional: unless we are going to implement a periodic scrape + train GH action for now I think static stored weights from our latest training suffice
I have submitted a PR (projectdiscovery/utils#208) which, once merged, can be utilized in this implementation.
I think we can do the latter; using stored weights from our latest training is fine. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's cleanup the code and reuse projectdiscovery/utils#208 (merged)
This PR adds an error page classifier and error page filtering support. Closes #1201.
w/o filtering:
w/ filtering: