-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support crawling websites behind a login #202
Comments
Thanks for the feature request! Implementing something like this would require really changing the entire way the library works. This would mean moving from stateless HTTP requests that are driven from node, to using something like puppeteer to make the request and maintain state. I don't have immediate plans to implement something like this, but I'd be happy to talk through a PR proposal. |
what about exposing the request so headers et cetera could be set?
|
That's probably something that could be pretty easily done. If we did it though, I would only want to support it via the API (not the CLI). Would that get you where you wanna go? |
Actually it turned out that the issue I was trying to solve was nothing to do with missing headers. That said, I still think that exposing the request like this would make the package more versatile and I think it would benefit a lot of users. |
This could be easily solved by allowing cookies to be sent with every request. Either allow a cookies.txt file (e.g. linkchecker/linkchecker supports this) or allow cookies to be set from the config file or via the CLI. Requests are still stateless - you just send the cookies with every request. |
It would be great if the initial request could be a http-post with additional fields like username/password etc. After the initial post linkinator should handle the http-session (like receive/sending a session cookie) and crawl all other links via get-method (like it does already)
That way one could check links even if behind a login
The text was updated successfully, but these errors were encountered: