Blocking URLs when Traversing Links and Parsing HTML

[carloseduardobanjar]

I'm encountering a challenge while utilizing requests_html for web scraping tasks. My requirement entails traversing a large number of links and extracting specific HTML elements from each page. However, I consistently encounter blockages with certain URLs (the code seems to halt -- not in the sense of terminating execution, but rather it appears to be waiting for something to proceed), which leads me to believe that the issue might be related to requests_html's functionality.

For further context and insights into the issue, please refer to the following resources:

• Stack Overflow Question: Python Script Stuck in a For Loop

• GitHub Repository: nvd-linked-content-crawler-bug