refactor: update Crawler to support selenium>=4.11.0 and simplify it

[anakin87]

Related Issues

• fixes Crawler does not work with selenium>=4.11.0 #5500

Proposed Changes:

With the recent release of Selenium, the Crawler started failing both locally and in Colab.
Thanks to changes recently introduced in Selenium, we can get rid of the webdriver-manager dependency:
now Selenium automatically finds or installs the required drivers.

• I pinned selenium>=4.11.0 (and removed webdriver-manager)
• I changed the crawler initialization to make it work
• I took the opportunity to refactor and simplify the code

How did you test it?

Existing tests (CI), manual tests on Ubuntu and in Colab.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 5764270194

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 81 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.003%) to 46.887%

---

Files with Coverage Reduction	New Missed Lines	%
nodes/connector/crawler.py	81	37.64%

Totals
Change from base Build 5764030079:	0.003%
Covered Lines:	10891
Relevant Lines:	23228

---

💛 - Coveralls

[vblagoje]

PR Analysis

• 🎯 Main theme: Refactoring the Crawler to support selenium>=4.11.0 and simplifying the code
• 📌 Type of PR: Refactoring
• 🏅 Score: 90
• 🧪 Relevant tests added: Yes
• ✨ Focused PR: Yes, the PR has a clear and coherent title and description, and all PR code diff changes are properly derived from the title and description.
• 🔒 Security concerns: No, the PR does not introduce any apparent security concerns. The changes are mainly related to the update of a dependency and the simplification of the code, which do not involve any security-sensitive operations.

PR Feedback

• 💡 General PR suggestions: The PR is well-structured and the changes are well-documented. The removal of the 'webdriver-manager' dependency and the simplification of the Crawler initialization are good improvements. However, it would be beneficial to provide more detailed information on the manual tests conducted on Ubuntu and in Colab, to ensure the changes work as expected in all environments.

[vblagoje]

LGTM @anakin87, let's leave a link to your Colab on this PR

[anakin87]

Colab notebook: https://colab.research.google.com/drive/1qvnKmjLt_OdloVo56QjMcQatYtCqVSsk?usp=sharing