Crawler does not work with selenium>=4.11.0

[anakin87]

Describe the bug
Due to recent changes in selenium, the Chromium driver can't be found though it is properly installed.
Related issue: SeleniumHQ/selenium#12466

Error message

NoSuchDriverException Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/haystack/nodes/connector/crawler.py in init(self, urls, crawler_depth, filter_urls, id_hash_keys, extract_hidden_text, loading_wait_time, output_dir, overwrite_existing_files, file_path_meta_field_name, crawler_naming_function, webdriver_options)
122 try:
--> 123 self.driver = webdriver.Chrome(service=Service("chromedriver"), options=options)
124 except WebDriverException as exc:

5 frames
NoSuchDriverException: Message: Unable to locate or obtain driver for chrome; For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/driver_location

The above exception was the direct cause of the following exception:

NodeError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/haystack/nodes/connector/crawler.py in init(self, urls, crawler_depth, filter_urls, id_hash_keys, extract_hidden_text, loading_wait_time, output_dir, overwrite_existing_files, file_path_meta_field_name, crawler_naming_function, webdriver_options)
123 self.driver = webdriver.Chrome(service=Service("chromedriver"), options=options)
124 except WebDriverException as exc:
--> 125 raise NodeError(
126 """
127 'chromium-driver' needs to be installed manually when running colab. Follow the below given commands:

NodeError:
'chromium-driver' needs to be installed manually when running colab. Follow the below given commands: ...

Expected behavior
The crawler should find the driver and properly work.

Additional context
As a workaround, selenium can be downgraded: pip install "selenium<4.11.0"
We should support the new version and make it work. (It doesn't seem like a difficult change...)

To Reproduce

from haystack.nodes import Crawler

crawler = Crawler(output_dir="crawled_files")

System:

• OS: Ubuntu 22.04
• Haystack version (commit or version number): 1.19.0