Skip to content

Update/Fix for the Issue: Not able to scrape jobs list because of depreciated selenium function #190

@HaniaArif

Description

@HaniaArif

Code used:

  from linkedin_scraper import JobSearch, actions
  from selenium import webdriver
  
  driver = webdriver.Chrome()
  email = "some-email@email.address"
  password = "password123"
  actions.login(driver, email, password) # if email and password isnt given, it'll prompt in terminal
  input("Press Enter")
  job_search = JobSearch(driver=driver, close_on_complete=False, scrape=False)
  # job_search contains jobs from your logged in front page:
  # - job_search.recommended_jobs
  # - job_search.still_hiring
  # - job_search.more_jobs
  
  job_listings = job_search.search("Machine Learning Engineer") # returns the list of `Job` from the first page

Error:

Traceback (most recent call last):
File "C:/Users/mz/Downloads/linkedin_test.py", line 25, in
job_listings = job_search.search("Machine Learning Engineer") # returns the list of Job from the first page
File "C:\Users\mz\AppData\Local\Programs\Python\Python310\lib\site-packages\linkedin_scraper\job_search.py", line 89, in search
job = self.scrape_job_card(job_card)
File "C:\Users\mz\AppData\Local\Programs\Python\Python310\lib\site-packages\linkedin_scraper\job_search.py", line 39, in scrape_job_card
company = base_element.find_element_by_class_name("artdeco-entity-lockup__subtitle").text
AttributeError: 'WebElement' object has no attribute 'find_element_by_class_name'

Solution:

In linkedin_scraper\job_search.py do the following updates:
At line 39 and 40 change:
company = base_element.find_element_by_class_name("artdeco-entity-lockup__subtitle").text
location = base_element.find_element_by_class_name("job-card-container__metadata-wrapper").text
to this:
company = base_element.find_element(By.CLASS_NAME,"artdeco-entity-lockup__subtitle").text
location = base_element.find_element(By.CLASS_NAME,"job-card-container__metadata-wrapper").text

Result:

Code works and scrapes data for the job as follows:
[<Job (Senior) Software Engineer – ML/DL Algorithms for Autonomous Driving (m/f/d) Embedded AI Solutions>, <Job Computer Vision Engineer Motive>, <Job Senior iOS Developer Coder Crew>, <Job AI-Augmented Software Developer, Trilogy (Remote) - $200,000/year USD Crossover>, <Job ML Engineer/Data Scientist Turing>, <Job Machine Learning/Data Science Engineer Arbisoft>, <Job Full Stack Developer Voyager Global Mobility>, <Job Principal AI Engineer CureMD>, <Job Big Data Engineer Daraz>, <Job Senior Software Engineer Stewart Pakistan Private Limited>, <Job Machine Learning Engineer Turing>, <Job Senior Data Engineer Systems Limited>, <Job Senior Solutions Engineer, ML/AI DigitalOcean>, <Job DevOps Engineer A4767 Nisum>, <Job Lead Data Engineer Teradata>, <Job Senior Big Data Engineer Daraz>]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions