This repository is established to explore the data on 51Job website, where a number of companies post their wanted positions, and at the same time employees could share their own profiles to boost their career development. Overall, the work in this repo could be summarized in following aspects:
- Collect the job information using python crawler.
- Preprocess the data (clean, discretize, match, normalize, etc).
- Conduct feature engineering to analyse the data.
- Design two tasks for real scenarios (salary and job type prediction).
- Apply various machine learning algorithms to our tasks.
The documents of our work are available here: [report], [notebook].
- scrapy (web crawling)
- numpy and pandas (data preprocessing)
- scikit-learn (ML-algorithms)
- matplotlib and seaborn (data visualization)
We use scrapy to crawl raw data from 51Job website. See the directory /job51spider for codes. XPath is used to parse the html and extract data information.
After entering the directory, input the command in cmd.exe to run the spider.
scrapy crawl 51job
We use python libraries pandas (using class dataframe) and re to preprocess the raw data. See /preprocess/preprocess.py for code. You can find the preprocessed data in /data, where middleData.csv is the preprocessed data suitable for drawing pics, while quantityData.csv quantifies all data and fits further data analysis.
See directory /pics.
We analyzed feature coorelation and feature distribution respectively. We found two some main features which affect salary level: education level requirements, work experience requirements and area location.
| Model | R2 value | Mean Error ￥/year | time / s |
|---|---|---|---|
| Ada-Boost | 0.2350 | 37483.9 | 0.79 |
| Grad-Boost | 0.3237 | 34031.4 | 3.13 |
| SVR (RBF) | 0.0092 | 43301.9 | 350.08 |
| Bayesian Ridge | 0.2667 | 34031.4 | 0.05 |
| Elastic Net | 0.0426 | 44784.4 | 0.03 |
| MLPs | 0.2682 | 36207.3 | 19.29 |
| Model | Accuracy / % | time / s | Model | Accuracy / % | time / s |
|---|---|---|---|---|---|
| LP | 7.79% | 3.89 | MLP | 20.91% | 20.05 |
| GNB | 7.32% | 0.23 | SVM | 29.31% | 1032.75 |
| KNN | 25.19% | 2.60 | XGBoost | 27.53% | 303.21 |
| RF | 28.44% | 1.80 |
The accuracy & time plot of the above models:
