Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crawler V2 Beta is able to merge #471

Merged
merged 97 commits into from
Jun 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
16660ce
Initial commit
Miniast Sep 16, 2023
5dc84e4
Half init.
Miniast Sep 17, 2023
51f3829
test
Sep 18, 2023
fcd71c7
huaji
Sep 21, 2023
b326e7a
fix bug
Sep 25, 2023
7338cf8
change
Sep 25, 2023
0f28b24
fix bug 2
Sep 25, 2023
a34171c
fix bug
Sep 25, 2023
fa645a0
step
Sep 26, 2023
c2841f9
step
Sep 26, 2023
a39c632
step
Sep 26, 2023
e688b17
step
Miniast Sep 27, 2023
acda4e8
step
Miniast Sep 27, 2023
b37873a
step
Miniast Jan 7, 2024
088b66f
step
Miniast Jan 8, 2024
31f1104
step
Miniast Apr 26, 2024
8a6877a
step
Miniast Apr 26, 2024
4be5a43
step
Miniast Apr 28, 2024
5873d66
step
Miniast Apr 28, 2024
9ab3397
Merge pull request #1 from Miniast/preview
Miniast Apr 28, 2024
da17af9
step
Miniast Apr 28, 2024
62e8c54
milestone
Miniast Apr 28, 2024
40f5589
step
Miniast Apr 29, 2024
18ebe93
step
Miniast Apr 29, 2024
a425b63
Merge pull request #2 from Miniast/preview
Miniast Apr 29, 2024
1cd0d5d
step
Miniast Apr 29, 2024
f4e564c
step
Miniast Apr 29, 2024
46bb680
step
Miniast Apr 30, 2024
082f9d0
step
Miniast Apr 30, 2024
6fc0409
Merge pull request #3 from Miniast/preview
Miniast Apr 30, 2024
8276305
step
Miniast Apr 30, 2024
a9805c8
milestone
Miniast Apr 30, 2024
caf4806
step
Miniast May 6, 2024
225246f
step
Miniast May 6, 2024
a870239
milestone
Miniast May 6, 2024
ce447e3
step
Miniast May 6, 2024
53a9986
step
Miniast May 7, 2024
92aa377
step
Miniast May 7, 2024
4da09a1
Merge pull request #5 from Miniast/beta
Miniast May 7, 2024
375fe81
step
Miniast May 8, 2024
0134766
step
Miniast May 8, 2024
932c233
step
Miniast May 8, 2024
0ddb502
step
Miniast May 8, 2024
e2682f8
milestone
Miniast May 8, 2024
2934841
Merge pull request #6 from Miniast/beta
Miniast May 8, 2024
0cf8d3d
rename retryTimeout to retryInterval
Miniast May 8, 2024
bb4b93b
ready
Miniast May 8, 2024
305dd43
fix bug
Miniast May 8, 2024
f872d30
message
Miniast May 8, 2024
4f112f3
proxies step
Miniast May 8, 2024
1e0c37e
ready
Miniast May 8, 2024
212b0e0
test
Miniast May 8, 2024
4b75fae
1.5.1-beta.0
Miniast May 8, 2024
f7019f1
Merge pull request #7 from Miniast/beta
Miniast May 8, 2024
5e2ec9f
Merge branch 'beta' into newbeta
Miniast May 10, 2024
30458cb
step
Miniast May 10, 2024
a04bdd2
step
Miniast May 11, 2024
5dbc46f
step
Miniast May 11, 2024
a3d328c
step
Miniast May 11, 2024
13bb7b6
step
Miniast May 11, 2024
187f6ae
step
Miniast May 11, 2024
b0cfc5a
milestone
Miniast May 11, 2024
7e8ee09
step
Miniast May 11, 2024
d756dbb
step
Miniast May 11, 2024
1009d6a
step
Miniast May 11, 2024
e6fd4cd
step on type
Miniast May 13, 2024
df96adf
step on type
Miniast May 13, 2024
0d8f6cd
step on type
Miniast May 13, 2024
834058e
step
Miniast May 14, 2024
dc3eef5
step docs
Miniast May 14, 2024
9ae3f76
step
Miniast May 14, 2024
ca97675
readme docs
Miniast May 15, 2024
fe92ecd
doc
Miniast May 15, 2024
61dfed0
step
Miniast May 15, 2024
a0a35a8
step
Miniast May 16, 2024
dc974fe
Update README.md
Miniast May 17, 2024
22c6ed1
step
Miniast May 17, 2024
d0f360b
Update README.md
Miniast May 21, 2024
e64ea26
step
Miniast May 27, 2024
458d563
fix
Miniast May 27, 2024
a6fafc4
remove dist
Miniast May 27, 2024
9d2df6f
new version test
Miniast May 27, 2024
11300b8
step
Miniast May 27, 2024
a8c1809
add typescript dependency
Miniast May 27, 2024
7d89970
add lock file
Miniast May 27, 2024
e3fbb2d
step
Miniast May 28, 2024
241f03f
step
Miniast May 28, 2024
18c9d49
Update README.md
Miniast May 31, 2024
8d28bb4
test
Miniast Jun 4, 2024
172aa98
step
Miniast Jun 5, 2024
0a134a5
lint step
Miniast Jun 5, 2024
9d89d78
step
Miniast Jun 5, 2024
1f468ba
fix lint
Miniast Jun 5, 2024
0b54958
fix test
Miniast Jun 6, 2024
f756478
ready
Miniast Jun 6, 2024
7f61668
add ci
Miniast Jun 6, 2024
2d50c45
try to fix ci
Miniast Jun 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,18 @@ jobs:
# You can specify an image from Dockerhub or use one of our Convenience Images from CircleCI's Developer Hub.
# A list of available CircleCI Docker Convenience Images are available here: https://circleci.com/developer/images/image/cimg/node
docker:
- image: cimg/node:14.18.1
- image: cimg/node:18.20.3
# Then run your tests!
# CircleCI will report the results back to your VCS provider.
steps:
# Checkout the code as the first step.
- checkout
- run: |
npm i
npm run hint
npm test
npm run cover
npm install --prefix=$HOME/.local -g pnpm
pnpm install
pnpm eslint
pnpm build
pnpm test
workflows:
# Below is the definition of your workflow.
# Inside the workflow, you provide the jobs you want to run, e.g this workflow runs the build-and-test job above.
Expand Down
2 changes: 0 additions & 2 deletions .coveralls.yml

This file was deleted.

32 changes: 0 additions & 32 deletions .eslintrc.js

This file was deleted.

11 changes: 0 additions & 11 deletions .github/dependabot.yml

This file was deleted.

12 changes: 5 additions & 7 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
.vscode
.idea

dist
node_modules
.DS_Store
.idea/
npm-debug.log
.vscode/
*~
coverage
.nyc_*
archive
9 changes: 9 additions & 0 deletions .npmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
.vscode
.gitignore
.npmignore
node_modules
src
test
CHANGELOG.md
pnpm-lock.yaml
tsconfig.json
19 changes: 0 additions & 19 deletions .travis.yml

This file was deleted.

4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
node-crawler ChangeLog
-------------------------

2.0.0
- Crawler V2 has been released, which is a complete rewrite of the original crawler.
- see [README.md](https://github.com/bda-research/node-crawler/blob/master/README.md) for details.

1.5.0
- fix bug of `:path` and `:authority` header in http/2 (@mike442144)
- add `ciphers` to both http/1.1 and http/2 (@mike442144)
Expand Down
1 change: 0 additions & 1 deletion CNAME

This file was deleted.

14 changes: 0 additions & 14 deletions Dockerfile

This file was deleted.

21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2023 MiniAst

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
7 changes: 0 additions & 7 deletions LICENSE.txt

This file was deleted.

Loading