Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improved SemanticScholar search #96

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Improved SemanticScholar search #96

wants to merge 6 commits into from

Conversation

r-wrobel
Copy link

@r-wrobel r-wrobel commented May 3, 2024

SemanticScholar:

  • Added improved search presented at SemanticScholar search optimization #94 (fastsearch_semanticscholar())
  • JSON import load_semanticscholar_json()
  • Two functions to get a list of all referenced docs generate_reference_list() and download the metadata mass_fetch_semanticscholar()
  • Bugfix in crossref.py
  • Temporary bugfix (?) in types.py: Skip None values when matching ids. Bug is similar to DocumentIdentifier.matches() is case-sensitive #95: Detected multiple paper, which were matched as references, because the only id beside title was None.

Wrobel added 6 commits May 3, 2024 17:02
Improve Query functions
923ab9e 
* Add PubMed to id
* Add ability to pass extra parameters to query (request_query and request_paper). This will be needed for new functions.
* Add missing timeout to request_query
* Exclude "Too Many Requests" and "Endpont request timed out" Errors from raising an Exception and return magic-string "TIMEOUT". search_semantic will retry with increasing interval until success or other error type.
Add function to load json file
ef491d3
Add function to search 1,000 docs in 10 requests instead of 1,000+
92a9692
Add functions to download metadata for referenced docs
0aedf85
Bugfix
f0c0bc6
Ignore None values when matching identifiers
253e918
@r-wrobel r-wrobel mentioned this pull request May 3, 2024
Open
@stijnh
Copy link
Member

stijnh commented May 17, 2024

Thanks for the PR. This looks very promising! The SemanticScholar backend definitely needed some more work!

It seems to be many small improvements, I will need to go over the changes carefully so that may take some time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@r-wrobel @stijnh