Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add stash scraper type #269

Merged
merged 6 commits into from
Dec 21, 2019
Merged

Add stash scraper type #269

merged 6 commits into from
Dec 21, 2019

Conversation

WithoutPants
Copy link
Collaborator

Adds stash as a scraper type. This allows one to scrape performer and scene metadata from another remote stash instance.

Here is an example yaml config for a stash scraper:

name: stash
performerByName:
  action: stash
performerByFragment:
  action: stash
sceneByFragment:
  action: stash
stashServer:
  url: http://username:password@stashaddress:9999

Allows searching performer by name and scene by fragment. sceneByFragment is implemented by searching by scene checksum.

@bnkai
Copy link
Collaborator

bnkai commented Dec 16, 2019

Performers scrape seem to work ok.

For the scenes i found the following from a quick test

  • details and date seem ok
  • title,rating,url are not updated even though the scene was found and had field data
  • tag,studio,performers are getting the scraper's id ok but if the client has different name already for the same id we end up with wrong tag,studio,performer
  • if you scrap a scene and a result is found then on save you get Error: GraphQL error: FOREIGN KEY constraint failed even if nothing seems changed in the UI.
    EDIT this seems to be related to the above ,it happens if the scraper has tags , or studios or performers or anything other with an ID that doesn't exist in the client

@bnkai bnkai added the feature Pull requests that add a new feature label Dec 16, 2019
@WithoutPants
Copy link
Collaborator Author

Note that the current implementation does not overwrite fields if they are already populated.

@WithoutPants WithoutPants changed the title Add stash scraper type [WIP] Add stash scraper type Dec 18, 2019
@bnkai
Copy link
Collaborator

bnkai commented Dec 18, 2019

Note that the current implementation does not overwrite fields if they are already populated.

that might explain the title .... but the url and rating were definetely blank on the test i did..
also the title by default gets the filename from the scan , so it will always be populated ...

@WithoutPants
Copy link
Collaborator Author

URL should now be populating correctly. Rating was left off intentionally as I didn't think it should be included by default. In future, I think we should show the difference between the current and scraped values and allow the user to set accordingly.

The IDs of performers, studio and tags should never have been returned by the scrape, as they will seldom match between instances, and the local stash sets the IDs explicitly after matching them. This has now been fixed.

@bnkai
Copy link
Collaborator

bnkai commented Dec 19, 2019

URL is ok now
Performers,studio and tag seem ok too.

A small bug left...
I get this error
Error: GraphQL error: Internal system error. Error <runtime error: invalid memory address or nil pointer dereference>
when the scene is not found and this is the dump.

@Leopere
Copy link
Collaborator

Leopere commented Dec 20, 2019

Can we have the system programmatically generate a clean demo config so that the end-users always have the most accurate example in their installs?

@bnkai
Copy link
Collaborator

bnkai commented Dec 20, 2019

Everything seems fine now ...

@Leopere the config for this PR is an extra file like this ~/.stash/scrapers/stash.yml , it doesn't change anything for the main config file
If you want a clean config file or maybe better a config file sample we can just run stash , tick/fill out all options and hit save i guess...
In due time i think making a config section in the wiki might be more convenient

@WithoutPants WithoutPants changed the title [WIP] Add stash scraper type Add stash scraper type Dec 20, 2019
@Leopere Leopere merged commit f52db4f into stashapp:develop Dec 21, 2019
@WithoutPants WithoutPants mentioned this pull request Feb 4, 2020
10 tasks
@WithoutPants WithoutPants deleted the scrape_stash branch May 15, 2020 07:13
Tweeticoats pushed a commit to Tweeticoats/stash that referenced this pull request Feb 1, 2021
* Add stash scraper type

* Add graphql client to vendor

* Embed stash credentials in URL

* Fill URL from scraped scene

* Nil IDs returned from remote stash

* Nil check
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Pull requests that add a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants