Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Scanner + People Pages #3286

Merged
merged 17 commits into from
Oct 23, 2024
Merged

New Scanner + People Pages #3286

merged 17 commits into from
Oct 23, 2024

Conversation

majora2007
Copy link
Member

@majora2007 majora2007 commented Oct 16, 2024

This PR brings a brand new scanner to Kavita with a focus on reliability and tuned to help support our larger comic hoarders. The scanner now works on a system of partial information and has some areas parallelized when there is enough data for it to make sense. This also takes a major stab at fixing the FK issue in the scanner and while it's not fixed, it's reduced in likelihood.

In addition, special detection has been slimmed down to just files with SP marker and specials can be flat in the series folder or ideally in Specials/ folder. This is something that I have had an open issue for many months and it was finally time to make the change. While I will miss it for the cases it did work, I'm sure many users will enjoy not having some series claimed as a special due to false positives.

Lastly, a major rework of People is landing with this PR. All people will be removed from your series on updating and a force scan will be needed to get them added back. People are now entity based and that means we can have more rich metadata for each person available. This is the beginning of richer metadata around people (including characters) and plans are currently being architected to bring people images to servers (you can manually upload them via UI/API currently).

I'd like to thank cducharme, Morsok, Robbie, Diesel, and Horo for help in the 3 weeks of extended testing to get this new scanner ready for the nightly users. This PR is a massive undertaking of not only rewriting how the scanner works, but also redoing everything around People.

image
image
image

Added

Changed

  • Changed: Optimized a number of methods within the Scanner to reduce memory and CPU time
  • Changed: Moved a lot of the chapter scanner logging to Trace.
  • Changed: Scanner can now choose to parallel parse files when there are over 100 in a directory
  • Changed: Changed how detection and scanning of dirty directories works. The scanner will now parse bottom-up to reduce any potential misses and avoid different layouts working differently. This has extra I/O checks but much greater reliability and should reduce the amount of work needed to ingest changes after the first scan.
  • Changed: LocalizedSeries merging with Series is now done at a higher level and performs much better (in terms of reliability)
  • Changed: People are now entity based which allows for a richer experience in Kavita.
  • Changed: Clicking on any person in Kavita will now open their dedicated page which showcases their different works (feedback needed).
  • Changed: (Performance) Some large refactors to the scanner to help with the FK bug. The scanner should be slower but consume much less memory, especially for well tagged systems.
  • Changed: Kavita no longer parses keywords from filenames and groups as a Special. The way to mark a special going forward will be explicitly with SP01, SP02, etc. Specials can be flat in the series folder or ideally in Specials/ folder. (Closes Special Keyword Removal #2967)
  • Changed: Changed logging level on some of the loggers that could become noisy

Fixed

  • Fixed: Fixed a bug where Series Cover image could choose a Volume 0 instead of Volume 1
  • Fixed: Fixed a bug where teams and locations wouldn't trigger updating people on update series metadata api
  • Fixed: Fixed a bug where calculating the Lowest Folder for a Series could be incorrect when there are nested folders. This includes a migration to clear out existing entries to avoid scan series not seeing files.
    Example:
    / <- Lib root assume
    /love hina/love hina/v01.cbz
    /love hina/specials/sp01.cbz
    The lowest series folder was /love hina/love hina/ for some reason, meaning series scan wasn't getting the full series (all folders) scanned in.
  • Fixed: Fixed up a check for using lowest folder path to lookup from a Folder Watcher task. This should find series much better from a lower file path.

API

  • Search now returns a list of PersonDto instead of strings for People results.

@majora2007 majora2007 merged commit ba20ad4 into develop Oct 23, 2024
@majora2007 majora2007 deleted the canary branch October 23, 2024 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

Refactor People to be Entity Focused Special Keyword Removal
1 participant