Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

base setup for mangadex import; handling base cases #550

Open
wants to merge 6 commits into
base: the-future
Choose a base branch
from

Conversation

toyhammered
Copy link
Member

@toyhammered toyhammered commented Aug 10, 2019

What

This feature will import all the data we scraped from mangadex and update our database.
This has the potential to update or create:

  1. Manga
  2. Chapters
  3. Genres
  4. Mappings (either for MyAnimeList if it doesn't exist, and for MangaUpdates
  5. MediaStaff

Why

Still in the process, but the general idea is to copy what we did for the tvdb scheduled job import.

Checklist

  • All files pass Rubocop
  • Any complex logic is commented
  • Any new systems have thorough documentation
  • Any user-facing changes are behind a feature flag (or: explain why they can't be)
  • All the tests pass
  • Tests have been added to cover the new code

@toyhammered toyhammered self-assigned this Aug 10, 2019
app/services/mangadex_import.rb Outdated Show resolved Hide resolved
app/services/mangadex_import.rb Outdated Show resolved Hide resolved
RSpec.describe MangadexImport::Row do
let(:mangadex_data_regular) { JSON.parse(fixture('mangadex_import/manga-batch-1-temp.ndjson')) }
let(:mangadex_data_hentai) { JSON.parse(fixture('mangadex_import/mangadex-hentai.ndjson')) }
# let(:kitsu_data) { JSON.parse(fixture('mangadex_import/kitsu_data/kyou-wa-kaisha-yasumimasu.json')) }

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return if mangadex_data['chapters'].blank?

mangadex_data['chapters'].each do |mangadex_chapter|
kitsu_chapter = kitsu_data.chapters.where(number: mangadex_chapter['chapter']).first_or_initialize

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


mangadex_data['chapters'].each do |mangadex_chapter|
kitsu_chapter = kitsu_data.chapters.where(number: mangadex_chapter['chapter']).first_or_initialize
kitsu_chapter.volume_number ||= mangadex_chapter['volume']
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NuckChorris Are we getting rid of volume_number and just going to be using the Volume association? Should I just find or create a new volume based on the chapter?

@@ -0,0 +1,34 @@
class MangadexImport
class Chapter1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NuckChorris I am having naming conflicts with ActiveRecord Chapter. Unsure how to fix

# NOTE: I don't think this works the way I intended.
# I was thinking that we can just do save at the end and it will save
# the associations.
@kitsu_data.categories << category
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This saves right away is what I realized

'hungarian' => 'hu',
'indonesian' => 'id_in',
'italian' => 'it',
'japanese' => 'en_jp',
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this correct?

'catalan' => 'ca',
'chinese (trad)' => 'zh_Hant',
'chinese (simp)' => 'zh_Hans',
'english' => 'en',
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this also be en_jp because when saving with Chapter it will fail validation...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants