Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metadata: fix dojson for virtua records #559

Merged
merged 1 commit into from
Nov 8, 2019

Conversation

rerowep
Copy link
Contributor

@rerowep rerowep commented Oct 11, 2019

  • Fixes dojson transformation errors.
  • Adds data path parameter to setup script.
  • Adds all languages to documents json schema and form.
  • Adds marc21tojson utils cli for parallel dojson transformation.

Co-Authored-by: Peter Weber peter.weber@rero.ch

Why are you opening this PR?

How to test?

  • pipenv run invenio utils marc21tojson data/documents_big.xml big_ok.json big_errors.xml -v -c 100

Code review check list

  • Commit message template compliance.
  • Commit message without typos.
  • File names.
  • Functions names.
  • Functions docstrings.
  • Unnecessary commited files?

@rerowep rerowep added the WIP label Oct 11, 2019
@rerowep rerowep force-pushed the wep-#1069-extract-virtua-bib-records branch 19 times, most recently from c7098a1 to 0a29dc1 Compare October 18, 2019 05:33
@rerowep rerowep force-pushed the wep-#1069-extract-virtua-bib-records branch 10 times, most recently from 4a8a764 to 3e62499 Compare October 21, 2019 04:18
@rerowep rerowep force-pushed the wep-#1069-extract-virtua-bib-records branch 12 times, most recently from a470615 to 66e9e7c Compare November 6, 2019 16:05
@rerowep rerowep requested a review from jma November 6, 2019 16:05
@rerowep rerowep force-pushed the wep-#1069-extract-virtua-bib-records branch from 66e9e7c to ac3fdb2 Compare November 6, 2019 21:39
@jma jma changed the title metadata: fix dojeson for virtua records metadata: fix dojson for virtua records Nov 7, 2019
rero_ils/dojson/utils.py Show resolved Hide resolved
@rerowep rerowep force-pushed the wep-#1069-extract-virtua-bib-records branch 4 times, most recently from c3690c2 to b17c121 Compare November 7, 2019 15:20
Copy link

@BadrAly BadrAly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Commit message missing information about the cli and loan code changes.
  • [not very important] The error files generated after running the setup or validate_json cli are placed by default in $DATA_PATH. Can these files be generated outside of the project? so they do not listed in the project changed files when you run git status.

rero_ils/modules/cli.py Show resolved Hide resolved
@rerowep rerowep force-pushed the wep-#1069-extract-virtua-bib-records branch from b17c121 to db52648 Compare November 7, 2019 17:36
@rerowep rerowep requested a review from BadrAly November 7, 2019 17:36
@rerowep rerowep force-pushed the wep-#1069-extract-virtua-bib-records branch from db52648 to cbe541d Compare November 7, 2019 21:11
* Fixes dojson transformation errors.
* Adds data path parameter to setup script.
* Adds all languages to documents json schema and form.
* Adds marc21tojson utils cli for parallel dojson transformation.
* Adds lazy loading of json records from file.
* Adds exception handlung for record creation.
* Adds pids to documents_big.json and documents.small.json.
* Better loading of loans with exception handling.

Co-Authored-by: Peter Weber <peter.weber@rero.ch>
Co-Authored-by: Gianni Pante <gianni.pante@rero.ch>
@rerowep rerowep force-pushed the wep-#1069-extract-virtua-bib-records branch from cbe541d to 2d93e6b Compare November 7, 2019 21:14
@rerowep
Copy link
Contributor Author

rerowep commented Nov 7, 2019

  • Commit message missing information about the cli and loan code changes.
  • [not very important] The error files generated after running the setup or validate_json cli are placed by default in $DATA_PATH. Can these files be generated outside of the project? so they do not listed in the project changed files when you run git status.

I have added commit message for the loan. The cli was already in the commit messages.
If you change the $DATA_PATH the error files will not be in the git repository. Otherwise a additionally cli parameter has to be implemented.

@rerowep rerowep merged commit 64b0c0f into rero:dev Nov 8, 2019
@rerowep rerowep deleted the wep-#1069-extract-virtua-bib-records branch November 13, 2019 07:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants