Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CSV importers #2

Open
mauromsl opened this issue Nov 12, 2018 · 2 comments
Open

Improve CSV importers #2

mauromsl opened this issue Nov 12, 2018 · 2 comments
Assignees

Comments

@mauromsl
Copy link
Member

Current csv importers could use some of the following improvements:

  • Schema validation: rather than directly unpacking the csv lines we could have some sort of schema to compare the structure against, and return meaningful errors in case of schema validation.
  • Allow users to download a template with the correct headers in .xls or .csv formats
  • Rewrite the html form with a Django forms.Form for the file upload and add a field to specify csv delimiter as well as header separator.
  • Abstract the validation an import so it works for all kinds of importers. Then each importer only has to provide a schema and the logic to process the csv
@mauromsl
Copy link
Member Author

mauromsl commented Jul 4, 2024

We have discussed a v1 implementation for this issue that would involve:

  • Moving the validation logic used in Janeway core to callables/functions that can be invoked by importers. This will allow the same logic to be re-used and avoid traps where logic changes across import routines
  • Update the CSV importer to invoke those functions programatically on a per field basis
  • Wrap article processing in an atomic transaction to ensure problems ingesting a row don't leak to the rest of the import procedure.
  • When errors occur, return a generated CSV with just the faulty rows that have been rolled back.

@mauromsl
Copy link
Member Author

mauromsl commented Jul 4, 2024

Note: Mauro to raise an issue in core and make this depend on it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant