Refactor CSV / XSLX exports to have 1 export format (and code) instead of 3

[stephanegigandet]

Is your feature request related to a problem? Please describe.

We currently have 3 different CSV exports, each with different fields, and with different code:

1. the full database CSV export, generated nightly with scripts/export_database.pl
2. the on demand CSV / XLSX export, available through the search form (code in lib/ProductOpener/Display.pm search_and_export_products()
3. the CSV export used internally to export and import data from the producers platform to the public platform (code in lib/ProductOpener/Export.pm, called from the producers platform, and also availble from export_csv.pl)

Describe the solution you'd like

Gradually converge to 1 common format and 1 single code base (Export.pm).

Desirable features:

• let producers download an export of their data on the producers platform, make some changes to it, and reimport it
• clear separation between columns that contain primary data (e.g. ingredients text, nutrition values, Nutri-Score computed by the producer) vs derived data (canonical ingredients from ingredients_tags, Nutri-Score computed by OFF, Eco-Score details). e.g. maybe prefix the column name with off: off:ingredients_tags

Additional context

Related issues:

• CSV export revision (Adding new fields and removing some) #2325
• Provide a fast and simpler way to get up-to-date CSV file export #1670 : we could put all the possible fields to be available in the full CSV, and then we could have other tools to slice and dice them, or we could store the tabular data in a table in a sql database
• On-demand CSV/XLS/ODS/JSONL/Google Spreadsheet(?) exports for ANY kind of list #4782
• Add support for OpenDocument ODS format for on-demand exports (through search feature) #3923

[stephanegigandet]

My plan is to first start to make the on demand search export on the website use the code from Export.pm. The search export is requested manually, so it's probably not a big issue if we change which columns are included and their names.

Once we have something that we are happy with, we can look into replacing the full database CSV export, probably in a new location, so that we don't break things.

[alexgarel]

Convergence of exports is a real big plus.

When refactoring, I think it would be great to separate processing into map like structure, from rendering to xls.

My only small concern is that for communication between producer and public platform, I would go away from the csv/xls format… because it seems to me that in introduces some tweaks that are not useful.

[github-actions]

This issue is stale because it has been open 90 days with no activity.