feat: compute packagings stats

[stephanegigandet]

This is a script to compute some packaging stats for categories of products. #7929

Sample output: https://world.openfoodfacts.org/data/categories_stats/categories_packagings_stats.packagings-with-weights.json
(for products that have some packaging weights)

Currently the stats are only on country / category / shape / material.

We will also add stats for packaging weight.

[alexgarel]

I approve, but I would better like to have the OpenAPI documentation shipped with it (see my comment).

[alexgarel]

I think it's better to document this kind of structure where we declare them, to avoid having to read the full algorithm to understand the structure.

Or even better we could document it as a json schema (in yml) in the docs/reference, that would be cool (it could also be considered an API), and put here the path to the OpenAPI schema.

[stephanegigandet]

There's a description of the structure at the top of the file:

Aggregation counts are stored in a structure of the form:
{
    countries => {
        "en:world" => ..
        "en:france" => {
            categories => {
                "all" => .. # stats for all categories
                "en:yogourts" => {
                    shapes => {
                        "en:unknown" => ..
                        "all" => .. # stats for all shapes
                        "en:bottle" => {
                            materials_parents => .. # stats for parents materials (e.g. PET will also count for plastic)
                            materials => {
                                "all" => ..
                                "en:plastic" => 12, # number of products sold in France that are yogurts and that have a plastic bottle packaging component
                            }
                        },
                        ..
                    }
                },
                ..
            }
        },
        ..
    }
}

Regarding the doc in OpenAPI, why not, but at this point this is completely experimental, I don't know if we will keep that structure or not. We can document in OpenAPI once it's a bit more stabilized.

[alexgarel]

ok.

[alexgarel]

Suggested change

	my $packagings_stats_ref = {};
	# this will contains the final result
	# see structure on top of this file
	my $packagings_stats_ref = {};

[stephanegigandet]

Thanks for all the suggestions @alexgarel , I think I implemented them all in the last commit.

Please don't merge yet, there are a few details I want to change (like filtering out bogus entries in countries).

[teolemon]

I'm not quite sure what to make of this (I looked at the JSON beautified).
@CharlesNepote Is the packaging structured field exported ? It would be easier to pivot, probably.

[stephanegigandet]

I added a special export just for French yogurts, to make the data easier to explore in just a browser:

https://world.openfoodfacts.org/data/categories_stats/categories_packagings_stats.fr.fermented-dairy-desserts.packagings-with-weights.json

So for instance we can easily see the materials used for pots of yogurts:

Note that there are fields for "shape" / "shape_parents", and "material", "material_parents". This is so that we can see stats for all "pots" even if we have some components listed as "pots" and others as "individual pots". Same thing for materials: in "materials" you see the exact values entered by users, and in "materials_parents" you have all the parents values as well.

[stephanegigandet]

I added weights (values + mean):

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information