feat: dq new facet for opposite tags #10378

benbenben2 · 2024-06-01T06:31:11Z

What

Added new facet for opposite tags in labels and categories taxonomies.

unit tests
description of the facet*

*Remark: facets don't contain language code "en:" because otherwise the description will not work.

Added new function in Tags.pm. I thought it might be useful for other use cases.

Screenshot

Related issue(s) and discussion

Fixes Add quality facets based on the opposite: property, and on mutually-exclusive tag values #10372

benbenben2 · 2024-06-01T20:46:50Z

Error with unit test attributes.t, and more particularly en-attributes.json. The file is initially (expected_test_results):

{
   "attributes" : [
      {
         "description" : "Organic farming aims to protect the environment and to conserve biodiversity by prohibiting or limiting the use of synthetic fertilizers, pesticides and food additives.",
         "description_short" : "Promotes ecological sustainability and biodiversity.",
         "grade" : "a",
         "icon_url" : "https://server_domain/images/attributes/dist/organic.svg",
         "id" : "labels_organic",
         "match" : 100,
         "name" : "Organic farming",
         "status" : "known",
         "title" : "Organic product"
      },
      {
         "description" : "When you buy fair trade products, producers in developing countries are paid an higher and fairer price, which helps them improve and sustain higher social and often environmental standards.",
         "description_short" : "Helps producers in developing countries.",
         "grade" : "a",
         "icon_url" : "https://server_domain/images/attributes/dist/fair-trade.svg",
         "id" : "labels_fair_trade",
         "match" : 100,
         "name" : "Fair trade",
         "status" : "known",
         "title" : "Fair trade product"
      }
   ],
   "id" : "labels",
   "name" : "Labels"
}

After changes from this PR it becomes

{
   'attributes' => [
                     {
                       'description' => 'Organic farming aims to protect the environment and to conserve biodiversity by prohibiting or limiting the use of synthetic fertilizers, pesticides and food additives.',
                       'description_short' => 'Organic products promote ecological sustainability and biodiversity.',
                       'grade' => 'e',
                       'icon_url' => 'https://server_domain/images/attributes/dist/not-organic.svg',
                       'id' => 'labels_organic',
                       'match' => 0,
                       'name' => 'Organic farming',
                       'status' => 'known',
                       'title' => 'Not an organic product'
                     },
                     {
                       'description' => 'When you buy fair trade products, producers in developing countries are paid an higher and fairer price, which helps them improve and sustain higher social and often environmental standards.',
                       'description_short' => 'Fair trade products help producers in developing countries.',
                       'grade' => 'e',
                       'icon_url' => 'https://server_domain/images/attributes/dist/not-fair-trade.svg',
                       'id' => 'labels_fair_trade',
                       'match' => 0,
                       'name' => 'Fair trade',
                       'status' => 'known',
                       'title' => 'Not a fair trade product'
                     }
                   ],
   'id' => 'labels',
   'name' => 'Labels'
}

This is definitely related to the PR because fair-trade and organic both contains opposites.
However, if labels.txt has been modified in this PR, nor fair-trade nor organic has been modified in the labels.txt file.
Not so clear to me how to fix that.

sonarqubecloud · 2024-06-01T21:02:49Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

benbenben2 · 2024-07-10T05:31:38Z

From data quality meeting minutes:

Errors rather than warnings.

-> done

Aleene: have a more generic approach: with some categories, the category should be only linked with its parents: eg. white pepper

-> done

we could use the “exclusive” tag

-> used incompatible_with instead, see below

labels and category/ingredients/labels: eg. kosher and pork, halal and pork, label/halal/category/pork => https://world.openfoodfacts.org/category/pork/label/kosher
Benoît: how would cross taxonomies: an opposite is a category.
For example label: vegetarian vs category: pork.
Pierre: eg. incompatible_with:label:en:kosher

-> done

Remark: there is a similar error for vegetarian/vegan labels and ingredients: https://hr.openfoodfacts.org/data-quality-error/en:vegan-label-but-non-vegan-ingredient

-> replace this similar error by the present one would imply to add incompatible_with for all vegan/vegetarian ingredients. Instead, keeping this similar error - using the vegan:en:yes tag - seems to be fine.

Comments

Couple of typos from previous PR #10392

missing facets in data_quality.txt taxonomy (fyi, for next PRs @aleene, @stephanegigandet )

Can we replace opposite tags in the taxonomy (labels) by incompatible_with (+ update Tags.pm)? (question for @stephanegigandet maybe, this tag existed already before the present PR)

alexgarel

Great PR, I have small suggestions.

alexgarel · 2024-07-11T14:42:08Z

lib/ProductOpener/Tags.pm

+
+For each tag of a given field ($tagtype, can be "labels" or "categories", for example),
+and a given property ($prop_name, without last column (:). Can be "incompatible_with:en", for example),
+return a hash of tagid <-> property_value


I don't know if it's what's wanted, but it does not handle property inheritance and we have to document it !

Suggested change

return a hash of tagid <-> property_value

return a hash of tagid <-> property_value

Note that this does not handle inheritance.

alexgarel · 2024-07-11T14:45:17Z