Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Start of implementation of API v3 product read and write + integration tests #7614

Merged
merged 103 commits into from
Nov 16, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
221650c
docs: start of proposal for new packagings API
stephanegigandet Oct 14, 2022
a90d868
remove examples
stephanegigandet Oct 14, 2022
629815e
docs: start of proposal for new packagings API
stephanegigandet Oct 14, 2022
d2ec6bc
update title
stephanegigandet Oct 14, 2022
035cae6
API updates
stephanegigandet Oct 17, 2022
6d30633
new shape structure
stephanegigandet Oct 18, 2022
9201ff1
fix file names
stephanegigandet Oct 18, 2022
05b5947
new response status object
stephanegigandet Oct 18, 2022
0f2c5be
new response status object
stephanegigandet Oct 18, 2022
d3146c4
new response status object
stephanegigandet Oct 18, 2022
d9e546d
moved V3 paths to new api-v3.yml
stephanegigandet Oct 19, 2022
f83b3b2
renamed api-v3.yml
stephanegigandet Oct 19, 2022
f214e48
minor changes
stephanegigandet Oct 19, 2022
ddf01e7
Merge branch 'main' into packagings-api
stephanegigandet Oct 20, 2022
3d1d840
fixes and suggestions from review
stephanegigandet Oct 21, 2022
07fac05
docs: change order in api-v3.yml to ease reading
alexgarel Oct 24, 2022
4b84654
response for read product api
stephanegigandet Oct 24, 2022
ee9f944
Merge branch 'packagings-api' of github.com:openfoodfacts/openfoodfac…
stephanegigandet Oct 24, 2022
bd9f6cb
small fix
stephanegigandet Oct 24, 2022
ab5f784
start of implementation of new packagings API
stephanegigandet Oct 26, 2022
2c616e7
API v3
stephanegigandet Oct 26, 2022
00a7751
API v3
stephanegigandet Oct 26, 2022
2e0e766
add way to get POSTed JSON content
stephanegigandet Oct 27, 2022
c5201c0
add way to get POSTed JSON content
stephanegigandet Oct 27, 2022
1fc2c71
start of API v3 tests
stephanegigandet Oct 27, 2022
f6606fd
new API v3 tests
stephanegigandet Oct 27, 2022
4adf11b
fix: generated new package-lock.json to make builds work #7616
stephanegigandet Oct 28, 2022
fce90f9
Merge branch 'fix-npm' into packagings-api-2
stephanegigandet Oct 28, 2022
fcb26cc
fix dependencies
stephanegigandet Oct 28, 2022
35042c7
refactor product post-processing after changes, update tests
stephanegigandet Oct 28, 2022
c95c1d3
save product in write_product_api
stephanegigandet Nov 2, 2022
8972b82
write product: get back updated fields + packaging changes
stephanegigandet Nov 2, 2022
7363361
move warnings and errors to the right place in API response
stephanegigandet Nov 2, 2022
4e26342
move read_product_api to separate module
stephanegigandet Nov 2, 2022
b93852f
allow matching when number of units is defined
stephanegigandet Nov 2, 2022
6424ff6
added tests, renamed number to number_of_units
stephanegigandet Nov 2, 2022
0d9e01e
added tests, fixed tags_lc
stephanegigandet Nov 2, 2022
3156db7
lint
stephanegigandet Nov 2, 2022
9d69090
lint and tests
stephanegigandet Nov 3, 2022
ad3e86d
lint
stephanegigandet Nov 3, 2022
f13a516
fix tests
stephanegigandet Nov 3, 2022
70aa1c9
Merge branch 'main' into packagings-api-2
stephanegigandet Nov 3, 2022
2aca594
add API result + localized messages
stephanegigandet Nov 3, 2022
49341f9
update messages
stephanegigandet Nov 3, 2022
8aeb116
fix local messages
stephanegigandet Nov 4, 2022
9d94544
move existing field number to number_of_units
stephanegigandet Nov 4, 2022
9d7cfc8
convert quantity_value to number
stephanegigandet Nov 4, 2022
c66ed71
update tests
stephanegigandet Nov 4, 2022
6b2a902
renamed number to number_of_units
stephanegigandet Nov 4, 2022
f10dc55
quantity -> quantity_per_unit
stephanegigandet Nov 4, 2022
799f902
lint
stephanegigandet Nov 4, 2022
afa1a41
Update lib/ProductOpener/API.pm
stephanegigandet Nov 7, 2022
1df9218
Update lib/ProductOpener/API.pm
stephanegigandet Nov 7, 2022
92e3c49
Update lib/ProductOpener/API.pm
stephanegigandet Nov 7, 2022
a2c78da
Update lib/ProductOpener/APITest.pm
stephanegigandet Nov 7, 2022
b16a6ec
changes from code review
stephanegigandet Nov 8, 2022
bacdf2f
Update lib/ProductOpener/Packaging.pm
stephanegigandet Nov 8, 2022
731b2d8
Update lib/ProductOpener/Products.pm
stephanegigandet Nov 8, 2022
d34a005
Update lib/ProductOpener/API.pm
stephanegigandet Nov 8, 2022
50910a1
Update lib/ProductOpener/API.pm
stephanegigandet Nov 8, 2022
f20c713
Apply suggestions from code review
stephanegigandet Nov 8, 2022
9d36402
suggestions from code review
stephanegigandet Nov 8, 2022
7a34e95
suggestions from code review
stephanegigandet Nov 8, 2022
0522ee1
Merge branch 'packagings-api-2' of github.com:openfoodfacts/openfoodf…
stephanegigandet Nov 8, 2022
160c340
lint
stephanegigandet Nov 8, 2022
93268cb
small fix
stephanegigandet Nov 8, 2022
7e6d5ae
fixes
stephanegigandet Nov 8, 2022
ac1be16
fixes
stephanegigandet Nov 8, 2022
60cb9f2
separate function get_images_to_update()
stephanegigandet Nov 8, 2022
508f9f6
lint
stephanegigandet Nov 8, 2022
3a46d03
perlcritic
stephanegigandet Nov 8, 2022
7151dba
fix test
stephanegigandet Nov 9, 2022
910b157
fix and update tests
stephanegigandet Nov 9, 2022
576055d
Merge branch 'main' into packagings-api-2
stephanegigandet Nov 9, 2022
e84292e
replaced POST with PATCH
stephanegigandet Nov 10, 2022
8637ab1
replaced POST with PATCH
stephanegigandet Nov 10, 2022
0965d66
replaced POST with PATCH
stephanegigandet Nov 10, 2022
c1f71e4
replaced POST with PATCH
stephanegigandet Nov 10, 2022
c038487
replaced POST with PATCH
stephanegigandet Nov 10, 2022
46186d0
merged main
stephanegigandet Nov 10, 2022
c519c0d
lint
stephanegigandet Nov 10, 2022
af774fb
Merge branch 'main' into packagings-api-2
stephanegigandet Nov 10, 2022
728f5be
fix status code for api v2
stephanegigandet Nov 14, 2022
b5f0569
lint
stephanegigandet Nov 14, 2022
e3d9732
fix: do not preload Minion module #7695
stephanegigandet Nov 14, 2022
91bc2d1
Merge branch 'minion' into packagings-api-2
stephanegigandet Nov 14, 2022
271ea04
Merge branch 'main' into packagings-api-2
stephanegigandet Nov 15, 2022
f9d1ff7
preload new modules
stephanegigandet Nov 15, 2022
14df636
remove some perl modules #7699
stephanegigandet Nov 15, 2022
2cb20ed
remove Data::Dump
stephanegigandet Nov 15, 2022
596c1af
make product available in template
stephanegigandet Nov 15, 2022
b9ee95d
fix tests
stephanegigandet Nov 15, 2022
7d769a0
fix en:world country
stephanegigandet Nov 15, 2022
d05797d
Merge branch 'main' into packagings-api-2
stephanegigandet Nov 16, 2022
d2b48e0
fixes
stephanegigandet Nov 16, 2022
d64954f
remove Data::Dump and use Test::More explain()
stephanegigandet Nov 16, 2022
b7c04cf
lint
stephanegigandet Nov 16, 2022
b1297a2
fix test
stephanegigandet Nov 16, 2022
0ae1163
Merge branch 'main' into packagings-api-2
stephanegigandet Nov 16, 2022
97f5801
fix tests
stephanegigandet Nov 16, 2022
7b89360
Merge branch 'main' into packagings-api-2
stephanegigandet Nov 16, 2022
6b9f468
wait_application_ready();
stephanegigandet Nov 16, 2022
5438d11
Merge branch 'packagings-api-2' of github.com:openfoodfacts/openfoodf…
stephanegigandet Nov 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 32 additions & 7 deletions cgi/display.pl
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,11 @@
use ProductOpener::Config qw/:all/;
use ProductOpener::Store qw/:all/;
use ProductOpener::Index qw/:all/;
use ProductOpener::Routing qw/:all/;
use ProductOpener::Display qw/:all/;
use ProductOpener::Users qw/:all/;
use ProductOpener::Lang qw/:all/;
use ProductOpener::API qw/:all/;

use CGI qw/:cgi :form escapeHTML/;
use URI::Escape::XS;
Expand All @@ -38,6 +40,19 @@
use Apache2::RequestRec ();
use Apache2::Const qw(:common);

# The API V3 write product request uses POST / PUT / PATCH with a JSON body
# if we have such a request, we need to read the body before CGI.pm tries to read it to get multipart/form-data parameters

my $env_query_string = $ENV{QUERY_STRING};

$log->debug("display.pl - start", {env_query_string => $env_query_string});

my $body_request_ref = {};

if ($env_query_string =~ /^\/?api\/v3(\.\d+)?\/product/) {
read_request_body($body_request_ref);
}

# The nginx reverse proxy turns /somepath?someparam=somevalue to /cgi/display.pl?/somepath?someparam=somevalue
# so that all non /cgi/ queries are sent to display.pl and that we can get the path in the query string
# CGI.pm thus adds somepath? at the start of the name of the first parameter.
Expand All @@ -58,16 +73,21 @@

my $request_ref = ProductOpener::Display::init_request();

# Add the HTTP request body if we have one
if (defined $body_request_ref->{body}) {
$request_ref->{body} = $body_request_ref->{body};
}

$log->debug("before analyze_request", {query_string => $request_ref->{query_string}});

# analyze request will fill request with action and parameters
analyze_request($request_ref);

# If we have an error, display the error page and return

if (defined $request_ref->{error_status}) {
if (defined $request_ref->{error_message}) {
$log->debug("analyze_request error", {request_ref => $request_ref});
display_error($request_ref->{error_message}, $request_ref->{error_status});
display_error($request_ref->{error_message}, $request_ref->{status_code});
$log->debug("analyze_request error - return Apache2::Const::OK");
return Apache2::Const::OK;
}
Expand Down Expand Up @@ -97,24 +117,29 @@
display_error_and_exit(lang("no_owner_defined"), 200);
}

if ((defined $request_ref->{api}) and (defined $request_ref->{api_method})) {
if (single_param("api_method") eq "search") {
if ((defined $request_ref->{api}) and (defined $request_ref->{api_action})) {

# V3 API use a generic request and response format
if ($request_ref->{api_version} =~ /3((\.)\d+)?/) {
process_api_request($request_ref);
}
elsif ($request_ref->{api_action} eq "search") {
# /api/v0/search
# FIXME: for an unknown reason, using display_search_results() here results in some attributes being randomly not set
# because of missing fields like nova_group or nutriscore_data, but not for all products.
# this does not seem to happen with display_tag()
# display_search_results($request_ref);
display_tag($request_ref);
}
elsif (single_param("api_method") =~ /^preferences(_(\w\w))?$/) {
elsif ($request_ref->{api_action} =~ /^preferences(_(\w\w))?$/) {
# /api/v0/preferences or /api/v0/preferences_[language code]
display_preferences_api($request_ref, $2);
}
elsif (single_param("api_method") =~ /^attribute_groups(_(\w\w))?$/) {
elsif ($request_ref->{api_action} =~ /^attribute_groups(_(\w\w))?$/) {
# /api/v0/attribute_groups or /api/v0/attribute_groups_[language code]
display_attribute_groups_api($request_ref, $2);
}
elsif (single_param("api_method") eq "taxonomy") {
elsif ($request_ref->{api_action} eq "taxonomy") {
display_taxonomy_api($request_ref);
}
else {
Expand Down
83 changes: 19 additions & 64 deletions cgi/product_jqm_multilingual.pl
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,18 @@
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.

=head1 NAME

product_jqm_multilingual.pl - implementation of the product WRITE API v0, v1 and v2

=head1 DESCRIPTION

This CGI script is v0 to v2 of the Product WRITE API.

API v3 is handled by a different /api/v3/product route, implemented in APIProductWrite.pm

=cut

use ProductOpener::PerlStandards;

use CGI::Carp qw(fatalsToBrowser);
Expand All @@ -42,6 +54,7 @@
use ProductOpener::Packaging qw/:all/;
use ProductOpener::ForestFootprint qw/:all/;
use ProductOpener::Text qw/:all/;
use ProductOpener::API qw/:all/;

use Apache2::RequestRec ();
use Apache2::Const ();
Expand All @@ -55,6 +68,11 @@

my $request_ref = ProductOpener::Display::init_request();

# Response structure to keep track of warnings and errors
# Note: currently some warnings and errors are added,
# but we do not yet do anything with them
my $response_ref = get_initialized_response();

my $comment = '(app)';

my $interface_version = '20150316.jqm2';
Expand Down Expand Up @@ -392,38 +410,6 @@
}
}

if ( (defined $product_ref->{nutriments}{"carbon-footprint"})
and ($product_ref->{nutriments}{"carbon-footprint"} ne ''))
{
push @{$product_ref->{"labels_hierarchy"}}, "en:carbon-footprint";
push @{$product_ref->{"labels_tags"}}, "en:carbon-footprint";
}

# For fields that can have different values in different languages, copy the main language value to the non suffixed field

foreach my $field (keys %language_fields) {
if ($field !~ /_image/) {
if (defined $product_ref->{$field . "_$product_ref->{lc}"}) {
$product_ref->{$field} = $product_ref->{$field . "_$product_ref->{lc}"};
}
}
}

compute_languages($product_ref); # need languages for allergens detection and cleaning ingredients

# Ingredients classes
clean_ingredients_text($product_ref);
extract_ingredients_from_text($product_ref);
extract_ingredients_classes_from_text($product_ref);
detect_allergens_from_text($product_ref);

# Food category rules for sweeetened/sugared beverages
# French PNNS groups from categories

if ((defined $options{product_type}) and ($options{product_type} eq "food")) {
ProductOpener::Food::special_process_product($product_ref);
}

# Nutrition data

# Do not allow nutrition edits through API for data provided by producers
Expand All @@ -435,35 +421,7 @@
assign_nutriments_values_from_request_parameters($product_ref, $nutriment_table);
}

# Compute nutrition data per 100g and per serving

$log->trace("compute_serving_size_date") if ($admin and $log->is_trace());

fix_salt_equivalent($product_ref);

compute_serving_size_data($product_ref);

compute_nutrition_score($product_ref);

compute_nova_group($product_ref);

compute_nutrient_levels($product_ref);

compute_unknown_nutrients($product_ref);

# Until we provide an interface to directly change the packaging data structure
# erase it before reconstructing it
# (otherwise there is no way to remove incorrect entries)
$product_ref->{packagings} = [];

analyze_and_combine_packaging_data($product_ref);

if ((defined $options{product_type}) and ($options{product_type} eq "food")) {
compute_ecoscore($product_ref);
compute_forest_footprint($product_ref);
}

ProductOpener::DataQuality::check_quality($product_ref);
analyze_and_enrich_product_data($product_ref, $response_ref);

$log->info("saving product", {code => $code}) if ($log->is_info() and not $log->is_debug());
$log->debug("saving product", {code => $code, product => $product_ref})
Expand All @@ -474,9 +432,6 @@
my $time = time();
$comment = $comment . remove_tags_and_quote(decode utf8 => single_param('comment'));
if (store_product($User_id, $product_ref, $comment)) {
# Notify robotoff
send_notification_for_product_change($product_ref, "updated");

$response{status} = 1;
$response{status_verbose} = 'fields saved';
}
Expand Down
93 changes: 8 additions & 85 deletions cgi/product_multilingual.pl
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
use ProductOpener::Web qw(get_languages_options_list);
use ProductOpener::Text qw/:all/;
use ProductOpener::Events qw/:all/;
use ProductOpener::API qw/:all/;

use Apache2::RequestRec ();
use Apache2::Const ();
Expand Down Expand Up @@ -105,6 +106,11 @@ ()
);
}

# Response structure to keep track of warnings and errors
# Note: currently some warnings and errors are added,
# but we do not yet do anything with them
my $response_ref = get_initialized_response();

my $type = single_param('type') || 'search_or_add';
my $action = single_param('action') || 'display';

Expand Down Expand Up @@ -547,51 +553,6 @@ ()
}
}

if ( (defined $product_ref->{nutriments}{"carbon-footprint"})
and ($product_ref->{nutriments}{"carbon-footprint"} ne ''))
{
push @{$product_ref->{"labels_hierarchy"}}, "en:carbon-footprint";
push @{$product_ref->{"labels_tags"}}, "en:carbon-footprint";
}

if ((defined $product_ref->{nutriments}{"glycemic-index"}) and ($product_ref->{nutriments}{"glycemic-index"} ne ''))
{
push @{$product_ref->{"labels_hierarchy"}}, "en:glycemic-index";
push @{$product_ref->{"labels_tags"}}, "en:glycemic-index";
}

# For fields that can have different values in different languages, copy the main language value to the non suffixed field

foreach my $field (keys %language_fields) {
if ($field !~ /_image/) {
if (defined $product_ref->{$field . "_$product_ref->{lc}"}) {
$product_ref->{$field} = $product_ref->{$field . "_$product_ref->{lc}"};
}
}
}

$log->debug("compute_languages") if $log->is_debug();

compute_languages($product_ref); # need languages for allergens detection and cleaning ingredients
$log->debug("clean_ingredients") if $log->is_debug();

# Ingredients classes
clean_ingredients_text($product_ref);
$log->debug("extract_ingredients_from_text") if $log->is_debug();
extract_ingredients_from_text($product_ref);
$log->debug("extract_ingredients_classes_from_text") if $log->is_debug();
extract_ingredients_classes_from_text($product_ref);
$log->debug("detect_allergens_from_text") if $log->is_debug();
detect_allergens_from_text($product_ref);

# Food category rules for sweetened/sugared beverages
# French PNNS groups from categories

if ((defined $options{product_type}) and ($options{product_type} eq "food")) {
$log->debug("Food::special_process_product") if $log->is_debug();
ProductOpener::Food::special_process_product($product_ref);
}

# Obsolete products

if (($User{moderator} or $Owner_id) and (defined single_param('obsolete_since_date'))) {
Expand Down Expand Up @@ -627,42 +588,9 @@ ()
delete $product_ref->{last_checker};
delete $product_ref->{last_checked_t};
}

}

# Compute nutrition data per 100g and per serving

$log->debug("compute nutrition data") if $log->is_debug();

$log->trace("compute_serving_size_date - start") if $log->is_trace();

fix_salt_equivalent($product_ref);

compute_serving_size_data($product_ref);

compute_nutrition_score($product_ref);

compute_nova_group($product_ref);

compute_nutrient_levels($product_ref);

compute_unknown_nutrients($product_ref);

# Until we provide an interface to directly change the packaging data structure
# erase it before reconstructing it
# (otherwise there is no way to remove incorrect entries)
$product_ref->{packagings} = [];

analyze_and_combine_packaging_data($product_ref);

if ((defined $options{product_type}) and ($options{product_type} eq "food")) {
compute_ecoscore($product_ref);
compute_forest_footprint($product_ref);
}

ProductOpener::DataQuality::check_quality($product_ref);

$log->trace("end compute_serving_size_date - end") if $log->is_trace();
analyze_and_enrich_product_data($product_ref, $response_ref);

if ($#errors >= 0) {
$action = 'display';
Expand Down Expand Up @@ -1484,6 +1412,7 @@ ($product_ref, $field, $language)
$template_data_ref_display->{type} = $type;
$template_data_ref_display->{code} = $code;
$template_data_ref_display->{display_product_history} = display_product_history($code, $product_ref);
$template_data_ref_display->{product} = $product_ref;

process_template('web/pages/product_edit/product_edit_form_display.tt.html', $template_data_ref_display, \$html)
or $html = "<p>" . $tt->error() . "</p>";
Expand Down Expand Up @@ -1540,9 +1469,6 @@ ($product_ref, $field, $language)
}
elsif ($type eq 'delete') {

# Notify robotoff
send_notification_for_product_change($product_ref, "deleted");

my $email = <<MAIL
$User_id $Lang{has_deleted_product}{$lc}:

Expand All @@ -1555,9 +1481,6 @@ ($product_ref, $field, $language)
}
else {

# Notify robotoff
send_notification_for_product_change($product_ref, "updated");

# Create an event
send_event({user_id => $User_id, event_type => "product_edited", barcode => $code, points => 5});

Expand Down
4 changes: 2 additions & 2 deletions cpanfile
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,6 @@ requires 'Action::CircuitBreaker';
requires 'Action::Retry'; # deps: libmath-fibonacci-perl

on 'test' => sub {
requires 'Data::Dump';
requires 'Test::More', '>= 1.302186, < 2.0';
requires 'Test::MockModule';
requires 'Mock::Quick';
Expand Down Expand Up @@ -114,4 +113,5 @@ feature "off_server_dev_tools", "Optional development tools" => sub {
requires 'Term::ReadLine::Gnu', '>= 1.42, < 2.0'; # readline support for the Perl debugger. libterm-readline-gnu-perl is available.
requires 'Perl::LanguageServer';
requires 'Hash::SafeKeys'; # Perl::LanguageServer dependency
};
};

Loading