feat(providers): add MSG product types #1348

amarandon · 2024-10-16T09:52:30Z

Add the following MSG datasets in EODAG eumetsat provider (this provider is called eumetsat_ds in default eodag providers.yml config file).

Get datasets metadata from: https://data.eumetsat.int/

github-actions · 2024-10-16T09:56:14Z

Test Results

4 files ±0 4 suites ±0 5m 48s ⏱️ +8s
568 tests ±0 565 ✅ ±0 3 💤 ±0 0 ❌ ±0
2 272 runs ±0 2 176 ✅ ±0 96 💤 ±0 0 ❌ ±0

Results for commit 8ce13a6. ± Comparison against base commit e482098.

♻️ This comment has been updated with latest results.

github-actions · 2024-10-16T09:56:34Z

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         314      59  81.21%   657-716, 818-869, 873
config.py                                      384      27  92.97%   83-85, 94, 102, 106-108, 179, 190, 581-583, 696-699, 742-743, 752-753, 858, 921-926, 928
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    767      69  91.00%   374, 664, 708-711, 749, 793, 827, 872-877, 903, 994, 1062, 1200, 1285-1297, 1333, 1335, 1363, 1367-1378, 1391-1397, 1480-1483, 1516-1536, 1588, 1605-1609, 1621-1624, 1980, 2004-2010, 2261, 2265-2269, 2278-2280, 2312
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                680      68  90.00%   130-132, 227, 259-260, 306-307, 317-329, 331, 342, 348-360, 405-406, 443, 464-467, 490, 498-499, 575-576, 600-601, 607-610, 625-626, 775, 821, 992-997, 1124, 1138-1158, 1178, 1183, 1312, 1326, 1397, 1449, 1489-1493, 1508
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       2  90.48%   48, 55
plugins/manager.py                             172      15  91.28%   116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           91       8  91.21%   149-151, 198-199, 225-227
plugins/apis/usgs.py                           180      31  82.78%   132, 234, 268, 303-305, 310, 336-337, 342, 372-379, 390-395, 417-423, 425-431, 454
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   40, 50
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              48       4  91.67%   134, 160-165
plugins/authentication/oauth.py                 13       7  46.15%   32-34, 38-41
plugins/authentication/openid_connect.py       187      17  90.91%   114, 128-154, 162, 316-319, 345
plugins/authentication/qsauth.py                34       1  97.06%   83
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 89      16  82.02%   79, 108, 110, 133-146, 202-206
plugins/authentication/token_exchange.py        36      20  44.44%   74-80, 92-122
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   40
plugins/crunch/filter_date.py                   59      14  76.27%   49-54, 66, 75, 84, 87, 99-101, 110-112, 119
plugins/crunch/filter_latest_intersect.py       47       8  82.98%   51-52, 68, 77-80, 82, 89-92
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 68-71, 78-81, 87, 95, 106-122
plugins/crunch/filter_property.py               30       7  76.67%   54-59, 62-63, 79-83
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   266, 279, 346-349, 363-367, 409-411, 415, 447-448, 454-458, 487, 519, 523, 530, 560-568, 572, 604-612, 623-625, 656-730, 748-808, 819-824, 836-849, 874, 889-891, 894, 904-912, 920-933, 943-974, 981-993, 1031, 1057, 1102-1104, 1324
plugins/download/base.py                       253      51  79.84%   136, 164, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       533     120  77.49%   204, 300-303, 305-306, 313-318, 336-351, 368-370, 382, 430, 437-443, 461, 475, 489, 497-499, 515-520, 531, 549, 591-595, 617, 657, 702, 716-722, 751-815, 833, 863-872, 894-895, 922-927, 933, 936, 952, 969-970, 1000-1001, 1008, 1069-1075, 1130-1131, 1137, 1147, 1183, 1219, 1237-1250, 1276-1278
plugins/download/s3rest.py                     116      24  79.31%   113, 149, 156, 191, 218-225, 228-230, 234, 245-251, 259-260, 263-267, 290, 311-314
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         132      13  90.15%   100, 104, 115, 274, 294, 350-351, 371, 374-382
plugins/search/build_search_result.py          188      24  87.23%   105, 146-147, 153, 164, 298-301, 330, 365-382, 431, 460, 463, 473, 490, 518, 520
plugins/search/cop_marine.py                   236      50  78.81%   55, 63-65, 75-76, 81, 86-87, 103, 105, 108, 143-145, 157-158, 200, 206, 210, 214, 227, 238-239, 247, 275, 279, 294, 298, 302, 306, 310-314, 320-323, 326-340, 357, 406-410, 415, 427
plugins/search/creodias_s3.py                   55       3  94.55%   58, 76, 110
plugins/search/csw.py                          105      81  22.86%   58-59, 63-64, 72-120, 126-139, 147-179, 197-238
plugins/search/data_request_search.py          202      69  65.84%   90-93, 109, 120, 124-125, 136, 141, 146, 153, 166-169, 223-224, 228, 238-244, 249, 275-278, 286-297, 314, 316, 323-326, 328-329, 347-351, 384, 394, 405, 418, 424-439, 444
plugins/search/qssearch.py                     718      94  86.91%   390, 394-400, 515-527, 571, 587, 597, 616-631, 668-671, 742-743, 791, 810, 817, 829, 886, 907, 910-911, 920-921, 930-931, 940-941, 968, 1039-1044, 1048-1057, 1091, 1113, 1173, 1222, 1286, 1289-1290, 1372-1376, 1438, 1441, 1447-1448, 1469, 1496-1508, 1515, 1547-1549, 1559-1565, 1595, 1618, 1633, 1649, 1724-1727, 1732-1735, 1762-1763, 1778
plugins/search/static_stac_search.py            72      10  86.11%   98-125, 141, 154
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  25       0  100.00%
rest/constants.py                                6       0  100.00%
rest/core.py                                   251      63  74.90%   216-217, 273, 281, 299-316, 331-369, 460, 502-533, 680, 687-735
rest/errors.py                                  69       5  92.75%   106, 116, 127, 143-144
rest/server.py                                 188      24  87.23%   89, 112-114, 277-282, 310, 495-497, 514-519, 548, 550, 554-555, 559-560
rest/stac.py                                   319      63  80.25%   306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 778-782, 789, 843-844, 905, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     176       5  97.16%   225-229, 282, 285, 353
rest/types/queryables.py                        55       1  98.18%   163
rest/types/stac_search.py                      125       7  94.40%   138, 184, 199-201, 209, 213
rest/utils/__init__.py                          93      12  87.10%   108-109, 128-130, 182, 192-206
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       3  86.96%   48, 60, 62
types/__init__.py                              114      14  87.72%   65, 82, 141-144, 211, 225-234, 244, 265, 278
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             80       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              499      36  92.79%   85, 189-190, 199-226, 229, 243, 323-327, 401-405, 424-426, 505, 520, 556-557, 921-924, 932-933, 971-972, 1143
utils/constraints.py                           119      38  68.07%   62, 89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244
utils/exceptions.py                             41       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   30       8  73.33%   36, 38, 42, 76, 94-101
utils/requests.py                               55      11  80.00%   64, 86, 88, 90, 92, 94, 110, 118-120, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9599    1588  83.46%

Diff against develop

Filename      Stmts    Miss  Cover
----------  -------  ------  --------
TOTAL             0       0  +100.00%

Results for commit: 8ce13a6

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

github-actions · 2024-10-16T09:56:48Z

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         314      59  81.21%   657-716, 818-869, 873
config.py                                      384      28  92.71%   83-85, 94, 102, 106-108, 179, 190, 581-583, 696-699, 742-743, 752-753, 858, 889, 921-926, 928
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    767      69  91.00%   374, 664, 708-711, 749, 793, 827, 872-877, 903, 994, 1062, 1200, 1285-1297, 1333, 1335, 1363, 1367-1378, 1391-1397, 1480-1483, 1516-1536, 1588, 1605-1609, 1621-1624, 1980, 2004-2010, 2261, 2265-2269, 2278-2280, 2312
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                680      68  90.00%   130-132, 227, 259-260, 306-307, 317-329, 331, 342, 348-360, 405-406, 443, 464-467, 490, 498-499, 575-576, 600-601, 607-610, 625-626, 775, 821, 992-997, 1124, 1138-1158, 1178, 1183, 1312, 1326, 1397, 1449, 1489-1493, 1508
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       3  85.71%   48, 55, 68
plugins/manager.py                             172      15  91.28%   116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           91       8  91.21%   149-151, 198-199, 225-227
plugins/apis/usgs.py                           180      31  82.78%   132, 234, 268, 303-305, 310, 336-337, 342, 372-379, 390-395, 417-423, 425-431, 454
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   40, 50
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              48       4  91.67%   134, 160-165
plugins/authentication/oauth.py                 13       7  46.15%   32-34, 38-41
plugins/authentication/openid_connect.py       187      17  90.91%   114, 128-154, 162, 316-319, 345
plugins/authentication/qsauth.py                34       1  97.06%   83
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 89      16  82.02%   79, 108, 110, 133-146, 202-206
plugins/authentication/token_exchange.py        36      20  44.44%   74-80, 92-122
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   40
plugins/crunch/filter_date.py                   59      14  76.27%   49-54, 66, 75, 84, 87, 99-101, 110-112, 119
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   48-53, 66-111
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 68-71, 78-81, 87, 95, 106-122
plugins/crunch/filter_property.py               30       7  76.67%   54-59, 62-63, 79-83
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   266, 279, 346-349, 363-367, 409-411, 415, 447-448, 454-458, 487, 519, 523, 530, 560-568, 572, 604-612, 623-625, 656-730, 748-808, 819-824, 836-849, 874, 889-891, 894, 904-912, 920-933, 943-974, 981-993, 1031, 1057, 1102-1104, 1324
plugins/download/base.py                       253      53  79.05%   136, 164, 231-233, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       533     121  77.30%   204, 300-303, 305-306, 313-318, 336-351, 368-370, 382, 430, 437-443, 461, 475, 489, 497-499, 515-520, 531, 549, 591-595, 617, 657, 702, 716-722, 751-815, 833, 863-872, 894-895, 922-927, 933, 936, 952, 969-970, 983, 1000-1001, 1008, 1069-1075, 1130-1131, 1137, 1147, 1183, 1219, 1237-1250, 1276-1278
plugins/download/s3rest.py                     116      24  79.31%   113, 149, 156, 191, 218-225, 228-230, 234, 245-251, 259-260, 263-267, 290, 311-314
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         132      14  89.39%   100, 104, 115, 274, 294, 350-351, 371, 374-382, 384
plugins/search/build_search_result.py          188      31  83.51%   105, 146-147, 153, 164, 298-301, 330, 365-382, 431, 460, 463, 473, 490, 510-525
plugins/search/cop_marine.py                   236      50  78.81%   55, 63-65, 75-76, 81, 86-87, 103, 105, 108, 143-145, 157-158, 200, 206, 210, 214, 227, 238-239, 247, 275, 279, 294, 298, 302, 306, 310-314, 320-323, 326-340, 357, 406-410, 415, 427
plugins/search/creodias_s3.py                   55       3  94.55%   58, 76, 110
plugins/search/csw.py                          105      81  22.86%   58-59, 63-64, 72-120, 126-139, 147-179, 197-238
plugins/search/data_request_search.py          202      69  65.84%   90-93, 109, 120, 124-125, 136, 141, 146, 153, 166-169, 223-224, 228, 238-244, 249, 275-278, 286-297, 314, 316, 323-326, 328-329, 347-351, 384, 394, 405, 418, 424-439, 444
plugins/search/qssearch.py                     718     123  82.87%   390, 394-400, 515-527, 571, 574, 587, 597, 616-631, 668-671, 742-743, 791, 810, 817, 829, 886, 907, 910-911, 920-921, 930-931, 940-941, 968, 1039-1044, 1048-1057, 1091, 1113, 1173, 1222, 1286, 1289-1290, 1372-1376, 1438, 1441, 1447-1448, 1469, 1496-1508, 1515, 1547-1549, 1559-1565, 1595, 1618, 1633, 1649, 1707-1807
plugins/search/static_stac_search.py            72      10  86.11%   98-125, 141, 154
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  25       1  96.00%   35
rest/constants.py                                6       0  100.00%
rest/core.py                                   251     139  44.62%   162, 164, 166, 169-170, 184-194, 206-207, 209-210, 216-217, 220, 223, 264-318, 331-369, 400-434, 449-465, 481-490, 502-533, 550, 592-641, 680, 687-735
rest/errors.py                                  69      49  28.99%   60, 65-100, 105-108, 115-118, 126-147, 155-160, 175-181
rest/server.py                                 188     188  0.00%    18-573
rest/stac.py                                   319      68  78.68%   240, 306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 713, 778-782, 789, 843-844, 850, 905, 943, 976, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     176      16  90.91%   225-229, 262-264, 282, 285, 291, 295, 353, 370-380
rest/types/queryables.py                        55      13  76.36%   50-51, 58-59, 66-67, 93-98, 107-108, 163
rest/types/stac_search.py                      125      11  91.20%   136-138, 184, 199-201, 209, 213, 261, 264
rest/utils/__init__.py                          93      30  67.74%   79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       5  78.26%   43-44, 48, 60, 62
types/__init__.py                              114      39  65.79%   65, 78-82, 93-105, 132-134, 141-144, 184, 211, 221-237, 242, 244, 265, 270, 278, 288
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             80       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              499      36  92.79%   85, 189-190, 199-226, 229, 243, 323-327, 401-405, 424-426, 505, 520, 556-557, 921-924, 932-933, 971-972, 1143
utils/constraints.py                           119      38  68.07%   62, 89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244
utils/exceptions.py                             41       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   30       8  73.33%   36, 38, 42, 76, 94-101
utils/requests.py                               55      11  80.00%   64, 86, 88, 90, 92, 94, 110, 118-120, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9599    2032  78.83%

Diff against develop

Filename      Stmts    Miss  Cover
----------  -------  ------  --------
TOTAL             0       0  +100.00%

Results for commit: 8ce13a6

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

amarandon linked an issue Oct 16, 2024 that may be closed by this pull request

Add MSG datasets #1345

Closed

amarandon requested a review from jlahovnik October 16, 2024 13:59

amarandon changed the base branch from develop to master October 16, 2024 14:29

amarandon changed the base branch from master to develop October 16, 2024 14:29

jlahovnik approved these changes Oct 16, 2024

View reviewed changes

alambare marked this pull request as draft October 21, 2024 08:40

amarandon and others added 13 commits October 21, 2024 11:06

feat: add MSG_CLM product

4ce1c92

feat: add product MSG_CLM_IODC

1044dad

Add product type MSG_HRSEVIRI

82e5584

feat: add MSG_GSAL2R02 product type

2b3f3d7

feat: add product type MSG_HRSEVIRI_IODC

1ed8790

Add product MSG_RSS_CLM

78f74ee

Add product MSG_MSG15_RSS

ae8ef77

feat: Add product MSG_LSA_FRM

4b8e155

Add product MSG_LSA_LST_CDR

affa8bd

Add product SG_LSA_LSTDE

1a6852d

Remove metadata mappings irrelevant to MSG

894ad09

Update test for new MSG products

9b6df05

feat: add product MSG_AMVR02

8ce13a6

amarandon force-pushed the 1345-add-msg-datasets branch from daae64e to 8ce13a6 Compare October 21, 2024 09:33

amarandon marked this pull request as ready for review October 21, 2024 09:37

alambare requested a review from sbrunato October 21, 2024 15:42

alambare added the dedl DEDL related label Oct 21, 2024

sbrunato changed the title ~~1345 add msg datasets~~ feat(providers): add MSG product types Oct 22, 2024

sbrunato approved these changes Oct 23, 2024

View reviewed changes

sbrunato merged commit f65e582 into develop Oct 23, 2024
10 checks passed

sbrunato deleted the 1345-add-msg-datasets branch October 23, 2024 07:18

sbrunato added this to the 3.0.1.dev milestone Oct 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(providers): add MSG product types #1348

feat(providers): add MSG product types #1348

amarandon commented Oct 16, 2024 •

edited by sbrunato

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

feat(providers): add MSG product types #1348

feat(providers): add MSG product types #1348

Conversation

amarandon commented Oct 16, 2024 • edited by sbrunato Loading

github-actions bot commented Oct 16, 2024 • edited Loading

Test Results

github-actions bot commented Oct 16, 2024 • edited Loading

Code Coverage (Ubuntu)

Diff against develop

github-actions bot commented Oct 16, 2024 • edited Loading

Code Coverage (Windows)

Diff against develop

amarandon commented Oct 16, 2024 •

edited by sbrunato

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading