Skip to content

feat(providers): new mtg datasets in eumetsat_ds #1455

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jan 8, 2025

Conversation

amarandon
Copy link
Collaborator

@amarandon amarandon commented Dec 20, 2024

Integrate the following new product types with the provider eumetsat_ds.

  • Land Surface Temperature Climate Data Record – MSG
  • Land Surface Temperature with Directional Effects – MSG
  • GSA Level 2 Climate Data Record Release 2 - MFG - 57 degree
  • GSA Level 2 Climate Data Record Release 2 - MFG - 63 degree
  • GSA Level 2 Climate Data Record Release 2 - MFG and MSG - 0 degree
  • HIRS Level 1C Fundamental Data Record Release 1 - Multimission – Global
  • Optimal Cloud Analysis Climate Data Record Release 1 - MSG - 0 degree
  • Radio Occultation Level 1B Products - Sentinel-6
  • MTG LI Accumulated Flashes - MTG - 0 degree
  • MTG LI Lightning Flashes - MTG - 0 degree
  • MTG LI Lightning Groups - MTG - 0 degree
  • MTG LI Accumulated Flash Area - MTG - 0 degree
  • MTG LI Accumulated Flash Radiance - MTG - 0 degree
  • MTG LI Lightning Events Filtered - MTG - 0 degree
  • MTG FCI datasets (details to be provided in Q4 2024)

@amarandon amarandon linked an issue Dec 20, 2024 that may be closed by this pull request
Copy link
Contributor

github-actions bot commented Dec 20, 2024

Test Results

    4 files  ±0      4 suites  ±0   6m 20s ⏱️ +12s
  585 tests ±0    582 ✅ ±0   3 💤 ±0  0 ❌ ±0 
2 340 runs  ±0  2 242 ✅ ±0  98 💤 ±0  0 ❌ ±0 

Results for commit 492154f. ± Comparison against base commit 0f20c0a.

This pull request removes 1 and adds 1 tests. Note that renamed tests count towards both.
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_tar_file_with_zip_extension_ok
tests.units.test_download_plugins.TestDownloadPluginCreodiasS3 ‑ test_plugins_download_creodias_s3_without_assets

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Dec 20, 2024

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         323      64  80.19%   62-63, 87, 674-737, 839-890, 894
config.py                                      430      27  93.72%   83-85, 94, 102, 106-108, 179, 190, 695-697, 810-813, 856-857, 866-867, 972, 1035-1040, 1042
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    782      74  90.54%   317-318, 368, 658, 702-705, 743, 787, 821, 866-871, 897, 988, 1056, 1194, 1279-1291, 1327, 1329, 1357, 1361-1372, 1385-1391, 1474-1477, 1510-1530, 1582, 1599-1603, 1615-1618, 1954, 1978-1984, 2235, 2239-2243, 2252-2254, 2298-2299, 2328-2329
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        188      20  89.36%   70-72, 238-239, 314, 343, 400, 414-417, 430, 454-457, 500-506
api/product/metadata_mapping.py                700      76  89.14%   131-133, 230, 262-263, 309-310, 320-332, 334, 345, 351-363, 410-411, 448, 469-472, 495, 503-504, 590-591, 615-616, 622-625, 640-641, 790, 836, 989, 998-1002, 1019-1024, 1151, 1165-1185, 1205, 1210, 1339, 1353, 1424, 1476, 1499-1500, 1516-1520, 1536, 1544, 1563
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       2  90.48%   48, 55
plugins/manager.py                             172      14  91.86%   116-121, 171, 209, 231, 235, 259, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           97      10  89.69%   170-172, 219-220, 246-248, 301-302
plugins/apis/usgs.py                           180      26  85.56%   155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 477
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   42-44, 48-51
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                100      16  84.00%   141, 170, 172, 207-220, 276-280
plugins/authentication/token_exchange.py        36      14  61.11%   75, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47       8  82.98%   52-53, 69, 78-81, 83, 90-93
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       7  76.67%   55-60, 63-64, 80-84
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 504, 536, 540, 547, 577-585, 589, 621-629, 640-642, 673-747, 765-823, 834-839, 851-864, 889, 904-906, 909, 919-927, 935-948, 958-989, 996-1008, 1046, 1072, 1117-1119, 1339
plugins/download/base.py                       257      51  80.16%   136, 164, 244-247, 300-301, 349-353, 359-360, 402, 405-419, 431, 435, 499-503, 533-534, 542-559, 566-574, 576-580, 623, 645, 667, 675
plugins/download/creodias_s3.py                 25       9  64.00%   55-69
plugins/download/http.py                       526     103  80.42%   235, 331-334, 337, 344-349, 367-382, 399, 411, 459, 466-472, 490, 504, 518, 526-528, 544-549, 560, 578, 614-617, 646, 650, 744-811, 829, 859-868, 904, 929-930, 949-954, 960, 963, 980, 983, 998-999, 1026-1027, 1034, 1096, 1111, 1170-1171, 1177, 1187, 1223, 1259, 1279, 1314-1316
plugins/download/s3rest.py                     116      24  79.31%   118, 152, 159, 194, 221-228, 231-233, 237, 248-254, 262-263, 266-270, 293, 314-317
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         146      11  92.47%   100, 104, 128-134, 274, 294, 433
plugins/search/build_search_result.py          341      49  85.63%   249, 277-278, 314, 317, 386-389, 478-495, 523, 558, 560, 585-592, 622-624, 628, 650, 685, 736, 765-766, 781-796, 839, 864, 867, 870, 878, 997-998, 1004, 1015, 1073, 1124
plugins/search/cop_marine.py                   240      47  80.42%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445
plugins/search/creodias_s3.py                   57       3  94.74%   59, 77, 116
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     733      89  87.86%   453, 511, 525, 529-535, 563-567, 680-692, 737-740, 811-812, 860, 879, 886, 898, 955, 976, 979-980, 989-990, 999-1000, 1009-1010, 1037, 1112-1117, 1121-1130, 1164, 1186, 1263, 1353, 1438-1439, 1449, 1526-1530, 1592, 1595, 1601-1602, 1623, 1651-1663, 1670, 1702-1704, 1714-1720, 1750, 1773, 1778-1779, 1794, 1810, 1895-1898, 1903-1906, 1915, 1945-1949, 1955
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  25       0  100.00%
rest/constants.py                                6       0  100.00%
rest/core.py                                   260      66  74.62%   258, 266, 284-301, 316-352, 446, 483-522, 705, 712-765
rest/errors.py                                  69       5  92.75%   106, 116, 127, 143-144
rest/server.py                                 192      24  87.50%   94, 117-119, 293-298, 326, 522-524, 541-546, 575, 577, 581-582, 586-587
rest/stac.py                                   319      63  80.25%   306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 778-782, 789, 843-844, 905, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     180       6  96.67%   225-229, 282, 285, 353, 385
rest/types/queryables.py                        57       5  91.23%   93-98, 162
rest/types/stac_search.py                      125       7  94.40%   138, 184, 199-201, 209, 213
rest/utils/__init__.py                          94      12  87.23%   111-112, 131-133, 185, 195-209
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       3  86.96%   48, 60, 62
types/__init__.py                              128      42  67.19%   66, 79-83, 94-106, 134-136, 143-148, 189, 208, 230, 240-256, 261, 263, 285, 290, 298, 308
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                            108       1  99.07%   63
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 82      16  80.49%   131-134, 138-145, 157-163, 176-178
utils/__init__.py                              520      47  90.96%   85, 201-202, 211-238, 241, 256, 336-340, 413-417, 436-438, 451-465, 544, 559, 599-600, 964-967, 975-976, 1014-1015, 1062-1063, 1197
utils/exceptions.py                             46       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       8  78.95%   51, 53, 57, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9964    1654  83.40%

Diff against develop

Filename      Stmts    Miss  Cover
----------  -------  ------  --------
TOTAL             0       0  +100.00%

Results for commit: 492154f

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented Dec 20, 2024

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         323      64  80.19%   62-63, 87, 674-737, 839-890, 894
config.py                                      430      28  93.49%   83-85, 94, 102, 106-108, 179, 190, 695-697, 810-813, 856-857, 866-867, 972, 1003, 1035-1040, 1042
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    782      74  90.54%   317-318, 368, 658, 702-705, 743, 787, 821, 866-871, 897, 988, 1056, 1194, 1279-1291, 1327, 1329, 1357, 1361-1372, 1385-1391, 1474-1477, 1510-1530, 1582, 1599-1603, 1615-1618, 1954, 1978-1984, 2235, 2239-2243, 2252-2254, 2298-2299, 2328-2329
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        188      20  89.36%   70-72, 238-239, 314, 343, 400, 414-417, 430, 454-457, 500-506
api/product/metadata_mapping.py                700      77  89.00%   131-133, 230, 262-263, 309-310, 320-332, 334, 345, 351-363, 410-411, 448, 469-472, 495, 503-504, 590-591, 615-616, 622-625, 640-641, 790, 836, 989, 998-1002, 1019-1024, 1151, 1165-1185, 1205, 1210, 1339, 1353, 1424, 1476, 1499-1500, 1503, 1516-1520, 1536, 1544, 1563
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       3  85.71%   48, 55, 68
plugins/manager.py                             172      14  91.86%   116-121, 171, 209, 231, 235, 259, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           97      10  89.69%   170-172, 219-220, 246-248, 301-302
plugins/apis/usgs.py                           180      26  85.56%   155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 477
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   42-44, 48-51
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                100      16  84.00%   141, 170, 172, 207-220, 276-280
plugins/authentication/token_exchange.py        36      14  61.11%   75, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-112
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       7  76.67%   55-60, 63-64, 80-84
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 504, 536, 540, 547, 577-585, 589, 621-629, 640-642, 673-747, 765-823, 834-839, 851-864, 889, 904-906, 909, 919-927, 935-948, 958-989, 996-1008, 1046, 1072, 1117-1119, 1339
plugins/download/base.py                       257      53  79.38%   136, 164, 231-233, 244-247, 300-301, 349-353, 359-360, 402, 405-419, 431, 435, 499-503, 533-534, 542-559, 566-574, 576-580, 623, 645, 667, 675
plugins/download/creodias_s3.py                 25       9  64.00%   55-69
plugins/download/http.py                       526     103  80.42%   235, 331-334, 337, 344-349, 367-382, 399, 411, 459, 466-472, 490, 504, 518, 526-528, 544-549, 560, 578, 614-617, 646, 650, 744-811, 829, 859-868, 904, 929-930, 949-954, 960, 963, 980, 983, 998-999, 1026-1027, 1034, 1096, 1111, 1170-1171, 1177, 1187, 1223, 1259, 1279, 1314-1316
plugins/download/s3rest.py                     116      24  79.31%   118, 152, 159, 194, 221-228, 231-233, 237, 248-254, 262-263, 266-270, 293, 314-317
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         146      11  92.47%   100, 104, 128-134, 274, 294, 433
plugins/search/build_search_result.py          341      50  85.34%   249, 277-278, 314, 317, 386-389, 478-495, 523, 558, 560, 585-592, 622-624, 628, 650, 685, 736, 750, 765-766, 781-796, 839, 864, 867, 870, 878, 997-998, 1004, 1015, 1073, 1124
plugins/search/cop_marine.py                   240      47  80.42%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445
plugins/search/creodias_s3.py                   57       3  94.74%   59, 77, 116
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     733     125  82.95%   453, 511, 525, 529-535, 563-567, 680-692, 737-740, 811-812, 860, 879, 886, 898, 955, 976, 979-980, 989-990, 999-1000, 1009-1010, 1037, 1112-1117, 1121-1130, 1164, 1186, 1263, 1353, 1438-1439, 1449, 1526-1530, 1592, 1595, 1601-1602, 1623, 1651-1663, 1670, 1702-1704, 1714-1720, 1750, 1773, 1778-1779, 1794, 1810, 1878-1988
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  25       1  96.00%   35
rest/constants.py                                6       0  100.00%
rest/core.py                                   260     153  41.15%   157, 159, 161, 164-165, 179-189, 198-199, 205, 208, 249-303, 316-352, 383-420, 435-451, 467-476, 483-522, 539, 581-666, 705, 712-765
rest/errors.py                                  69      49  28.99%   60, 65-100, 105-108, 115-118, 126-147, 155-160, 175-181
rest/server.py                                 192     192  0.00%    18-600
rest/stac.py                                   319      68  78.68%   240, 306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 713, 778-782, 789, 843-844, 850, 905, 943, 976, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     180      20  88.89%   225-229, 262-264, 282, 285, 291, 295, 353, 371-386
rest/types/queryables.py                        57      13  77.19%   50-51, 58-59, 66-67, 93-98, 107-108, 162
rest/types/stac_search.py                      125      11  91.20%   136-138, 184, 199-201, 209, 213, 261, 264
rest/utils/__init__.py                          94      30  68.09%   82-88, 108, 111-112, 131-133, 146, 153, 178-186, 193-214
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       5  78.26%   43-44, 48, 60, 62
types/__init__.py                              128      44  65.62%   66, 70, 79-83, 94-106, 134-136, 143-148, 189, 203, 208, 230, 240-256, 261, 263, 285, 290, 298, 308
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                            108       1  99.07%   63
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 82      16  80.49%   131-134, 138-145, 157-163, 176-178
utils/__init__.py                              520      47  90.96%   85, 201-202, 211-238, 241, 256, 336-340, 413-417, 436-438, 451-465, 544, 559, 599-600, 964-967, 975-976, 1014-1015, 1062-1063, 1197
utils/exceptions.py                             46       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       8  78.95%   51, 53, 57, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9964    2089  79.03%

Diff against develop

Filename      Stmts    Miss  Cover
----------  -------  ------  --------
TOTAL             0       0  +100.00%

Results for commit: 492154f

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@amarandon amarandon requested a review from jlahovnik December 20, 2024 13:03
@amarandon amarandon self-assigned this Dec 20, 2024
@jlahovnik jlahovnik self-assigned this Jan 6, 2025
@jlahovnik jlahovnik requested a review from sbrunato January 6, 2025 11:26
@sbrunato sbrunato changed the title 1454 new mtg datasets in eumetsat feat(providers): new mtg datasets in eumetsat_ds Jan 8, 2025
@sbrunato sbrunato merged commit 6dc0509 into develop Jan 8, 2025
10 checks passed
@sbrunato sbrunato deleted the 1454-new-mtg-datasets-in-eumetsat branch January 8, 2025 07:44
@sbrunato sbrunato added this to the 3.0.2.dev milestone Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New MTG datasets in eumetsat
3 participants