Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(core): handle exceptions occuring on failed index write #1428

Merged
merged 2 commits into from
Dec 18, 2024

Conversation

alambare
Copy link
Collaborator

@alambare alambare commented Dec 5, 2024

@alambare alambare requested a review from sbrunato December 5, 2024 22:36
@alambare alambare self-assigned this Dec 5, 2024
@alambare alambare added bug Something isn't working dedl DEDL related labels Dec 5, 2024
@alambare alambare marked this pull request as ready for review December 5, 2024 22:37
Copy link
Contributor

github-actions bot commented Dec 5, 2024

Test Results

    4 files  ±0      4 suites  ±0   6m 12s ⏱️ ±0s
  582 tests ±0    579 ✅ ±0   3 💤 ±0  0 ❌ ±0 
2 328 runs  ±0  2 230 ✅ ±0  98 💤 ±0  0 ❌ ±0 

Results for commit e98789a. ± Comparison against base commit 132ff00.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Dec 5, 2024

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         323      64  80.19%   62-63, 87, 674-735, 837-888, 892
config.py                                      428      27  93.69%   83-85, 94, 102, 106-108, 179, 190, 690-692, 805-808, 851-852, 861-862, 967, 1030-1035, 1037
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    782      74  90.54%   317-318, 368, 658, 702-705, 743, 787, 821, 866-871, 897, 988, 1056, 1194, 1279-1291, 1327, 1329, 1357, 1361-1372, 1385-1391, 1474-1477, 1510-1530, 1582, 1599-1603, 1615-1618, 1954, 1978-1984, 2235, 2239-2243, 2252-2254, 2298-2299, 2328-2329
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                700      76  89.14%   131-133, 230, 262-263, 309-310, 320-332, 334, 345, 351-363, 410-411, 448, 469-472, 495, 503-504, 590-591, 615-616, 622-625, 640-641, 790, 836, 989, 998-1002, 1019-1024, 1151, 1165-1185, 1205, 1210, 1339, 1353, 1424, 1476, 1499-1500, 1516-1520, 1536, 1544, 1563
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       2  90.48%   48, 55
plugins/manager.py                             172      15  91.28%   116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           97      10  89.69%   170-172, 219-220, 246-248, 301-302
plugins/apis/usgs.py                           180      31  82.78%   155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 448-454, 477
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   42-44, 48-51
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 92      16  82.61%   114, 143, 145, 176-189, 245-249
plugins/authentication/token_exchange.py        36      14  61.11%   75, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47       8  82.98%   52-53, 69, 78-81, 83, 90-93
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       7  76.67%   55-60, 63-64, 80-84
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 503, 535, 539, 546, 576-584, 588, 620-628, 639-641, 672-746, 764-824, 835-840, 852-865, 890, 905-907, 910, 920-928, 936-949, 959-990, 997-1009, 1047, 1073, 1118-1120, 1340
plugins/download/base.py                       258      49  81.01%   136, 164, 300-301, 355-359, 365-366, 408, 411-425, 437, 441, 505-509, 539-540, 548-565, 572-580, 582-586, 629, 651, 673, 681
plugins/download/creodias_s3.py                 17       9  47.06%   53-67
plugins/download/http.py                       545     106  80.55%   237, 333-336, 339, 346-351, 369-384, 401, 413, 461, 468-474, 492, 506, 520, 528-530, 546-551, 562, 580, 623-626, 648, 688, 733, 747-753, 782-846, 864, 894-903, 939, 964-965, 984-989, 995, 998, 1015, 1032-1033, 1063-1064, 1071, 1133, 1148, 1207-1208, 1214, 1224, 1260, 1296, 1316, 1351-1353
plugins/download/s3rest.py                     116      24  79.31%   118, 152, 159, 194, 221-228, 231-233, 237, 248-254, 262-263, 266-270, 293, 314-317
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         146      11  92.47%   100, 104, 128-134, 274, 294, 433
plugins/search/build_search_result.py          341      49  85.63%   249, 277-278, 314, 317, 386-389, 478-495, 523, 558, 560, 585-592, 622-624, 628, 650, 685, 736, 765-766, 781-796, 839, 864, 867, 870, 878, 997-998, 1004, 1015, 1073, 1124
plugins/search/cop_marine.py                   240      47  80.42%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445
plugins/search/creodias_s3.py                   55       3  94.55%   59, 77, 111
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     732      89  87.84%   450, 508, 522, 526-532, 560-564, 677-689, 734-737, 808-809, 857, 876, 883, 895, 952, 973, 976-977, 986-987, 996-997, 1006-1007, 1034, 1105-1110, 1114-1123, 1157, 1179, 1256, 1346, 1431-1432, 1442, 1519-1523, 1585, 1588, 1594-1595, 1616, 1644-1656, 1663, 1695-1697, 1707-1713, 1743, 1766, 1771-1772, 1787, 1803, 1888-1891, 1896-1899, 1908, 1938-1942, 1948
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  25       0  100.00%
rest/constants.py                                6       0  100.00%
rest/core.py                                   257      63  75.49%   258, 266, 284-301, 316-352, 446, 483-522, 705, 712-760
rest/errors.py                                  69       5  92.75%   106, 116, 127, 143-144
rest/server.py                                 192      24  87.50%   94, 117-119, 293-298, 326, 522-524, 541-546, 575, 577, 581-582, 586-587
rest/stac.py                                   319      63  80.25%   306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 778-782, 789, 843-844, 905, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     180       6  96.67%   225-229, 282, 285, 353, 385
rest/types/queryables.py                        57       5  91.23%   93-98, 162
rest/types/stac_search.py                      125       7  94.40%   138, 184, 199-201, 209, 213
rest/utils/__init__.py                          94      12  87.23%   111-112, 131-133, 185, 195-209
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       3  86.96%   48, 60, 62
types/__init__.py                              128      42  67.19%   66, 79-83, 94-106, 134-136, 143-148, 189, 208, 230, 240-256, 261, 263, 285, 290, 298, 308
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                            108       1  99.07%   63
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 82      16  80.49%   131-134, 138-145, 157-163, 176-178
utils/__init__.py                              520      47  90.96%   85, 194-195, 204-231, 234, 249, 329-333, 406-410, 429-431, 444-458, 537, 552, 592-593, 957-960, 968-969, 1007-1008, 1055-1056, 1190
utils/exceptions.py                             46       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       8  78.95%   51, 53, 57, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9959    1658  83.35%

Diff against develop

Filename           Stmts    Miss  Cover
---------------  -------  ------  -------
api/core.py           +3      +2  -0.22%
types/whoosh.py      +67     +16  -19.51%
TOTAL                +70     +18  -0.07%

Results for commit: e98789a

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented Dec 5, 2024

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         323      64  80.19%   62-63, 87, 674-735, 837-888, 892
config.py                                      428      28  93.46%   83-85, 94, 102, 106-108, 179, 190, 690-692, 805-808, 851-852, 861-862, 967, 998, 1030-1035, 1037
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    782      74  90.54%   317-318, 368, 658, 702-705, 743, 787, 821, 866-871, 897, 988, 1056, 1194, 1279-1291, 1327, 1329, 1357, 1361-1372, 1385-1391, 1474-1477, 1510-1530, 1582, 1599-1603, 1615-1618, 1954, 1978-1984, 2235, 2239-2243, 2252-2254, 2298-2299, 2328-2329
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                700      77  89.00%   131-133, 230, 262-263, 309-310, 320-332, 334, 345, 351-363, 410-411, 448, 469-472, 495, 503-504, 590-591, 615-616, 622-625, 640-641, 790, 836, 989, 998-1002, 1019-1024, 1151, 1165-1185, 1205, 1210, 1339, 1353, 1424, 1476, 1499-1500, 1503, 1516-1520, 1536, 1544, 1563
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       3  85.71%   48, 55, 68
plugins/manager.py                             172      15  91.28%   116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           97      10  89.69%   170-172, 219-220, 246-248, 301-302
plugins/apis/usgs.py                           180      31  82.78%   155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 448-454, 477
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   42-44, 48-51
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 92      16  82.61%   114, 143, 145, 176-189, 245-249
plugins/authentication/token_exchange.py        36      14  61.11%   75, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-112
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       7  76.67%   55-60, 63-64, 80-84
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 503, 535, 539, 546, 576-584, 588, 620-628, 639-641, 672-746, 764-824, 835-840, 852-865, 890, 905-907, 910, 920-928, 936-949, 959-990, 997-1009, 1047, 1073, 1118-1120, 1340
plugins/download/base.py                       258      51  80.23%   136, 164, 231-233, 300-301, 355-359, 365-366, 408, 411-425, 437, 441, 505-509, 539-540, 548-565, 572-580, 582-586, 629, 651, 673, 681
plugins/download/creodias_s3.py                 17       9  47.06%   53-67
plugins/download/http.py                       545     107  80.37%   237, 333-336, 339, 346-351, 369-384, 401, 413, 461, 468-474, 492, 506, 520, 528-530, 546-551, 562, 580, 623-626, 648, 688, 733, 747-753, 782-846, 864, 894-903, 939, 964-965, 984-989, 995, 998, 1015, 1032-1033, 1046, 1063-1064, 1071, 1133, 1148, 1207-1208, 1214, 1224, 1260, 1296, 1316, 1351-1353
plugins/download/s3rest.py                     116      24  79.31%   118, 152, 159, 194, 221-228, 231-233, 237, 248-254, 262-263, 266-270, 293, 314-317
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         146      11  92.47%   100, 104, 128-134, 274, 294, 433
plugins/search/build_search_result.py          341      50  85.34%   249, 277-278, 314, 317, 386-389, 478-495, 523, 558, 560, 585-592, 622-624, 628, 650, 685, 736, 750, 765-766, 781-796, 839, 864, 867, 870, 878, 997-998, 1004, 1015, 1073, 1124
plugins/search/cop_marine.py                   240      47  80.42%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445
plugins/search/creodias_s3.py                   55       3  94.55%   59, 77, 111
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     732     125  82.92%   450, 508, 522, 526-532, 560-564, 677-689, 734-737, 808-809, 857, 876, 883, 895, 952, 973, 976-977, 986-987, 996-997, 1006-1007, 1034, 1105-1110, 1114-1123, 1157, 1179, 1256, 1346, 1431-1432, 1442, 1519-1523, 1585, 1588, 1594-1595, 1616, 1644-1656, 1663, 1695-1697, 1707-1713, 1743, 1766, 1771-1772, 1787, 1803, 1871-1981
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  25       1  96.00%   35
rest/constants.py                                6       0  100.00%
rest/core.py                                   257     150  41.63%   157, 159, 161, 164-165, 179-189, 198-199, 205, 208, 249-303, 316-352, 383-420, 435-451, 467-476, 483-522, 539, 581-666, 705, 712-760
rest/errors.py                                  69      49  28.99%   60, 65-100, 105-108, 115-118, 126-147, 155-160, 175-181
rest/server.py                                 192     192  0.00%    18-600
rest/stac.py                                   319      68  78.68%   240, 306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 713, 778-782, 789, 843-844, 850, 905, 943, 976, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     180      20  88.89%   225-229, 262-264, 282, 285, 291, 295, 353, 371-386
rest/types/queryables.py                        57      13  77.19%   50-51, 58-59, 66-67, 93-98, 107-108, 162
rest/types/stac_search.py                      125      11  91.20%   136-138, 184, 199-201, 209, 213, 261, 264
rest/utils/__init__.py                          94      30  68.09%   82-88, 108, 111-112, 131-133, 146, 153, 178-186, 193-214
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       5  78.26%   43-44, 48, 60, 62
types/__init__.py                              128      44  65.62%   66, 70, 79-83, 94-106, 134-136, 143-148, 189, 203, 208, 230, 240-256, 261, 263, 285, 290, 298, 308
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                            108       1  99.07%   63
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 82      16  80.49%   131-134, 138-145, 157-163, 176-178
utils/__init__.py                              520      47  90.96%   85, 194-195, 204-231, 234, 249, 329-333, 406-410, 429-431, 444-458, 537, 552, 592-593, 957-960, 968-969, 1007-1008, 1055-1056, 1190
utils/exceptions.py                             46       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       8  78.95%   51, 53, 57, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9959    2094  78.97%

Diff against develop

Filename           Stmts    Miss  Cover
---------------  -------  ------  -------
api/core.py           +3      +2  -0.22%
types/whoosh.py      +67     +16  -19.51%
TOTAL                +70     +18  -0.04%

Results for commit: e98789a

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@alambare alambare force-pushed the fix-build-index-failure branch from e9aa94b to e98789a Compare December 12, 2024 20:10
@sbrunato sbrunato changed the title fix: handle exceptions occuring on failed index write fix(core): handle exceptions occuring on failed index write Dec 18, 2024
@sbrunato sbrunato merged commit 076a0f8 into develop Dec 18, 2024
10 checks passed
@sbrunato sbrunato deleted the fix-build-index-failure branch December 18, 2024 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working dedl DEDL related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants