Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(plugins): auth only when needed during http download #1370

Merged
merged 3 commits into from
Nov 4, 2024

Conversation

jlahovnik
Copy link
Collaborator

@jlahovnik jlahovnik commented Oct 25, 2024

  • So far if the download of one asset was failing, this was logged but no error was raised which hid problems during the download and could lead to incomplete or even empty results while the response was still 200. This is changed now:
    → in HTTPDownload if a problem occurs during an asset download, a Download error is raised and the whole product download is interrupted.
  • After this change it became visible that for planetary_computer some assets where not downloaded because, for the same product, there are assets where authentication is required for the download and assets where authentication should not be added.
    → In HTTPDownload, the asset url is now compared to matching_url from the config to check if authentication should be added to the request.

@jlahovnik jlahovnik self-assigned this Oct 25, 2024
Copy link
Contributor

github-actions bot commented Oct 25, 2024

Test Results

    4 files  ±0      4 suites  ±0   5m 52s ⏱️ ±0s
  573 tests +2    570 ✅ +2   3 💤 ±0  0 ❌ ±0 
2 292 runs  +8  2 196 ✅ +8  96 💤 ±0  0 ❌ ±0 

Results for commit 3845bc2. ± Comparison against base commit e20c12b.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Oct 25, 2024

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         314      59  81.21%   657-716, 818-869, 873
config.py                                      429      27  93.71%   83-85, 94, 102, 106-108, 179, 190, 695-697, 810-813, 856-857, 866-867, 972, 1035-1040, 1042
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    767      69  91.00%   374, 664, 708-711, 749, 793, 827, 872-877, 903, 994, 1062, 1200, 1285-1297, 1333, 1335, 1363, 1367-1378, 1391-1397, 1480-1483, 1516-1536, 1588, 1605-1609, 1621-1624, 1980, 2004-2010, 2261, 2265-2269, 2278-2280, 2312
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                680      68  90.00%   130-132, 227, 259-260, 306-307, 317-329, 331, 342, 348-360, 405-406, 443, 464-467, 490, 498-499, 575-576, 600-601, 607-610, 625-626, 775, 821, 992-997, 1124, 1138-1158, 1178, 1183, 1312, 1326, 1397, 1449, 1489-1493, 1508
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       2  90.48%   48, 55
plugins/manager.py                             172      15  91.28%   116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           91       8  91.21%   159-161, 208-209, 235-237
plugins/apis/usgs.py                           180      31  82.78%   155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 448-454, 477
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   42-44, 48-51
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 92      16  82.61%   114, 143, 145, 176-189, 245-249
plugins/authentication/token_exchange.py        36      20  44.44%   72-78, 90-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47       8  82.98%   52-53, 69, 78-81, 83, 90-93
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       7  76.67%   55-60, 63-64, 80-84
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 503, 535, 539, 546, 576-584, 588, 620-628, 639-641, 672-746, 764-824, 835-840, 852-865, 890, 905-907, 910, 920-928, 936-949, 959-990, 997-1009, 1047, 1073, 1118-1120, 1340
plugins/download/base.py                       253      51  79.84%   136, 164, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677
plugins/download/creodias_s3.py                 17       9  47.06%   53-67
plugins/download/http.py                       537     108  79.89%   235, 331-334, 336-337, 344-349, 367-382, 399-401, 413, 461, 468-474, 492, 506, 520, 528-530, 546-551, 562, 580, 623-626, 648, 688, 733, 747-753, 782-846, 864, 894-903, 925-926, 955-960, 966, 969, 986, 1003-1004, 1034-1035, 1042, 1104, 1119, 1178-1179, 1185, 1195, 1231, 1267, 1287, 1322-1324
plugins/download/s3rest.py                     116      24  79.31%   118, 154, 161, 196, 223-230, 233-235, 239, 250-256, 264-265, 268-272, 295, 316-319
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         132      13  90.15%   100, 104, 115, 274, 294, 350-351, 371, 374-382
plugins/search/build_search_result.py          188      24  87.23%   100, 141-142, 148, 159, 292-295, 324, 359-376, 425, 454, 457, 467, 484, 512, 514
plugins/search/cop_marine.py                   240      47  80.42%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445
plugins/search/creodias_s3.py                   55       3  94.55%   58, 76, 110
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     774     103  86.69%   458, 516, 530, 534-540, 568-572, 681-693, 737, 753, 763, 782-797, 834-837, 908-909, 957, 976, 983, 995, 1052, 1073, 1076-1077, 1086-1087, 1096-1097, 1106-1107, 1134, 1205-1210, 1214-1223, 1257, 1279, 1356, 1446, 1533, 1536-1537, 1619-1623, 1685, 1688, 1694-1695, 1716, 1743-1755, 1762, 1794-1796, 1806-1812, 1842, 1865, 1870-1871, 1886, 1902, 1987-1990, 1995-1998, 2007, 2027-2028, 2037-2041, 2047
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  25       0  100.00%
rest/constants.py                                6       0  100.00%
rest/core.py                                   251      63  74.90%   216-217, 273, 281, 299-316, 331-369, 460, 502-533, 680, 687-735
rest/errors.py                                  69       5  92.75%   106, 116, 127, 143-144
rest/server.py                                 188      24  87.23%   89, 112-114, 277-282, 310, 495-497, 514-519, 548, 550, 554-555, 559-560
rest/stac.py                                   319      63  80.25%   306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 778-782, 789, 843-844, 905, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     176       5  97.16%   225-229, 282, 285, 353
rest/types/queryables.py                        55       1  98.18%   163
rest/types/stac_search.py                      125       7  94.40%   138, 184, 199-201, 209, 213
rest/utils/__init__.py                          93      12  87.10%   108-109, 128-130, 182, 192-206
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       3  86.96%   48, 60, 62
types/__init__.py                              114      14  87.72%   65, 82, 141-144, 211, 225-234, 244, 265, 278
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             80       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      36  92.83%   85, 193-194, 203-230, 233, 248, 328-332, 405-409, 428-430, 509, 524, 564-565, 929-932, 940-941, 979-980, 1151
utils/constraints.py                           119      38  68.07%   62, 89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244
utils/exceptions.py                             41       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   30       8  73.33%   36, 38, 42, 76, 94-101
utils/requests.py                               55      11  80.00%   64, 86, 88, 90, 92, 94, 110, 118-120, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9734    1598  83.58%

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
plugins/download/http.py       +3     -12  +2.36%
TOTAL                          +3     -12  +0.13%

Results for commit: 3845bc2

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented Oct 25, 2024

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         314      59  81.21%   657-716, 818-869, 873
config.py                                      429      28  93.47%   83-85, 94, 102, 106-108, 179, 190, 695-697, 810-813, 856-857, 866-867, 972, 1003, 1035-1040, 1042
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    767      69  91.00%   374, 664, 708-711, 749, 793, 827, 872-877, 903, 994, 1062, 1200, 1285-1297, 1333, 1335, 1363, 1367-1378, 1391-1397, 1480-1483, 1516-1536, 1588, 1605-1609, 1621-1624, 1980, 2004-2010, 2261, 2265-2269, 2278-2280, 2312
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                680      68  90.00%   130-132, 227, 259-260, 306-307, 317-329, 331, 342, 348-360, 405-406, 443, 464-467, 490, 498-499, 575-576, 600-601, 607-610, 625-626, 775, 821, 992-997, 1124, 1138-1158, 1178, 1183, 1312, 1326, 1397, 1449, 1489-1493, 1508
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       3  85.71%   48, 55, 68
plugins/manager.py                             172      15  91.28%   116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           91       8  91.21%   159-161, 208-209, 235-237
plugins/apis/usgs.py                           180      31  82.78%   155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 448-454, 477
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   42-44, 48-51
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 92      16  82.61%   114, 143, 145, 176-189, 245-249
plugins/authentication/token_exchange.py        36      20  44.44%   72-78, 90-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-112
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       7  76.67%   55-60, 63-64, 80-84
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 503, 535, 539, 546, 576-584, 588, 620-628, 639-641, 672-746, 764-824, 835-840, 852-865, 890, 905-907, 910, 920-928, 936-949, 959-990, 997-1009, 1047, 1073, 1118-1120, 1340
plugins/download/base.py                       253      53  79.05%   136, 164, 231-233, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677
plugins/download/creodias_s3.py                 17       9  47.06%   53-67
plugins/download/http.py                       537     109  79.70%   235, 331-334, 336-337, 344-349, 367-382, 399-401, 413, 461, 468-474, 492, 506, 520, 528-530, 546-551, 562, 580, 623-626, 648, 688, 733, 747-753, 782-846, 864, 894-903, 925-926, 955-960, 966, 969, 986, 1003-1004, 1017, 1034-1035, 1042, 1104, 1119, 1178-1179, 1185, 1195, 1231, 1267, 1287, 1322-1324
plugins/download/s3rest.py                     116      24  79.31%   118, 154, 161, 196, 223-230, 233-235, 239, 250-256, 264-265, 268-272, 295, 316-319
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         132      14  89.39%   100, 104, 115, 274, 294, 350-351, 371, 374-382, 384
plugins/search/build_search_result.py          188      31  83.51%   100, 141-142, 148, 159, 292-295, 324, 359-376, 425, 454, 457, 467, 484, 504-519
plugins/search/cop_marine.py                   240      47  80.42%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445
plugins/search/creodias_s3.py                   55       3  94.55%   58, 76, 110
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     774     135  82.56%   458, 516, 530, 534-540, 568-572, 681-693, 737, 740, 753, 763, 782-797, 834-837, 908-909, 957, 976, 983, 995, 1052, 1073, 1076-1077, 1086-1087, 1096-1097, 1106-1107, 1134, 1205-1210, 1214-1223, 1257, 1279, 1356, 1446, 1533, 1536-1537, 1619-1623, 1685, 1688, 1694-1695, 1716, 1743-1755, 1762, 1794-1796, 1806-1812, 1842, 1865, 1870-1871, 1886, 1902, 1970-2076
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  25       1  96.00%   35
rest/constants.py                                6       0  100.00%
rest/core.py                                   251     139  44.62%   162, 164, 166, 169-170, 184-194, 206-207, 209-210, 216-217, 220, 223, 264-318, 331-369, 400-434, 449-465, 481-490, 502-533, 550, 592-641, 680, 687-735
rest/errors.py                                  69      49  28.99%   60, 65-100, 105-108, 115-118, 126-147, 155-160, 175-181
rest/server.py                                 188     188  0.00%    18-573
rest/stac.py                                   319      68  78.68%   240, 306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 713, 778-782, 789, 843-844, 850, 905, 943, 976, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     176      16  90.91%   225-229, 262-264, 282, 285, 291, 295, 353, 370-380
rest/types/queryables.py                        55      13  76.36%   50-51, 58-59, 66-67, 93-98, 107-108, 163
rest/types/stac_search.py                      125      11  91.20%   136-138, 184, 199-201, 209, 213, 261, 264
rest/utils/__init__.py                          93      30  67.74%   79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       5  78.26%   43-44, 48, 60, 62
types/__init__.py                              114      39  65.79%   65, 78-82, 93-105, 132-134, 141-144, 184, 211, 221-237, 242, 244, 265, 270, 278, 288
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             80       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      36  92.83%   85, 193-194, 203-230, 233, 248, 328-332, 405-409, 428-430, 509, 524, 564-565, 929-932, 940-941, 979-980, 1151
utils/constraints.py                           119      38  68.07%   62, 89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244
utils/exceptions.py                             41       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   30       8  73.33%   36, 38, 42, 76, 94-101
utils/requests.py                               55      11  80.00%   64, 86, 88, 90, 92, 94, 110, 118-120, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9734    2045  78.99%

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
plugins/download/http.py       +3     -12  +2.36%
TOTAL                          +3     -12  +0.13%

Results for commit: 3845bc2

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@alambare alambare requested a review from sbrunato October 28, 2024 09:01
@alambare alambare added the dedl DEDL related label Oct 28, 2024
@sbrunato sbrunato marked this pull request as draft October 31, 2024 13:21
@jlahovnik jlahovnik marked this pull request as ready for review October 31, 2024 16:00
@jlahovnik jlahovnik requested a review from sbrunato October 31, 2024 16:00
Copy link
Collaborator

@sbrunato sbrunato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was failing if matching_url or matching_conf was not set

@sbrunato sbrunato marked this pull request as draft November 4, 2024 08:42
@jlahovnik jlahovnik marked this pull request as ready for review November 4, 2024 10:54
@jlahovnik jlahovnik requested a review from sbrunato November 4, 2024 10:54
@sbrunato sbrunato changed the title fix: adaptions for problems with asset download fix(plugins): adaptions for problems with asset download Nov 4, 2024
@sbrunato sbrunato changed the title fix(plugins): adaptions for problems with asset download fix(plugins): auth only when needed during http download Nov 4, 2024
@sbrunato sbrunato merged commit 6b6d64c into develop Nov 4, 2024
10 checks passed
@sbrunato sbrunato deleted the asset-download-errors branch November 4, 2024 13:32
@sbrunato sbrunato added this to the 3.0.1.dev milestone Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dedl DEDL related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants