add md5 check for downloaded JGI file #9
Conversation
|
Sometime curl exit with code 0 after finished download file, but the file in fact is not intact, maybe the network problem cause this. So validated the completeness of downloaded file by md5 value maybe needed, that is why this pull request is here. |
orangeSi
changed the title
| @@ -577,32 +582,50 @@ def is_broken(filename, min_size_bytes=20): | |||
| else: | |||
| return False | |||
|
|
|||
| def check_md5(md5, filename): | |||
There was a problem hiding this comment.
I'm going to replace this with a native Python implementation in the interest of cross-platform compatibility.
| @@ -219,7 +219,7 @@ def get_file_list(xml_file, filter_categories=False): | |||
| except KeyError: | |||
| continue | |||
| uid += 1 | |||
|
|
|||
| #print(f"descriptors={descriptors}") ##myth | |||
There was a problem hiding this comment.
Don't leave commented-out code in the PR.
| #print(f"descriptors={descriptors}") ##myth |
| @@ -391,6 +393,9 @@ def print_data(data, org_name, display=True): | |||
| print_list.append("{}:".format(sub_cat)) | |||
| for index, i in sorted(items.items()): | |||
| dict_to_get[catID][index] = i["url"] | |||
| url_to_md5[i["url"]] = i["md5"] | |||
| #print(f"md5={i['md5']}, url={i['url']}, catID={catID}, index={index}\n")#myth | |||
There was a problem hiding this comment.
Remove commented-out code
| #print(f"md5={i['md5']}, url={i['url']}, catID={catID}, index={index}\n")#myth |
|
Commit d89dc10 integrates my implementation of the changes initialized with this PR. |
I didn't say before, but this is a good feature I've been meaning to add for some time. Thanks for your initial work setting it up—I modified it slightly to match the style/approach of the script overall, but the general design was good. |
No description provided.