Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wis2-downloader filename should not derive filename from URL used in message, also to avoid duplicates #14

Closed
maaikelimper opened this issue Jun 3, 2024 · 2 comments
Closed

wis2-downloader filename should not derive filename from URL used in message, also to avoid duplicates #14

maaikelimper opened this issue Jun 3, 2024 · 2 comments
Assignees
@david-i-berry
Labels
bug Something isn't working

Comments

@maaikelimper
Copy link
Collaborator

maaikelimper commented Jun 3, 2024 •
edited
Loading

While subscribing to cache-topic, I received the following:
image

The difference in file-extension is due to the difference caches using different approaches for the download-url ...

We need to use the data_id to name the file, as this is the only value that uniquely identifies the data.

Also to avoid downloading the same data twice as the filename is used for de-duplication:
https://github.com/wmo-im/wis2-downloader/blob/6d1611be1de1724b85cccf64bec0784d7cf2c6ac/wis2downloader/downloader/__init__.py#L127-L135
@maaikelimper maaikelimper added the bug Something isn't working label Jun 3, 2024
david-i-berry added a commit that referenced this issue Jun 4, 2024
@david-i-berry
Updates to address
72adc2a 
- #15
- #14
- #13
@david-i-berry
Copy link
Member

david-i-berry commented Jun 7, 2024 •
edited
Loading

hash (sha256) of data id used as the filename. See:

https://github.com/wmo-im/wis2-downloader/blob/4eded2bcfdb88ab63b1646c0c9005d1e3800cacd/wis2downloader/downloader/__init__.py#L152-L153

Extension set based on link type, mapping performed by:

https://github.com/wmo-im/wis2-downloader/blob/4eded2bcfdb88ab63b1646c0c9005d1e3800cacd/wis2downloader/downloader/__init__.py#L75-L86

@david-i-berry david-i-berry mentioned this issue Jun 10, 2024
Merged
maaikelimper added a commit that referenced this issue Jun 12, 2024
@david-i-berry @maaikelimper
Refactor - non dev mode, disk space limits and file naming (#18)
2dfa7cc 
* Updates to address
- #15
- #14
- #13

* - Refactor of app.py allow running in none dev mode.
- Change to use get, post and delete to manage subscriptions
- addition of openapi doc

* Addition of swagger template to render openapi doc.
Update of README.md

* conditional tls-set

* Update of readme. Update ot GET, POST and DELETE methods, e.g. POST content now via request body.

Improved error handling and response codes.

* Tweaks to readme.

* Tweaks to readme.

* Update validate_topic.py

print input-topic in case of validation error

* Option to disable topic validation added.

* Unlimited disk usage possible by setting limit to 0.

* Reversion to using filename from global cache.
Switch to using minimum free space rather than maximum download dir size.

* Update to readme to include validat_topics option.

* Update to config.json and swagger page to allow customisable base url.

* Log levels configurable.

---------

Co-authored-by: Maaike <mlimper@wmo.int>
@david-i-berry
Copy link
Member

@maaikelimper can we close following the decision to keep with the filename and reopen if necessary?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@david-i-berry david-i-berry

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@maaikelimper @david-i-berry