Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can I use this repo to download many small http files or strings in parallel #79

Open
zydjohnHotmail opened this issue Dec 21, 2023 · 0 comments

Comments

@zydjohnHotmail
Copy link

I am writing to inquire about the suitability of your GitHub repository for a specific use case.

I am interested in using your repository to download multiple small files concurrently. The files I need to download are relatively small, ranging from 2KB to 5.6KB. I anticipate having up to 50 URLs, though typically no more than 15. The frequency of downloads varies throughout the day, with some periods having only 1 or 2 URLs, and occasionally none at all.

The URLs resemble the following examples:
http://worldtimeapi.org/Europe/London
http://worldtimeapi.org/Europe/Berlin ......

What I am trying to do is to send a group of http requests to different URLs, but the base URLs are the same, just like http://worldtimeapi.org/Europe/; but the last part of the URLs are different, like: London; Berlin.
I want to send all http requests and get responses in parallel, and process all responses within a short time, less than 1 second. So I need some multiple http download solutions. But my task in not like Free Download Manager, which will download a big file with chunks. Since all the files or json string from each URL is rather small, less than 5.6KB, so better to download each one in only one shot.
But the solution has to fix the http client for concurrency management, and other related issues.
These URLs represent endpoints that provide JSON data, and I need to download them quickly and efficiently, I also have to use Newtonsoft.json to parse the json format string get the necessary information. I plan to implement this functionality in C# using .NET 8.0 and Visual Studio 2022 on Windows 10.

Could you kindly confirm if your repository supports this use case? Additionally, if it does, I would greatly appreciate some general instructions on how to effectively leverage your repository for this purpose.

Thank you for your time and assistance. I look forward to your guidance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant