Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up data retrieval process by using yearly freq #124

Closed
veenstrajelmer opened this issue Aug 28, 2024 · 1 comment
Closed

Speed up data retrieval process by using yearly freq #124

veenstrajelmer opened this issue Aug 28, 2024 · 1 comment

Comments

@veenstrajelmer
Copy link
Collaborator

veenstrajelmer commented Aug 28, 2024

With the many duplicates documented in Rijkswaterstaat/wm-ws-dl#39, it is not always possible to retrieve timeseries per year. The maximum allowed of values in a returned timeseries by the DDL is 157681 and because of duplicates some year timeseries were larger. However, many duplicates were cleaned by now, so it should be possible to retrieve these without issues now. First check this by retrieving all measurements amounts again.

This will probably be somewhat faster, although longer timeseries also take longer to retrieve from DDL. When retrieving monthly timeseries, all 1848 iterations take approximately 1.5 seconds, totalling to approximately 40 minutes. When retrieving yearly timeseries, the empty or low-coverage years take 1.5 seconds, but the 10-minute years might take up to 20 seconds per iterations. For HOEKVHLD this totals to 17 minutes. All timings on WCF, seems to be significantly faster on laptop.

@veenstrajelmer veenstrajelmer mentioned this issue Aug 28, 2024
9 tasks
@veenstrajelmer
Copy link
Collaborator Author

Too tricky to depend on this for now. Already with DONAR, and we do not know how performant WADAR will be.

@veenstrajelmer veenstrajelmer closed this as not planned Won't fix, can't repro, duplicate, stale Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant