-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speed up data retrieval process by using yearly freq #124
Comments
Too tricky to depend on this for now. Already with DONAR, and we do not know how performant WADAR will be. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With the many duplicates documented in Rijkswaterstaat/wm-ws-dl#39, it is not always possible to retrieve timeseries per year. The maximum allowed of values in a returned timeseries by the DDL is 157681 and because of duplicates some year timeseries were larger. However, many duplicates were cleaned by now, so it should be possible to retrieve these without issues now. First check this by retrieving all measurements amounts again.
This will probably be somewhat faster, although longer timeseries also take longer to retrieve from DDL. When retrieving monthly timeseries, all 1848 iterations take approximately 1.5 seconds, totalling to approximately 40 minutes. When retrieving yearly timeseries, the empty or low-coverage years take 1.5 seconds, but the 10-minute years might take up to 20 seconds per iterations. For HOEKVHLD this totals to 17 minutes. All timings on WCF, seems to be significantly faster on laptop.
The text was updated successfully, but these errors were encountered: