-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance tuning #115
Comments
Hard to say. Arrows/barbs need to read twice as much data as single variable fields (the components are calculated on-the-fly), so they will always be slower than average rendering. Generally speaking, the biggest bottleneck is reading the data from disk. How is your data available within your VM? |
Hard disk. But IO is measured via IOPS. So I will try increasing IOPS. |
I doubled everything (cpu/ram, iops) and only got marginal speedup. There must be some caching to improve things. I set caching at 4GB. Note that our clients do not like tiling so I am using the single image capability of OpenLayers. It all works great but I would like to speed it up. So fyi if mag/dir are precomputed and then just used in a group and the arrow/barb is just rotated by SLD then it would be much faster. You said this was a future idea. How far out in the future is it? |
Yes, single images will almost always have worse performance than tiles, because of the way the cache works (we cache the data read on each request). If a different area is requested each time, as is the case with single images, the cache rarely gets used. If you precompute mag/dir (in the NetCDF files), it will speed things up (once I implement that), but if you are plotting both magnitude and direction, then both fields will still need to be read, so you won't see much of an improvement. If you are only plotting direction then in theory it should be slightly over twice as quick to render. I don't currently have a great deal of time to work on ncWMS, but if you'd be happy using a development build, I could add automatic grouping of magnitude and direction pretty quickly. If you have an example data file containing a magnitude and direction fields for a single variable, that would be very helpful. |
I would be glad to be your tester. I have lots of files with mag,dir as that is how 'Mean wave direction' (mwd) and 'Significant height of combined wind waves and swell' (swh) come. |
OK, that's an interesting one. It's obviously sensible to plot wave height and direction together, but I don't think they're fundamentally related in the same way that speed and direction are. I've got a solution in mind which could work, but it will need some finessing. Let me see what I can do for a quick fix and you could test it to see whether it solves your issue. |
Were you able to download the file and see the data? |
Yes, thanks. I managed to get them grouped manually, and behaving properly, but there are some complications trying to pick up the grouping automatically. Unfortunately I now have a couple of deadlines coming up so won't be able to work on it for a week or two. |
Very cool. I am wondering how hard is it for there to be a menu on the ingestion side to allow the end user to select properties to be grouped rather than having the code detect it automatically. Or it is simple enough to add a new metadata field to the netcdf file called GroupWith and then some name as the value. Since this is a quick fix the second approach might be easiest for you. |
The approach I was considering was a way to define a style such that if a dataset could create a new grouping to support it, it would. However, your GroupWith approach could be a good one. We did something similar a few years back with NetCDF-U, which was a way of defining relations between quantities and their uncertainties. I'll look into it when I get a little more time. |
Hi, I am ready to test it when you get a chance to gen something. I now have two different data sets with the same issue. My new data is ocean current data that has a direction and magnitude. |
Apologies, I don't have much time to work on ncWMS at the moment. I did manage to grab a couple of hours this afternoon to try something. You can download the development version from http://www.personal.rdg.ac.uk/~qx901922/ncWMS2.war Basically to get it to work, you need a few different attributes on your variables. You will need the Once that's up-and-running, you should find that your variables appear in groups in the menu, and you can plot them as usual, but clicking the parent variable will do nothing. I'd recommend you get to this stage before moving on. Next, you need a custom style. If you create
then ncWMS should look for groups containing children with the Restart ncWMS and then the top-level group should be clickable and plot swell + direction as expected. This was all rather rushed, so just drop me a line if you get any issues. Assuming it all works, I'll tidy it up and some point and make it into a proper release |
@guygriffiths , Thank you for spending time on this. I am trying to test it out, but I seem to have broken something. Regardless of the ncWMS version I use and I get the following error: I solved this issue by rebooting the VM. Not sure what happened to it, but I suspect MS changed something. |
Hi @guygriffiths , I do not know why I can't run on the ubuntu server, but I am able to run on my windows desktop using the windows linux support that now exists. So i made the updates you suggested for logical_group and every variable had a standard_name. I loaded the file into ncWMS ok, but no group names were displayed. Here is an ncdump of the file: I will also upload the file for your access at: Sorry, it is a large file. Please let me know what I am doing wrong. |
Sorry, I forgot to mention that you will need to remove and re-add the dataset - just modifying it won't make a difference. |
No, I tried to download it but the link had already expired. If you can make it available again I'll try running it on my version. |
https://i4insightcom-my.sharepoint.com/:u:/g/personal/greg_turner_i4-insight_com/EXs3RzAxsY5GnN7XHImaU4wBpyaou6cPj7_cIl58--5Dhg?e=yoBuM7
Gregory Turner
Chief Architect
…________________________________________
i4 Insight, Inc.
cell: +1.720.335.1744
greg.turner@i4-insight.com<mailto:greg.turner@i4-insight.com>
A member of the Lloyd’s Register Group
From: Guy Griffiths <notifications@github.com>
Sent: Thursday, March 14, 2019 3:35 AM
To: Reading-eScience-Centre/edal-java <edal-java@noreply.github.com>
Cc: Greg Turner <greg.turner@i4-insight.com>; Author <author@noreply.github.com>
Subject: Re: [Reading-eScience-Centre/edal-java] Performance tuning (#115)
No, I tried to download it but the link had already expired. If you can make it available again I'll try running it on my version.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#115 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ApVpH2tObcgCxlJYOg3Gc_yGYvwCzKNTks5vWhffgaJpZM4aEmnl>.
|
@guygriffiths , Hi, I know you are busy, but I am running into another issue with the new version. I am unable to select any color palette other than the default one for temperature and 'mean sea level pressure'. Hopefully, I am doing something wrong that is simple. I get the following error when I choose another palette: You should see the issue in the file i sent you. |
@docgregt - The file you supplied doesn't work with grouping, because you have tried to group multiple variables together with the same standard name ( I suspect that they are caused by the way that you have replaced the |
@guygriffiths - I downloaded the latest jar and i fixed the .nc file and no success. The errors are still there and there still is no grouping. I uploaded the latest .nc file for your debugging. Here is dump of .nc file.
|
Hi Guy, I have tried everything I can think of to get it to work. I just don't know what to do. I really need your help with it. Thanks, |
I am not sure where to ask this question. I searched quite a while for info, but could not find any. Rendering arrows or barbs can take a few seconds on my 4 core 16GB RAM VM. I only have 10 datasets loaded and they are at .75x.75 degree resolution. What kind of hardware will speed it up? Or is there some Java setting to speed it up? What affects the performance the most?
Greg
The text was updated successfully, but these errors were encountered: