[fmiweather] Add possibility to use fmi::forecast::edited in addition to fmi::forecast::harmonie

[masipila]

Problem / Motivation

The fmiweather binding fetches weather forecast from the Finnish Meteorology Instititute's (FMI) API. The binding uses currently the fmi::forecast::harmonie::surface::point::multipointcoverage stored query. The forecast provided as a response is based on the Harmonie weather model, which is different to what FMI publishes as their official forecast on their website and in their mobile app.

The meteorologists at FMI use multiple different weather models and their experience to provide an official FMI forecast which they publish on their website and in the FMI mobile app. This official forecast, which is often times more accurate than the Harmonie-based forecast, has been available via their API since 6 June 2023. During the past two winters, there have been cases where the Harmonie based forecast has been off by almost 10 degrees while the forecast provided by the FMI meteorologists have been accurate.

Having as accurate weather forecast as possible is cruicial for optimizing the heating of a house using openHAB. There are solutions like the openHAB Spot Price Optimizer to do this (I'm the maintainer of that solution). At the moment the Spot Price Optimizer doesn't use the FMI Binding, but has it's own javascript module to fetch the FMI forecast.

Proposed change

I would like to see this official FMI Weather Binding to allow users to be able to choose between the fmi::forecast::edited model and the current fmi::forecast::harmonie model.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/spot-price-optimizer-advanced-algorhitms-to-optimize-heating-charging-of-electric-vehicles-water-boilers-and-more/159539/14

[jlaur]

This query also needs a place parameter, which can be something like "helsinki". Also the response payload is different, so it's not a simple matter of adapting the request URL parameters, but also the handling of the response payload.

[masipila]

I had a (very) quick look at this. The binding seems to currently be using the multipointcoverage stored query. The "edited" forecast is also available as a multipointcoverage stored query.

Query against Harmonie weather model (I believe this is the URL structure that the binding currently uses)
https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::forecast::harmonie::surface::point::multipointcoverage&latlon=60.192059,24.945831&parameters=Temperature,Humidity,WindDirection,WindSpeedMS,WindGust,Pressure,Precipitation1h,TotalCloudCover,WeatherSymbol3

Similar query against the Edited forecast
https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::forecast::edited::weather::scandinavia::point::multipointcoverage&latlon=60.192059,24.945831&parameters=Temperature,Humidity,WindDirection,WindSpeedMS,WindGust,Pressure,Precipitation1h,TotalCloudCover,WeatherSymbol3

Disclaimer: As mentioned above, this is a result of very quick inspection. I have been using the fmi::forecast::edited::weather::scandinavia::point::simple stored query instead of fmi::forecast::harmonie::surface::point::multipointcoverage and the structure of the API response is significantly different between the simple and multipointcoverage versions.

[jlaur]

@masipila - can you try this: org.openhab.binding.fmiweather-4.3.0-SNAPSHOT.jar? It introduces a new configuration parameter so the stored query can be selected. I don't know if the descriptions make sense, please let me know if you have any suggestions.

[masipila]

Thanks @jlaur ! I'm travelling at the moment but I might have some time on Wednesday or Thursday.

Markus

[masipila]

Heya, I was able to install the snapshot JAR and noticed that it's now a configuration option to select which forecast model to use. :)

A bit stupid question since I haven't used this Binding or the other ones which are storing the forecasts as future time series (I just know that this capability has been around for a while now). This Binding has so many Channels that I'm confused what am I supposed to do to correctly create an Item which has the forecast stored as a proper time series?

[jlaur]

This Binding has so many Channels that I'm confused what am I supposed to do to correctly create an Item which has the forecast stored as a proper time series?

You need to persist the item you are using, see for example https://www.openhab.org/addons/bindings/energidataservice/#persisting-time-series.

[masipila]

Disregard the previous comment, I managed to figure it out.

If some other community members are wondering the same thing, it was as simple as to create an Item for the forecast#temperature channel and then configure the persistence startegy forecast for this newly created Item.

@jlaur : I'll continue my checks here but it looks very promising!

[masipila]

Allright, I compared the data I get with the Binding snapshot version and I get exactly the same temperature values compared to what I get with the openhab-spot-price-optimizer.

The only two difference that I noticed is that

a) the time horizon of the forecast with this binding is ~one day less than what I get with the other kind of API call that I use currently

b) the resolution of the persisted forecast with this binding is PT20M whereas I currently get PT60M.

But neither of these observations have anything to do with the scope of this feature request, they are explained by the fact that the API call is different to what I currently use and I can easily adapt to what the binding provides.

@jlaur : From my perspective this snapshot version is functionally fit for my purposes and good to go further!

[masipila]

@jlaur: And here's an example from tomorrow's forecast which illustrates the slight difference in the forecast made by a meteorologist compared to the "raw" Harmonie weather model.

Time will tell tomorrow which one is closer to the reality, but adding the configuration option to use the data from the fmi::forecast::edited dataset is a much welcome improvement IMHO.

Cheers,
Markus

[masipila]

Updated issue title summary to reflect the direction where users can choose between the edited model and harmonie model. I like this approach because it's fully backwards compatible for existing users of this binding.

[jlaur]

a) the time horizon of the forecast with this binding is ~one day less than what I get with the other kind of API call that I use currently

We could add a configuration parameter for this. This is part of the URL, but currently hardcoded:

openhab-addons/bundles/org.openhab.binding.fmiweather/src/main/java/org/openhab/binding/fmiweather/internal/ForecastWeatherHandler.java

Line 63 in ec998ac

private static final int FORECAST_HORIZON_HOURS = 50; // should be divisible by QUERY_RESOLUTION_MINUTES

b) the resolution of the persisted forecast with this binding is PT20M whereas I currently get PT60M.

See here:

openhab-addons/bundles/org.openhab.binding.fmiweather/src/main/java/org/openhab/binding/fmiweather/internal/ForecastWeatherHandler.java

Line 61 in ec998ac

private static final int QUERY_RESOLUTION_MINUTES = 20; // The channel group hours should be divisible by this

This one I don't understand, because the binding so far only published hourly forecasts, so it seems excessive to request three times more data than actually used.

I think we could add a parameter here as well, which would only be effective when one of the time series channels are linked. Otherwise we could use PT60M.

@ssalonen - WDYT?

But neither of these observations have anything to do with the scope of this feature request

Agreed, could be enhanced in another PR.

I like this approach because it's fully backwards compatible for existing users of this binding.

That was the idea and also why I kept harmonie as default.

[ssalonen]

@jlaur there is one detail with how the FMI API works that explains this detail. Indeed we query with 20min resolution but output only hourly forecasts.

FMI API "rounds" the start time to the future based on this "resolution". When we use 20min resolution, and round start and end time accordingly (1), the first data point pretty much matches current time.

If we would have 1h resolution, we might be off by one hour with the forecast. With 20min resolution, we are at most off by 20min. 20min was selected arbitrararily, compromise between data amount and this forecast "time accuracy" (in 20min, usually weather remains relatively stable)

(1)

openhab-addons/bundles/org.openhab.binding.fmiweather/src/main/java/org/openhab/binding/fmiweather/internal/ForecastWeatherHandler.java

Line 126 in 65004e5

return new ForecastRequest(location, floorToEvenMinutes(now, QUERY_RESOLUTION_MINUTES),

The API behaviour is perhaps best illustrated by these examples

#curl 'https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::forecast::harmonie::surface::point::multipointcoverage&starttime=2024-10-17T15:22:00Z&endtime=2024-10-17T18:22:00Z&timestep=60&latlon=60.19,24.94&parameters=Temperature,Humidity,WindDirection,WindSpeedMS,WindGust,Pressure,Precipitation1h,TotalCloudCover,WeatherSymbol3'
#starttime=2024-10-17T15:22:00Z&endtime=2024-10-17T18:22:00Z&timestep=60
#  <gml:TimePeriod gml:id="time-interval-1-1">
#                    <gml:beginPosition>2024-10-17T16:00:00Z</gml:beginPosition>
#                    <gml:endPosition>2024-10-17T18:00:00Z</gml:endPosition>
#                </gml:TimePeriod>

#curl 'https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::forecast::harmonie::surface::point::multipointcoverage&starttime=2024-10-17T15:00:00Z&endtime=2024-10-17T18:00:00Z&timestep=60&latlon=60.19,24.94&parameters=Temperature,Humidity,WindDirection,WindSpeedMS,WindGust,Pressure,Precipitation1h,TotalCloudCover,WeatherSymbol3'
#starttime=2024-10-17T15:00:00Z&endtime=2024-10-17T18:00:00Z&timestep=60
#<gml:TimePeriod gml:id="time-interval-1-1">
#                    <gml:beginPosition>2024-10-17T15:00:00Z</gml:beginPosition>
#                    <gml:endPosition>2024-10-17T18:00:00Z</gml:endPosition>
#                </gml:TimePeriod>

#curl 'https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::forecast::harmonie::surface::point::multipointcoverage&starttime=2024-10-17T15:01:00Z&endtime=2024-10-17T18:01:00Z&timestep=120&latlon=60.19,24.94&parameters=Temperature,Humidity,WindDirection,WindSpeedMS,WindGust,Pressure,Precipitation1h,TotalCloudCover,WeatherSymbol3'
#starttime=2024-10-17T15:01:00Z&endtime=2024-10-17T18:01:00Z&timestep=120
#                <gml:TimePeriod gml:id="time-interval-1-1">
#                    <gml:beginPosition>2024-10-17T16:00:00Z</gml:beginPosition>
#                    <gml:endPosition>2024-10-17T18:00:00Z</gml:endPosition>
#                </gml:TimePeriod>


#curl 'https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::forecast::harmonie::surface::point::multipointcoverage&starttime=2024-10-17T15:00:00Z&endtime=2024-10-17T18:00:00Z&timestep=120&latlon=60.19,24.94&parameters=Temperature,Humidity,WindDirection,WindSpeedMS,WindGust,Pressure,Precipitation1h,TotalCloudCover,WeatherSymbol3'
#starttime=2024-10-17T15:00:00Z&endtime=2024-10-17T18:00:00Z&timestep=120
#                <gml:TimePeriod gml:id="time-interval-1-1">
#                    <gml:beginPosition>2024-10-17T16:00:00Z</gml:beginPosition>
#                    <gml:endPosition>2024-10-17T18:00:00Z</gml:endPosition>
#                </gml:TimePeriod>

#curl 'https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::forecast::harmonie::surface::point::multipointcoverage&starttime=2024-10-17T14:20:00Z&endtime=2024-10-17T21:00:00Z&timestep=20&latlon=60.19,24.94&parameters=Temperature,Humidity,WindDirection,WindSpeedMS,WindGust,Pressure,Precipitation1h,TotalCloudCover,WeatherSymbol3'
#starttime=2024-10-17T14:20:00Z&endtime=2024-10-17T21:00:00Z&timestep=20
#                <gml:TimePeriod gml:id="time-interval-1-1">
#                    <gml:beginPosition>2024-10-17T14:20:00Z</gml:beginPosition>
#                    <gml:endPosition>2024-10-17T21:00:00Z</gml:endPosition>
#                </gml:TimePeriod>

[ssalonen]

Re. hardcoded forecast horizon, I was gauging true available forecast with a simple script #13693 (comment)

HIRLAM was available at the time for 59h always, sometimes up to 63h. Note how available forecasts depend the hour-of-day as the forecast model is run certain time of day.

The HIRLAM PR only introduced channels for up to 50h only...as commented here "additional channels...can be done in a separate PR" #13693 (comment) . Separate PR never came :D

[jlaur]

there is one detail with how the FMI API works that explains this detail.

Thank you for your very detailed explanations. I'm on a short holiday, but will have a closer look when I'm back home, where it will be easier to check things out. 🙂