improve filling messaging for knmi

[dbrakenhoff]

Small improvement to better understand what's going on :)

[OnnoEbbens]

I think the information in your new 'info' log was also in two other 'debug' logs:

• logger.debug(f"station {stn} has {missing.sum()} missing measurements")
• logger.debug(f"Trying to fill {n_missing} missing measurements with station {stn_comp}")

I remember these were 'info' logs before but we changed the level to 'debug' because people didn't like all these messages.

I think your message is more clear so we can remove the debug messages. I don't really know what is the best log level is so we can keep it at 'info' if you like.

[dbrakenhoff]

I thought the current messaging on INFO level was a bit unclear. Often times it wasn't clear to me which station was being used to fill, and how many observations were being filled. On DEBUG level you ran into the fact that the logger is set to DEBUG for all packages in get_color_logger(), causing the number of messages to become so great, you could not figure out what was happening anymore. That last one can be fixed by only getting the color logger for hydropandas using the relatively new logger_name kwarg. Having said that, the word "trying" in the debug message also bothered me a bit, since the successful completion wasn't really logged.

Below is my current take on the messaging levels for this particular case. Though to be fair it isn't always clear which one is most appropriate.

• INFO for some indication of what task is being performed and anything that modifies the time series (different end dates, filling, etc.)
• WARNING for partly unsuccessful completion of the requested task (could not fill, no data, etc.)
• ERROR for failure in completing the task
• DEBUG to view a breakdown of each (or at least a bigger number of) steps

I think this change makes it clear on INFO level what is happening, but you're right the debug level maybe has duplicated messaging now. I will check that out.

[dbrakenhoff]

The DEBUG log trace for a KNMI download with fill missing obs:

DEBUG:hydropandas.io.knmi._get_default_settings:254:set raise_exceptions=False because fill_missing_obs is True
INFO:hydropandas.io.knmi.get_knmi_obs:100:get data from station 275 and variable EV24 from 1958-01-01 to 2021-09-30 01:00:00
DEBUG:hydropandas.io.knmi.download_knmi_data:734:download KNMI EV24 data from station 275-DEELEN between 1958-01-01 00:00:00 and 2021-09-30 01:00:00
DEBUG:hydropandas.io.knmi._transform_variables:968:transform EV24, Referentiegewasverdamping (Makkink) (in 0.1 mm) [/](https://file+.vscode-resource.vscode-cdn.net/) Potential evapotranspiration (Makkink) (in 0.1 mm) from 0.1 to 1
DEBUG:hydropandas.io.knmi._transform_variables:978:transform EV24, Referentiegewasverdamping (Makkink) (in mm) [/](https://file+.vscode-resource.vscode-cdn.net/) Potential evapotranspiration (Makkink) (in mm) from mm to m
INFO:hydropandas.io.knmi._add_missing_indices:1634:station 275 has no measurements before 1987-04-11 01:00:00
DEBUG:hydropandas.io.knmi.fill_missing_measurements:605:station 275 has 10692 missing measurements
DEBUG:hydropandas.io.knmi.fill_missing_measurements:633:Trying to fill 10692 missing measurements with station 265 SOESTERBERG
DEBUG:hydropandas.io.knmi.download_knmi_data:734:download KNMI EV24 data from station 265-SOESTERBERG between 1958-01-01 00:00:00 and 2021-09-30 01:00:00
DEBUG:hydropandas.io.knmi._transform_variables:968:transform EV24, Referentiegewasverdamping (Makkink) (in 0.1 mm) [/](https://file+.vscode-resource.vscode-cdn.net/) Potential evapotranspiration (Makkink) (in 0.1 mm) from 0.1 to 1
DEBUG:hydropandas.io.knmi._transform_variables:978:transform EV24, Referentiegewasverdamping (Makkink) (in mm) [/](https://file+.vscode-resource.vscode-cdn.net/) Potential evapotranspiration (Makkink) (in mm) from mm to m
DEBUG:hydropandas.io.knmi.fill_missing_measurements:663:No new data available from 265 SOESTERBERG for filling missing measurements
DEBUG:hydropandas.io.knmi.fill_missing_measurements:633:Trying to fill 10692 missing measurements with station 375 VOLKEL
DEBUG:hydropandas.io.knmi.download_knmi_data:734:download KNMI EV24 data from station 375-VOLKEL between 1958-01-01 00:00:00 and 2021-09-30 01:00:00
DEBUG:hydropandas.io.knmi._transform_variables:968:transform EV24, Referentiegewasverdamping (Makkink) (in 0.1 mm) [/](https://file+.vscode-resource.vscode-cdn.net/) Potential evapotranspiration (Makkink) (in 0.1 mm) from 0.1 to 1
DEBUG:hydropandas.io.knmi._transform_variables:978:transform EV24, Referentiegewasverdamping (Makkink) (in mm) [/](https://file+.vscode-resource.vscode-cdn.net/) Potential evapotranspiration (Makkink) (in mm) from mm to m
DEBUG:hydropandas.io.knmi.fill_missing_measurements:663:No new data available from 375 VOLKEL for filling missing measurements
DEBUG:hydropandas.io.knmi.fill_missing_measurements:633:Trying to fill 10692 missing measurements with station 278 HEINO
DEBUG:hydropandas.io.knmi.download_knmi_data:734:download KNMI EV24 data from station 278-HEINO between 1958-01-01 00:00:00 and 2021-09-30 01:00:00
DEBUG:hydropandas.io.knmi._transform_variables:968:transform EV24, Referentiegewasverdamping (Makkink) (in 0.1 mm) [/](https://file+.vscode-resource.vscode-cdn.net/) Potential evapotranspiration (Makkink) (in 0.1 mm) from 0.1 to 1
DEBUG:hydropandas.io.knmi._transform_variables:978:transform EV24, Referentiegewasverdamping (Makkink) (in mm) [/](https://file+.vscode-resource.vscode-cdn.net/) Potential evapotranspiration (Makkink) (in mm) from mm to m
DEBUG:hydropandas.io.knmi.fill_missing_measurements:663:No new data available from 278 HEINO for filling missing measurements
DEBUG:hydropandas.io.knmi.fill_missing_measurements:633:Trying to fill 10692 missing measurements with station 260 DE-BILT
DEBUG:hydropandas.io.knmi.download_knmi_data:734:download KNMI EV24 data from station 260-DE-BILT between 1958-01-01 00:00:00 and 2021-09-30 01:00:00
DEBUG:hydropandas.io.knmi._transform_variables:968:transform EV24, Referentiegewasverdamping (Makkink) (in 0.1 mm) [/](https://file+.vscode-resource.vscode-cdn.net/) Potential evapotranspiration (Makkink) (in 0.1 mm) from 0.1 to 1
DEBUG:hydropandas.io.knmi._transform_variables:978:transform EV24, Referentiegewasverdamping (Makkink) (in mm) [/](https://file+.vscode-resource.vscode-cdn.net/) Potential evapotranspiration (Makkink) (in mm) from mm to m
INFO:hydropandas.io.knmi.fill_missing_measurements:658:Filled 10692 observations from station 260 DE-BILT -> 275 DEELEN

The INFO trace then is

INFO:hydropandas.io.knmi.get_knmi_obs:100:get data from station 275 and variable EV24 from 1958-01-01 to 2021-09-30 01:00:00
INFO:hydropandas.io.knmi._add_missing_indices:1634:station 275 has no measurements before 1987-04-11 01:00:00
INFO:hydropandas.io.knmi.fill_missing_measurements:658:Filled 10692 observations from station 260 DE-BILT -> 275 DEELEN

[dbrakenhoff]

I think it's pretty clear this way, and I don't mind the duplication in the DEBUG level really, it seems clear what it's doing :).