Implement dynamic faiman model and function to fit this to measurements

[adriesse]

• Closes Function to fit Faiman temperature model to measurements #1877
• I am familiar with the contributing guidelines
• Tests added
• Updates entries in docs/sphinx/source/reference for API changes.
• Adds description and name entries in the appropriate "what's new" file in docs/sphinx/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
• New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
• Pull request is nearly complete and ready for detailed review.
• Maintainer: Appropriate GitHub Labels (including remote-data) and Milestone are assigned to the Pull Request and linked Issue.

[adriesse]

@pvlib/pvlib-maintainer This PR request is ready for review.

[cwhanse]

This extension of faiman smooths the inputs, and prilliman smooths the output, both are intended to account for thermal inertia. Is that right?

My reaction is that we may want to build this into faiman as a kwarg-activated option, along with the extension in faiman_rad. Thoughts?

[williamhobbs]

If general commentary is helpful, I support this premise of this additional because 1) transient (dynamic) temperature modeling is important for sub-hourly modeling and 2) I think tools to fit temperature models to observations are valuable for monitoring in ongoing plant operation (e.g., to make sure back of module temperature measurements don't drift over time due to sensor detachment).

[adriesse]

This extension of faiman smooths the inputs, and prilliman smooths the output, both are intended to account for thermal inertia. Is that right?

Fundamentally correct. The main difference is that my focus is on extracting model parameters, and being able to simulate better is a nice added benefit.

My reaction is that we may want to build this into faiman as a kwarg-activated option, along with the extension in faiman_rad. Thoughts?

I see both faiman_rad and faiman_dyn as baby-steps toward more advanced modeling of PV operating temperature. As such their formulations (APIs) are also no more complex than they need to be, in the hope that more people will try them and like them. I have more comprehensive models that pull them together, but those are even further from being published.

Never-the-less, there may be some kwarg-activated option that fits the bill.

[williamhobbs]

Here is some more specific feedback:

• in the gallery example, I would add units to the x-axis label in the first plot (e.g., "Thermal inertia (minutes)").
• I like it! I ran an example on EPRI's SSRC data, where there was a known back of module temperature sensor issue right after commissioning. A fit Faiman model appears to match observations better than sapm_module, so I think could do a better job of detecting partial sensor detachment. A full gist is here https://gist.github.com/williamhobbs/7ae27e1d5202e58d526314cd53439643, and here's the main output:
  
  I think sapm_cell would have fit better than sapm_module in this case, but it isn't clear to me why.

edit: I added a boxplot and calcs to the gist. The fit version of faiman_dyn does agree better with observations than sapm_module after the sensor was fixed. SAPM+Prilliman has roughly the same IRQ, but 95% CI is better for faiman_dyn.

[adriesse]

Great to see it worked out of the box, @williamhobbs! Do the stats include night-time values?

[williamhobbs]

Do the stats include night-time values?

No, I filtered on POA > 200 W/m^2 (chosen arbitrarily). Nighttime matched regular Faiman and overestimated relative to measured:

[williamhobbs]

Edit to clarify my intention: It looks like tmod_sample_data_subset.csv is new for this PR. If a new measured timeseries dataset is going to be added to pvlib, perhaps it could be useful for demonstrating more than this set of functions. For example, if it included downwelling longwave, it could be used faiman_rad. If it included GHI/DNI/DHI in addition to POA, it could be used for demonstrating many irradiance models. I know of at least one dataset from EPRI that might be a good fit.

@adriesse, I just remembered that the EPRI SSRC dataset* has downwelling longwave/IR irradiance, which could be useful for testing faiman_rad. It looks like tmod_sample_data_subset.csv is new for this PR. Maybe the EPRI dataset would be more generally useful for testing/demonstrating temperature models? I could check with EPRI on license/copyright/etc. for including a subset of the data in pvlib, if that's of interest to the pvlib team. Alternatively, maybe there's data from Sandia/NREL/NIST/etc. that has everything as well.

*Southeastern Solar Research Center, Birmingham, AL, operated ~2014-2020, down-sampled dataset here https://dashboards.epri.com/ssrc.

You can get the EPRI SSRC data with something like:

# Import SSRC data, based on https://github.com/pvcaptest/pvcaptest/issues/89#issue-1816276179
import pandas as pd
import requests
from io import BytesIO

# (takes about 2 minutes with my internet connection)
# Import EPRI SSRC data. Data descriptions are at:
# https://github.com/epri-dev/PV-Subhourly-Clipping/blob/main/Data%20Channel%20Descriptions.csv
try:
    df_temp = pd.read_pickle('ssrc_all_df_temp.pkl')
except:
    urls = ['https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC01.zip', 
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC02.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC03.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC04.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC05.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC06.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC07.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC08.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC09.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC10.zip',]
    dfs = [pd.read_csv(BytesIO(requests.get(url).content),compression='zip', header=0, sep=',', quotechar='"', index_col=0) for url in urls]
    df_temp = dfs[0].join(dfs[1:])
    # Convert timestamp to datetime (takes about 1 minute)
    df_temp.index = pd.to_datetime(df_temp.index)
    df_temp.to_pickle('ssrc_all_df_temp.pkl')

# keep relevant columns
# SAT_Temp_PV_Module_10 is known bad in 2014, see https://github.com/pvlib/pvanalytics/issues/143
df_ssrc = df_temp[[
    'SAT_AC_Power', 
    'SAT_Irradiance_POA_South', 
    'SAT_Irradiance_POA_North',
    'SAT_Temp_PV_Module_5',
    'SAT_Temp_PV_Module_10', 
    'WS_Temp_Air_Outdoor_Amb', 
    'WS_Wind_Speed_Avg',
    'WS_Irradiance_Global_Horiz',
    'WS_Irradiance_NDLW_GR3Shaded']].copy()

df = df_ssrc[['SAT_Irradiance_POA_South',
             'WS_Wind_Speed_Avg',
             'WS_Temp_Air_Outdoor_Amb',
             'SAT_Temp_PV_Module_5',
             'SAT_Temp_PV_Module_10',
             'WS_Irradiance_NDLW_GR3Shaded']].copy()
df.rename(columns={'SAT_Irradiance_POA_South': 'poa_global',
                   'WS_Wind_Speed_Avg': 'wind_speed',
                   'WS_Temp_Air_Outdoor_Amb': 'temp_air',
                   'SAT_Temp_PV_Module_5': 'temp_pv',
                   'SAT_Temp_PV_Module_10': 'temp_pv2',
                   'WS_Irradiance_NDLW_GR3Shaded': 'ir_down'},
                   inplace=True)

[adriesse]

Do the stats include night-time values?

No, I filtered on POA > 200 W/m^2 (chosen arbitrarily)

Yes, I should have seen that in the code. From my more attentive re-reading of the code, it appears that 4-day averages (resampled, not rolling) are the basis of the statistics. That's perhaps a bit unusual and probably the reason for the relatively small deviations in all cases.

[adriesse]

Edit to clarify my intention: It looks like tmod_sample_data_subset.csv is new for this PR. If a new measured timeseries dataset is going to be added to pvlib, perhaps it could be useful for demonstrating more than this set of functions. For example, if it included downwelling longwave, it could be used faiman_rad. If it included GHI/DNI/DHI in addition to POA, it could be used for demonstrating many irradiance models. I know of at least one dataset from EPRI that might be a good fit.

@adriesse, I just remembered that the EPRI SSRC dataset* has downwelling longwave/IR irradiance, which could be useful for testing faiman_rad. It looks like tmod_sample_data_subset.csv is new for this PR. Maybe the EPRI dataset would be more generally useful for testing/demonstrating temperature models? I could check with EPRI on license/copyright/etc. for including a subset of the data in pvlib, if that's of interest to the pvlib team. Alternatively, maybe there's data from Sandia/NREL/NIST/etc. that has everything as well.

tmod_sample_data_subset.csv is a subset of the data found here: https://datahub.duramat.org/dataset/module-temperature
which does indeed include down-welling long-wave, and the notebook found at that location also demonstrates faiman_rad toward the end.

I think there is an objective to avoid large data files within pvlib in order to keep it small.

[adriesse]

Maybe the EPRI dataset would be more generally useful for testing/demonstrating temperature models?

In my opinion, yes! I didn't see an easy way of getting it through the dashboard though. Maybe you could clean up and prepackage something into an hdf or netcdf file?

[williamhobbs]

Maybe you could clean up and prepackage something into an hdf or netcdf file?

I've asked @dfregosi and team at EPRI to assign a license (e.g., CC-BY-4.0) to their dataset (https://github.com/epri-dev/PV-Subhourly-Clipping) so that there are no issues with using, modifying, and redistributing the data.

Would it be worthwhile to start a GitHub Discussion on what portion(s) of that dataset to include, ideal file format, etc.?

[adriesse]

Would it be worthwhile to start a GitHub Discussion on what portion(s) of that dataset to include, ideal file format, etc.?

In my opinion, yes. I vaguely recall previous discussions about data sets in pvlib.

[williamhobbs]

I just started that dataset discussion here #1889.

On the dataset currently included in this PR, I think some sort of citation of the DuraMAT DataHub source may be needed. See the "publication and non-endorsement" section here https://datahub.duramat.org/terms. Maybe it's already included and I just missed it.

[adriesse]

On the dataset currently included in this PR, I think some sort of citation of the DuraMAT DataHub source may be needed. See the "publication and non-endorsement" section here https://datahub.duramat.org/terms. Maybe it's already included and I just missed it.

This is a detail I'm not sure about given that I created the larger data set also (on behalf of and using data from Sandia). But I'm sure by the end of this code review it will be acknowledged as it should be!

[adriesse]

So now we can get the discussion back to the essence of this PR: faiman_dyn and fit_faiman_dyn!

[adriesse]

This PR is currently blocked. See #1898.