Sample data to include with pvlib

[williamhobbs]

@adriesse and I were discussing sample data to include with pvlib in comments here #1878. Anton created tmod_sample_data_subset.csv, a subset of Sandia SLTE data from https://datahub.duramat.org/dataset/module-temperature, for a gallery example .

I think a slightly larger dataset might make sense to distribute with pvlib, so that additional types of functions can be demonstrated in the example gallery. I was recently involved in a meeting with other plant owners, and someone mentioned that more "code templates" for pvlib would be helpful, and this might support that need.

tmod_sample_data_subset.csv currently includes poa_global, wind_speed, temp_air, and temp_pv, in 1min intervals for 7 days. If it was expanded to include DNI/DHI/GHI, it could be used for demonstrating irradiance models. It downwelling IR was added, it could be used for faiman_rad. And then dc and ac power could be used for examples of many other functions (as inputs or for comparison). This would roughly double the size of the file, going from 5 columns, including time, to 10-11 columns. I think all of this is available from the SLTE facility.

There's also EPRI data from the Southeastern Solar Research Center (SSRC) which was operated in Birmingham, AL, for ~6 years. It had 3 fixed-tilt arrays and one SAT array, all 2.4 kWdc/3.0 kWac, plus 2x POA and 2x back of module temperature sensors per array, GHI/DNI/DHI, downwelling IR, and 10m wind and ambient temperature. A dashboard is here https://dashboards.epri.com/ssrc, and minute-interval data are here https://github.com/epri-dev/PV-Subhourly-Clipping.

Then there's maybe NIST, RTCs, NREL arrays, and likely others.

Some considerations, at least for Sandia SLTE and EPRI SSRC:

1. EPRI SSRC used thermocouples for back of module temperature. There was at least one known sensor detachment, which is an interesting test case (good), but there's also the possibility that there were cold junction compensation issues baked into the data (bad).
2. EPRI SSRC includes multiple array types (good)
3. EPRI SSRC exact module specs/make/model have not been published (bad)
4. Sandia SLTE should have some row-to-row self-shade (good for shade examples, something like Best way to model self-shade loss in c-Si with pvlib? #1880 (comment) could make a good gallery example)
5. Sandia SLTE sensor maintenance was likely better than at the SSRC (good for SLTE).

@adriesse, please add your thoughts, especially if I've started this discussion in a different direction from what you had in mind.

[williamhobbs]

Thanks to @dfregosi, the EPRI SSRC dataset, https://github.com/epri-dev/PV-Subhourly-Clipping, now has a CC-BY-4.0 license, which should make it easy to add a subset into pvlib, if that's the direction the pvlib team wants to go.

[adriesse]

Great! Maybe they could stay where they are if they can somehow be read in from an example or test script?

[williamhobbs]

Something like this script will import the dataset, but it's downloading ~6 years of data across 10 zip files, so not very time efficient for small examples. But, a small subset of data could be packaged up and hosted somewhere (like DuraMAT DataHub, as you suggest below).

import pandas as pd
import requests
from io import BytesIO

# (takes about 2 minutes with my internet connection)
# Import EPRI SSRC data. Data descriptions are at:
# https://github.com/epri-dev/PV-Subhourly-Clipping/blob/main/Data%20Channel%20Descriptions.csv
try:
    df_temp = pd.read_pickle('ssrc_all_df_temp.pkl')
except:
    urls = ['https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC01.zip', 
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC02.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC03.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC04.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC05.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC06.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC07.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC08.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC09.zip',
            'https://github.com/epri-dev/PV-Subhourly-Clipping/raw/main/SSRC10.zip',]
    dfs = [pd.read_csv(BytesIO(requests.get(url).content),compression='zip', header=0, sep=',', quotechar='"', index_col=0) for url in urls]
    df_temp = dfs[0].join(dfs[1:])
    # Convert timestamp to datetime (takes about 1 minute)
    df_temp.index = pd.to_datetime(df_temp.index)
    df_temp.to_pickle('ssrc_all_df_temp.pkl')

[adriesse]

I had no particular direction in mind. I do think it is good not to overload pvlib with data files. Perhaps we could look at all the ones that are there and see if we could replace multiple narrow time series with one broader and more detailed one. Another idea would be to create a separate repository for validation data sets. Or put datasets up on Duramat in such a way that they can be read directly from examples and test scripts.

[williamhobbs]

see if we could replace multiple narrow time series with one broader and more detailed one.

That seems ideal to me. I think I remember Kevin once telling me that downloading external data to generate the Example Gallery contents was not idea (I think we were discussing use of PSM3?).