Develop a common set of physical unit tests for the Python and MATLAB implementations.

[robwandrews]

Calama consulting is developing a set of notebooks using the NREL Regional Test Centre data which will compare the pvlib models to physical reality. Anyone who is interested in assisting with this, please get in touch

[bmu]

I am not sure about the use of these comparisons as "physical unit tests". The comparison of model output against physical reality (or at least measurements) is model validation. This is usually done by scientists and published in papers (maybe we are also scientists ;-), but the focus here is on programming). And this is not an easy task: e.g. for transposition models like Perez the uncertaitny of very good field measurements is about 2%, while I would expect deviations between model and measurement in the same order of magnitude for annual values. So the model uncertainty is difficult to access. However, there are a lot of publications about validation of the models we use here and we could reference them in the documentation.
These notebooks may be an interesting addition to our documentation but I think a comparison to reality is not the best for unit tests as I won't expect an exact matching with reality.

From my point of view for a library implementing such models it is more important to show, that the actual implementation of their models is correct (our implementation could be something like a reference implementation for these models). To show this, a comparison of the results of our models with the results of other implementations (e.g. pvlib matlab, pvsyst, meteonorm, SAM, ...)
is useful. This is difficult enough, I think, as I would expect some deviations between these implementations and we need to figure out, what is the truth.

[wholmgren]

You make a lot of good points and I think that we're actually all in agreement.

Physical unit tests may be a bad choice of words (and partially my fault). I think that @Calama-Consulting's notebooks will certainly be valuable documentation and will establish the validity of some of our library. Some of his code could be adapted for the test modules, but don't think that we'll want or need to do this for all of his work.

I think that the priority for the unit tests is making sure that things work as they should under different input parameter configurations.

Beyond that, I am mostly looking to be able to affirmatively answer the question: have you tested that pvlib python and pvlib matlab produce the same results? I would also like to be able to keep answering that question in automated way as we change the code, but there are multiple ways to establish this linkage.

[wholmgren]

Suggest that we make this a 0.3 or beyond milestone since it requires a lot of collaboration. I think I've seen something like a "someday" milestone on some projects.

[bmu]

I have created the milestone (stolen from pandas ;-).

[adriesse]

To share tests and results with Matlab the expected values would have to be taken out of source code and put into text files or something like that. Is that right? Is that likely to happen?

[cwhanse]

It is unlikely that we'll add tests to the Matlab library. It is not being actively developed.

[adriesse]

So, should the issue be closed?

[cwhanse]

So, should the issue be closed?

That would be my vote.