Testing Discussion

[Czarified]

I wanted to start a singular thread for all contributors to weigh in on our testing approach. I've been reading up on pytest, and remembered hypothesis(corrected, thank you Spectre!) as an additional library we could use. I apologize if none of this is really necessary, I just had thoughts and I wanted to write them down to try and help out. 😃

Firstly, it would benefit us to discuss timing. With #42 currently in progress (and looking good) it would likely mean a major version increment in sectionproperties. Do you want to wait until shapely is fully incorporated to begin developing tests? If we don't wait, do we progressively roll out testing, or lump it all in one release?

I'm completely inexperienced with software testing, so please weigh in if you know better! I don't even really know how a user could easily run our tests, after they're pushed with a release (or how to setup TravisCI to do it). However, I envision a two-fold approach to sectionproperties testing:

1. Functionality - The library needs to work for at least a majority of cases in which it could be used. If edge cases exist that cause unexpected behavior, they should be known. This is where hypothesis seems valuable, and from my reading this approach seems to be typically what people mean when they're talking about software testing.

2. Accuracy - Since this library performs engineering calculations, it should have some manner of proving its accuracy. Obviously each analyst who uses it is responsible for their own projects and calculations, but having a high level of confidence that the software can produce correct results, given the correct inputs, will be a boon to all users. This I see as setting up tests on known problems from academia, experimental results, and publications. It doesn't necessarily fall into an edge case that might break the code. We don't need to test and validate every step of the FE process, that would be frivolous, but comparing to examples in a textbook seems feasible (I don't have a copy of Pilkey, I'll have to pick one up eventually).

Full disclosure: I want to use this more at work and recruit others to start using this as well, so having that complete testing suite running really helps make the "sale" to tech fellows and managers. 😄

[Spectre5]

I'm glad you've started this as I've been thinking about starting a similar thread too.

I've read about hypothesis (corrected link) in the past and it does sound like a really interesting library, but I have no experience with it myself. I'd be interested in exploring it at some point, but I think it would be best to get a good set of tests written with pytest first.

Since #42 is such a major change, I think we wait until it is incorporated into master before adding many additional tests. That said, there are parts of the code that are not touched by it, so we could start sooner. And @connorferster is planning to add some tests for that new part of the code too.

For tests, you can always run tests locally as well. For example, pytest will read and work with the built-in python unittest framework, so you can try it right now. Install pytest into your section-properties virtual environment and simply run pytest from the top directory. It will discover the tests and run them locally.

I think we should have 3 types of tests:

1. Unit tests to target really specific parts of the code. A unit test is generally trying to test 1 function or method. An example would be testing specifically the geometric_properties method in the Tri6 class to make sure the area and other properties are correct for some known inputs. Similar for other functions. You want lots and lots of unit tests, which is fine as they also run really fast.
2. Full section tests. Getting a good set of cross sections with known results and computing everything for those sections to confirm the properties are correct: geometric, material, plastic properties and on at least some of them confirming stress fields (or some point stresses) are within an acceptably close result. There should be some simple ones like a rectangle and circle, as well as more complicated ones. There should be non-symmetric ones, single axis symmetric ones, and double symmetric ones, etc.
3. Timing/benchmark tests. I'm not sure exactly what the best way to incorporate this is (probably something only run locally for informational purposes but not run in the Travis CI). I'd like to get a set of cross sections to get good timings on. I know the library has a timing parameter for some methods (time_info), but ideally the test version would be something you can run and get a breakdown of all timings for some section with minimal wasted time elsewhere. There should be some quick ones, medium sized sections, and at least a few really big ones that take a bit to compute. Maybe some or all of these would be the same sections as from the second point above. These timing test can be used to ensure that changes are not introduced that significantly slow down the code, but also to be used to find opportunities to speed it up. I have a handful of ideas that will bring some nice speed improvements to the library, but I want to have a good set of benchmarks to work with before I spend time on it.

So 1 is checking for expected outputs of chunks of code, 2 is checking for accurate results of known sections, and 3 is benchmarking. Edge cases can and should be included in both 1 and 2 above. We won't know all of the edge cases up front, but as issues come in and get solved, new edge case tests should be added to ensure the fixes don't get reverted at some later point. #38 is a good example of this, we should add a test to ensure that this issue doesn't creep back up at some point. I wrote the code to solve that issue and I intended to write a few tests for it, but I haven't. This is partly because I wanted to see a more thorough testing framework (and the change to pytest) first.

[Spectre5]

Oh ya, and in terms of adding tests, I think we "roll out" tests as they are ready. No need to wait for all of them to be ready, that just introduces a barrier to getting more tests added sooner.

[Czarified]

@Spectre5 thanks for the reply. I think we're in general agreement. I'm probably going to make a new branch and try to learn hypothesis. It doesn't look much more complicated than pytest, so I have high hopes. I can start with small unit tests as you mention.

[Spectre5]

On a slightly separate note, is it possible to make the example in examples.rst just import the code from the examples directory so that it isn't repeated?

[Spectre5]

As for hypothesis, have you had much success? The example they use makes a lot of sense where they basically test to make sure that string = decode(encode(string)). But for most of the tests I'd see for this library, the resulting output depends on the input, so you need to know the input to know if the output is correct. We don't have an "undo" function like how decode undoes the encode. Maybe hypothesis has a nice way to handle this kind of situation that I'm just not aware of though. Of course it doesn't need to be used for all tests either, so maybe there is some subset of tests that it turns out to work nicely for.

[Czarified]

@Spectre5 I think this is what you're looking for regarding the sphinx docs.

And I've only got a couple basic tests. I was starting with my self-proclaimed "functional" style. For "accuracy" tests, yes, we'd need to know the inputs and the results. Here's what I've got so far. I had a couple issues setting up a hypothesis list strategy to test out the shift variable, and if you don't control the mesh size it will end up trying to mesh with a size of 0 or something very very small, locking up all your system resources.

import pytest
from hypothesis import given
from hypothesis import strategies as st
import sectionproperties.pre.sections as sections
from sectionproperties.analysis.cross_section import CrossSection


@given(
    d=st.floats(min_value=0),
    b=st.floats(min_value=0)
)
def test_rectangle_geom(d, b):
    '''
    This function tests the rectangular geometry.
    The pre module is currently undergoing an overhaul, so this is really just
    serving as the simplest point of entry for me to learn pytest/hypothesis.
    '''
    xsect = sections.RectangularSection(d, b)

    assert len(xsect.points) == 4
    assert len(xsect.facets) == 4
    assert len(xsect.control_points) == 1
    assert xsect.control_points[0][0] == b/2 
    assert xsect.control_points[0][1] == d/2



@given(
    d=st.floats(min_value=1, max_value=999),
    b=st.floats(min_value=1, max_value=999)
)
def test_rectangle_mesh(d, b):
    '''
    This function tests the rectangular mesh.
    The pre module is currently undergoing an overhaul, so this is really just
    serving as the simplest point of entry for me to learn pytest/hypothesis.
    '''
    xsect = sections.RectangularSection(d, b)
    xsect.create_mesh([20])

I plan on digging in a lot more today, and trying some more advanced tests.

[Spectre5]

Cool, these make sense. Looking forward to see what else you come up with!

As for the mesh test --- I'd probably consider this a good find on your part and a bug in the code. If zero is passed in (or perhaps a value below some tolerance/threshold value), then an exception should be raised, e.g. ValueError(f'mesh size of {size} must be greater than 0').

You can probably use a fixture in pytest or just a mock object to help control things like that for hypothesis though, to ensure acceptable values are used for certain tests.

[connorferster]

Hi all,

I think Hypothesis can be useful for some tests. I have been thinking about Hypothesis over the last year and a conversation I heard recently (I think on PythonBytes) made it clearer for me. As @Spectre5 was saying, for encoding/decoding it makes total sense, or for anything where the computation is difficult to perform but easy to check. For some particular sections (e.g. rectangle, circle, H/I sections, etc.), an exact analytic formula for perimeter, area, Ix, Iy, rx, ry, etc. is easy to write as a check function for Hypothesis's randomized input. These tests can provide some confidence on the level of "robustness" for the algorithms in sectionproperties.

Writing tests will be much easier with more people doing it! This is great!

I will be focusing on getting unit tests complete for the new Geometry and CompoundGeometry classes in addition to updating the docstrings for these. However, it would be great to get help on the all of the section generation functions (e.g. rectangular_section(...), circular_section(...), etc.) since that is more of a grunt-work effort.

[Spectre5]

Great point about using hypothesis for easily computes sections! That is a good idea.

[robbievanleeuwen]

Hi all,

Very keen on getting some more robust testing implemented! When I originally wrote sectionproperties I didn't even know testing existed so what currently exists is really just a bit of an afterthought.

In terms of formalising this do you think that continuing discussion in this issue is the best method? I was thinking maybe it's worth starting a discord channel for an easier place to gather some of this informal discussion which isn't really issue related but is more development chat related? It seems like we have a bit of a group of dedicated contributors now which is fantastic and maybe github isn't the best place for informal discussions? Let me know your thoughts on this!

Robbie

[Czarified]

I'm open to Discord, and I saw GitHub has a new "Discussions" feature specifically for this, if you wanted to keep it all in one place. I've never used it before, but supposedly you can turn it on in the repo settings?

As a counterpoint, just to thoroughly consider, is there significance to having this discussion in the issues for posterity and reference later on? If a new contributor joins and wants to specifically advance our testing, or switch to a new library/framework, they could search the issues and read through this thread. I think you can do the same on Discord, but it's just a bit more removed. Just a thought. I'm open to whatever the group thinks is best! 😄

[Spectre5]

I'm open to anything. I think "Discussions" on GitHub would be a good solution to try, although I've never personally used it. Maybe it would stay more easily searchable to address some of @Czarified's points though. I also like the idea of keeping it on GitHub so that even a new user or interested contributor could add some feedback or thoughts if they stumble upon the discussion, which seems pretty unlikely to happen if the discussions takes place on Discord.

[robbievanleeuwen]

Hi All,

I've enabled the discussions feature and it looks great, thanks for the recommendation!

I think that everything testing related is probably bigger than a single issue and may require significant coordination. Testing is probably more of an ongoing discussion rather than a single issue (which would probably never be 'closed')! I've referenced this thread in the new testing discussion thread #50 so I suggest we try that out and see if it works well.

Robbie

[Czarified]

I'm gonna close this, since we have that dedicated discussion thread. There's no specific issue with testing, right now.