Testing GCBV doesn't work as you'd expect

Following a couple of days (14th/15th June 2017) of investigation, head bashing, and discussion with the OxPy community, here's the abbreviated notes of what I've discovered about trying to do automated view testing of templates for GCBV based functions

From Slack thread:

I have a Django ListView that I am trying to test is loading in a really basic way (clean site, no data, admin user logged in), and to that end I’m checking that in my TestCase, .assertTemplateUsed(template_path) should be fine… but it isn’t Instead I get a failure of…

======================================================================
FAIL: test_adverse_event_view_basic (wp4.tests.test_views.CompareViewsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/Users/carl/Projects/py3_cope/cope_repo/wp4/tests/test_views.py", line 141, in test_adverse_event_view_basic
   ae_views.AdverseEventListView.as_view()
 File "/Users/carl/Projects/py3_cope/cope_repo/wp4/tests/test_views.py", line 33, in test_pattern
   this.assertEqual(response.status_code, 200)
 File "/Users/carl/.virtualenvs/py3_cope/lib/python3.5/site-packages/django/test/testcases.py", line 129, in __exit__
   self.test_case.fail(message)
AssertionError: adverse_event/event_list.html was not rendered. No template was rendered.

Now, I can manually build a clean database, loading the same fixtures and getting to the same default state, and when I click on the url requested - boom, I have the page as expected, which includes rendering the list view template

The test itself works fine for all the other views in the project so far (apart from another one with the same failure), and looks like this…

def test_pattern(this, url, template, view):
    """
    Test that a given url and view returns a 200 response and uses the expected template
    """
    request = this.factory.get(url)
    request.user = this.user  # Fake being logged in
    with this.assertTemplateUsed(template):
        response = view(request)
        this.assertEqual(response.status_code, 200)

and

    def test_adverse_event_view_basic(self):
        """
        Test urls from third level of lang/wp4/adverse-event
        """
        test_pattern(
            self,
            BASE_URL_WP4 + '/adverse-event/',
            'adverse_event/event_list.html',
            ae_views.AdverseEventListView.as_view()
        )

That latter function is inside the CompareViewsTestCase(CoreDataMixin, TestCase): class, whereas the former is outside

Looking through the assertTemplateUsed() suggests that I’m getting the matching template path being parsed into template_name, but template_names (which is the test that produces the No template was rendered. error is set to None… and I’m beginning to think this is a Django bug now

So… anyone had any similar experience using TestCase with CBVs? 🙂

ghickman [12:41 PM] Not exactly an answer but… why are you testing Django functionality?

Carl Marshall [12:42 PM] As in why am I checking that the template I expect to be loaded is being used, and then that the page returns a 200 status?

[12:42] Or are you reading this differently to how I’m understanding what I think I’m doing 😉

[12:43] The caveat here is that I’m attempting (bad I know) to belatedly add some basic tests to an 18 month old application, to ensure that pages are not returning 500 or 404 errors after changes.

ghickman [12:46 PM] You're testing a template was rendered on a CBV, so I'm assuming you're doing template_name = 'foo'? 1 reply Today at 12:49 PM View thread

ghickman [12:47 PM] Because the actual rendering is Django functionality to me

12 replies Carl Marshall [19 minutes ago] ghickman: I think I’ve reparsed what you’re saying, and you’re wondering why am I testing what should be an automatic Django mechanism for choosing the template? Basically because that was the pattern I’ve been using for all the view tests (as picked up from one of the TDD books), but also because whilst I have a mix of function-based and class-based views, I also have some cases where the template is overridden based on $factors$, so I was hoping to use this test pattern to pick up on that later.

I can’t find any examples of using this template assertion with CBV though, and several places where they suggest not testing CBV stuff too closely 😕

ghickman [17 minutes ago] I would agree with not testing GCBV stuff too closely (note the G there for Generic)

Carl Marshall [16 minutes ago] Any idea why? Is it a case of there be dragons like this?

Carl Marshall [15 minutes ago] (also realising I don’t have a good understanding of the difference between GCBV and CBV)

ghickman [14 minutes ago] GCBV: the generic views & mixins Django ships (documented at http://ccbv.co.uk ccbv.co.uk Django Class-Based-View Inspector -- Classy CBV The best way to understand Django's class-based views is to see it in Classy CBV, so pick your version and jump in at the deep end.

ghickman [14 minutes ago] CBV: a type of view, includes GCBVs

Carl Marshall [14 minutes ago] Am fond of that website 🙂

Carl Marshall [13 minutes ago] Ah cool, that makes more sense… noun versus description

ghickman [13 minutes ago] The reason not to test them is they are brittle in many ways. Many of the views have 11 parent classes/mixins and they all plug together nicely with the config you specify

ghickman [13 minutes ago] But testing them requires brittle tests too and why test what Django is already testing?

ghickman [12 minutes ago] I see what you're trying to do though, and I think if it were me I'd try to test the final template, i.e. Post rendering

Carl Marshall [11 minutes ago] Can appreciate that logic, though… (and jumps back to main thread)

ghickman [12:47 PM] (The status code sounds like a good idea btw, that's not what I was asking about 🙂 )

Carl Marshall [12:48 PM] Some extra info… I’ve tried adding some code in test_adverse_event_view_basic before the call to test pattern…

    request = self.factory.get(BASE_URL_WP4 + '/adverse-event/')
        request.user = self.user  # Fake being logged in
        response = ae_views.AdverseEventListView.as_view()(request)
        print("test_adverse_event_view_basic: {0} - {1} - {2}".format(
            response.template_name, response.is_rendered, response.rendered_content))

The output of which gave me a raw html of the rendered page as expected, but an unexpected answer to is_rendered….

test_adverse_event_view_basic: ['adverse_event/event_list.html'] - False - <!DOCTYPE html>

<html lang="en">
<meta http-equiv="content-type" content="text/html;charset=utf-8"/>
<head>...

My testing of the testing says this should pass… the result says it doesn’t… I can’t find a credible reason for why the TestCase logic of assertTemplateUsed which calls lib/python3.5/site-packages/django/test/testcases.py/SimpleTestCase:_assert_template_used does what it does… which is to say that if it can’t find contents in template_names, but a template_name has been supplied, then return template_name, None, and a message, which is then interpreted by _AssertTemplateUsedContext to fail because it tests…

    def test(self):
        return self.template_name in self.rendered_template_names

Clearly nothing will be found in None :-s

(this is all python 3.6 and django 1.11.2 btw)…

So unless wiser heads can correct me, I think I’ve found a logic bug in Django’s SimpleTestCase

I soooo don’t want to have to spend time writing a clean app and test setup to prove this again so I can submit it as a bug with all the evidence 😞

I’d settle for someone telling me I’m not going mad or crazy with this logic too 😉

Ok, the test point isn’t quite what I thought it was… _AssertTemplateUsedContext:test() references self.rendered_template_names which is initialised as []. The clue came when I thought to try the anti-pattern and ran assertTemplateNotUsed - which passed. That assertion test is an extension of class _AssertTemplateUsedContext, and thus caused me to note that it is rendered_template_names, and not just template_names… so now for for digging and testing of the test code…

The clue seems to be the is_rendered property of the ResponseTemplate class being generated in the test case… my previous poking shows it not to be rendered before the assertion is made (even when the output is clearly in the rendered_content at the same time!)… which means that self.rendered_template_names isn’t being populated by the time the test is performed…

So now I have to work out why a rendered template is being marked as not rendered??

Or I could get lunch…

ghickman [2:36 PM] I would agree with not testing GCBV stuff too closely (note the G there for Generic)

Carl Marshall [2:40 PM] I’m realising that this template test only works with my function based views, and not with any of the GCBV ones - which to my mind says something is fundamentally wrong with the test logic

ghickman [2:40 PM] But testing them requires brittle tests too and why test what Django is already testing?

Carl Marshall [2:42 PM] Since I’ve been chasing this like a squirrel with a nut for the past day or so, I think I’m going to have to relent to the wisdom here and determine a different test for my GCBV calls. Maybe sans template test for now, and figure out the specific cases I need to check for that later.