Unit Testing Django and Wagtail

Additional reading
- Documentation
- Inspiration
Setting up tests
Running tests
Common test patterns

Additional reading

Documentation

The Django testing documentation
The Wagtail testing documentation
Real Python's "Testing in Django"

Inspiration

The Django tests: it's always a good idea to take a look at how Django tests code that's similar to yours.
The Wagtail tests: it's also a good idea to see how Wagtail's built-in pages, blocks, etc, are tested.

Setting up tests

Choosing a `TestCase` base class

When writing unit tests for code in consumerfinance.gov there are multiple possible test case base classes that can be used.

unittest.TestCase: For testing Python code that does not interact with Django or Wagtail. This class provides all the base Python unit test assertions, such as:
django.test.SimpleTestCase: For testing Django or Wagtail code that does not interact with the database. This class provides all the assertions that unittest.TestCase provides as well as:
django.test.TestCase: For testing Django or Wagtail code that may need access to the database. This class provides all of the asserts that django.tests.SimpleTestCase provides as well as:
- assertQuerysetEqual(qs, values, transform=repr, ordered=True, msg=None)
- assertNumQueries(num, func, *args, **kwargs)[

Providing test data

There are a few different ways to provide data for your tests to operate on.

Using Django test fixtures to load specific data into the database.

Generally we use Django test fixtures when we need to test a fairly large amount of data with fixed values that matter to multiple tests. For example, when testing the interactive regulations search indexes.

class RegulationIndexTestCase(TestCase):
    fixtures = ['tree_limb.json']
    index = RegulationParagraphIndex()

    def test_index(self):
        self.assertEqual(self.index.get_model(), SectionParagraph)
        self.assertEqual(self.index.index_queryset().count(), 13)
        self.assertIs(
            self.index.index_queryset().first().section.subpart.version.draft,
            False
        )

The best way to create test fixtures is to add the objects manually and then use the Django manage.py dumpdata command to dump the objects to JSON.

Using Model Bakery to create test data automatically in code.

Generally use use Model Bakery when we need to test operations on a model whose values are unimportant to the outcome. Occasionlly we will pass specific values to Model Bakery when those values are important to the tests. An example is when testing operations around image models and the way they're handled, when we want to make sure that the model gets rendered correctly.

class TestMetaImage(TestCase):
    def setUp(self):
        self.preview_image = baker.prepare(CFGOVImage)
        self.social_sharing_image = baker.prepare(CFGOVImage)

    def test_meta_image_both(self):
        page = baker.prepare(
            AbstractFilterPage,
            social_sharing_image=self.social_sharing_image,
            preview_image=self.preview_image
        )
        self.assertEqual(page.meta_image, page.social_sharing_image)

Creating specific instances of models in code.

All other test data needs are generally accomplished by creating instances of models directly. For example, our Django-Flags DatabaseFlagsSource test case tests that a database-stored FlagState object is provided by the DatabaseFlagsSource class. To do that it creates a specific instance of FlagState using FlagStage.objects.create() that it expects to be returned:

class DatabaseFlagsSourceTestCase(TestCase):

    def test_get_flags(self):
        FlagState.objects.create(
            name='MY_FLAG',
            condition='boolean',
            value='False'
        )
        source = DatabaseFlagsSource()
        flags = source.get_flags()
        self.assertEqual(flags, {'MY_FLAG': [Condition('boolean', 'False'), ]})

Overriding settings

It is sometimes useful to modify Django settings when testing code that may behave differently with different settings. To do this there are two decorators that can be applied to either an entire TestCase class or to individual test methods:

@override_settings() will override the contents of a setting variable.

We use @override_settings any time the outcome of the code being tested depends on a settings variable.

This can include testing different values of feature flags:
```
@override_settings(FLAGS={'MYFLAG': [('boolean', True)]})
```
Or when we need to test behavior with AWS S3:
```
@override_settings(AWS_STORAGE_BUCKET_NAME='test.bucket')
```
@modify_settings() will modify specific values within a list setting.

Mocking

General Mocking with Mock

We use the Python Mock library when we need to. We recommend mocking for:

External service calls.

For example, we have a custom Wagtail admin view that allows users to flush the Akamai cache for specific URLs. One of the tests for that view mocks calls to the Akamai purge() method within the module being tested to ensure that it is called with the URL that needs to be purged:

from django.test import TestCase
import mock

class TestCDNManagementView(TestCase):

    @mock.patch('v1.models.akamai_backend.AkamaiBackend.purge')
    def test_submission_with_url(self, mock_purge):
        self.client.login(username='cdn', password='password')
        self.client.post(reverse('manage-cdn'),
                         {'url': 'https://fake.gov'})
        mock_purge.assert_called_with('https://fake.gov')

Logging introspection, to ensure that a message that should be logged does get logged.

For example, to mock a module-level logger initialzed with logger = logging.getLogger(__name__), we can patch the the logging.Logger.info method and make asserts based on its call arguments:
```
@patch('logging.Logger.info')
def test_message_gets_logged(self, mock_logger_info):
    function_that_logs()
    mock_logger_info.assert_called_with('A message that gets logged')
```

There are other potential uses of Mock, but generally we prefer to test our code operating on real objects as much as as possible rather than mocking.

Mocking Requests with Responses

For mocking HTTP calls that are made via the Requests library, we prefer the use of Responses. For example, to test whether an informative banner is displayed to users the ComplaintLandingView tests use responses to provide response data for calls to requests.get():

from django.test import TestCase

import responses

class ComplaintLandingViewTests(TestCase):

    @responses.activate
    def test_no_banner_when_data_invalid(self):
        data_json = {
            'wrong_key': 5
        }
        responses.add(responses.GET, self.test_url, json=data_json)
        response = ComplaintLandingView.as_view()(self.request)
        self.assertNotContains(response, 'show-')

Mocking boto with moto

When we need to mock AWS services that are called via boto we use the moto library. For example, to test our S3 utilities, we initialize moto in our test case's setUp method:

class S3UtilsTestCase(TestCase):
    def setUp(self):
        mock_s3 = moto.mock_s3()
        mock_s3.start()
        self.addCleanup(mock_s3.stop)

From there calls to boto's S3 API will use the moto mock S3.

Running tests

To run Django and Wagtail unit tests we prefer to use tox. tox creates and manages virtual environments for running tests against multiple versions of dependencies.

CFPB has a sample tox.ini that will test against Django 1.11 and 2.1 and Python 3.6. Additionally, running the tests for CFPB's consumerfinance.gov Django project are documented with that project.

Common test patterns

Django models

Any custom method or properties on Django models should be unit tested. We generally use django.test.TestCase as the base class because testing models is going to involve creating them in the test database.

For example, interactive regulations Part objects construct a full CFR title string from their fields:

class Part(models.Model):
    cfr_title_number = models.CharField(max_length=255)
    part_number = models.CharField(max_length=255)
    letter_code = models.CharField(max_length=10)

    @property
    def cfr_title(self):
        return "{} CFR Part {} (Regulation {})".format(
            self.cfr_title_number, self.part_number, self.letter_code
        )

This property can be tested against a Model Bakery-created model:

from django.test import TestCase
from regulations3k.models.django import Part

class RegModelTests(TestCase):
    def setUp(self):
        self.part = baker.make(Part)

    def test_part_cfr_title(self):
        self.assertEqual(
            self.part.cfr_title,
            "{} CFR Part {} (Regulation {})".format(
                part.cfr_title_number,
                part.part_number,
                part.letter_code
            )
        )

Django views

Testing Django views requires responding to a request object. Django provides multiple ways to provide such objects:

django.test.Client is a dummy web browser that can perform GET and POST requests on a URL and return the full HTTP response.

Client is useful for when you need to ensure the view is called appropriately from its URL pattern and whether it returns the correct HTTP headers and status codes in its response. All django.test.TestCase subclasses get a self.client object that is ready to make requests.

Note: Requests made with django.test.Client include all Django request handling, including middleware. See overriding settings if this is a problem.

The mortgage performance tests use a combination of fixtures and Model Bakery-created models to set up for testing the timeseries view's response code and response data:
```
from django.test import TestCase
from django.core.urlresolvers import reverse

class TimeseriesViewTests(django.test.TestCase):
    ...
    def test_metadata_request_bad_meta_name(self):
        response = self.client.get(
            reverse(
                'data_research_api_metadata',
                kwargs={'meta_name': 'xxx'})
            )
        self.assertEqual(response.status_code, 200)
        self.assertIn('No metadata object found.', response.content)
```
Note the use of django.core.urlresolvers.reverse and named URL patterns to look up the URL rather than hard-coding URLs directly in tests.

django.test.RequestFactory shares the django.test.Client API, but instead of performing the request it simply generates a request object that can then be passed to a view manually.

Using a RequestFactory generated request is useful when you wish to call the view function or class directly, without going through Django's URL dispatcher or any middleware.

The Data & Research conference registration handler tests use a RequestFactory to generate requests with various inputs, including how a GET parameter is handled:

from django.test import RequestFactory , TestCase

class TestConferenceRegistrationHandler(TestCase):
    def setUp(self):
        self.factory = RequestFactory()

    def test_request_with_query_string_marks_successful_submission(self):
        request = self.factory.get('/?success')
        handler = ConferenceRegistrationHandler(
            request=request,
        )
        response = handler.process(is_submitted=False)
        self.assertTrue(response['is_successful_submission'])

We generally do not recommend creating django.http.HttpRequest objects directly when testing views.

Wagtail pages

Wagtail pages are special kinds of Django models that form the basis of the Content Management System. They provide many opportunities to override default methods (like get_template()) which need testing just like Django models, but they also provide their own view via the serve() method, which makes them testable like Django views.

In general the same principle applies to Wagtail pages as to Django models: any custom method or properties should be unit tested.

For example, the careers JobListingPage overrides the default get_context() method to provide additional context when rendering the page.

from django.test import TestCase, RequestFactory
from jobmanager.models.django import City, Region, State
from jobmanager.models.pages import JobListingPage

class JobListingPageTestCase(TestCase):
    def setUp(self):
        self.factory = RequestFactory()

    def test_context_for_page_with_region_location(self):
        region = baker.make(Region)
        state = baker.make(State, region=region)
        city = City(name='Townsville', state=state)
        region.cities.add(city)
        page = baker.make(JobListingPage, location=region)

        test_context = page.get_context(self.factory.get('/'))

        self.assertEqual(len(test_context['states']), 1)
        self.assertEqual(test_context['states'][0], state.abbreviation)
        self.assertEqual(len(test_context['cities']), 1)
        self.assertIn(test_context['cities'][0].name, city.name)

Wagtail blocks

Wagtail StreamFields provide blocks that can be added to Wagtail pages. Blocks define a field schema and render those fields independent of the rest of the page.

As with Wagtail pages and Django models, any custom method or properties should be unit tested. Additionally, it may be important to test to the way a block renders. For example, the basic Wagtail DateTimeBlock unit test tests that the block renders the date value that's given:

from datetime import datetime
from django.test import SimpleTestCase
from wagtail.core import blocks

class TestDateTimeBlock(SimpleTestCase):
    def test_render_form_with_format(self):
        block = blocks.DateTimeBlock(format='%d.%m.%Y %H:%M')
        value = datetime(2015, 8, 13, 10, 0)
        result = block.render_form(value, prefix='datetimeblock')
        self.assertIn(
            '"format": "d.m.Y H:i"',
            result
        )
        self.assertInHTML(
            '<input id="datetimeblock" name="datetimeblock" placeholder="" type="text" value="13.08.2015 10:00" autocomplete="off" />',
            result
)

More complicated blocks' fields are stored as JSON on the page object. To prepare that JSON as the block's value to pass to the block's render() and other methods, the block.to_python() method is used.

For example, the TextIntroduction block requires its heading field when the eyebrow field is given. This is tested by checking if a ValidationError is raised in block.clean(value). To prepare a value to pass, to_python() is used:

from django.test import SimpleTestCase
from v1.atomic_elements.molecules import TextIntroduction

class TestTextIntroductionValidation(SimpleTestCase):
    def test_text_intro_with_eyebrow_but_no_heading_fails_validation(self):
        block = TextIntroduction()
        value = block.to_python({'eyebrow': 'Eyebrow'})

        with self.assertRaises(ValidationError):
            block.clean(value)

Wagtail admin customizations

It is occasionally useful to test custom Wagtail admin views and ModelAdmins for Django models.

When testing the admin Wagtail contains a useful mixin class for test cases, wagtail.tests.utils.WagtailTestUtils, which provides a login() method which will create a superuser and login for all self.client requests.

For example, this is used when testing the Wagtail-Flags admin views:

from django.test import TestCase
from wagtail.tests.utils import WagtailTestUtils

class TestWagtailFlagsViews(TestCase, WagtailTestUtils):
    def setUp(self):
        self.login()

The custom admin view that Wagtail-Flags provides can then be tested like any Django view:

    def test_flags_index(self):
        response = self.client.get('/admin/flags/')
        self.assertEqual(response.status_code, 200)

We generally recommend using WagtailTestUtils to login and test the admin unless specific user properties or permissions are needed for specific tests.

Django management commands

When testing custom management commands Django provides a call_command() function which will call the command direct the output into a StringIO object to be introspected.

For example, our custom makemessages command (that adds support for Jinja2 templates) is tested using call_command() (although it inspects the resulting .po file):

from django.core.management import call_command
from django.test import SimpleTestCase

class TestCustomMakeMessages(SimpleTestCase):
    def test_extraction_works_as_expected_including_jinja2_block(self):
        call_command('makemessages', locale=[self.LOCALE], verbosity=0)

        with open(self.PO_FILE, 'r') as f:
            contents = f.read()

        expected = '''...'''
        self.assertIn(expected, contents)

Feature-flagged code

We use Django-Flags to feature flag code that should be run under certain conditions, such as waiting for a particular launch window or to A/B test. Sometimes it is necessary to test that code is actually flagged.

In all cases, the easiest way to test with explicit flag states is to override the FLAGS setting and include only your specific flag with a boolean condition of True.

For example, to test a function that serves a URL via Wagtail depending on the state of a flag, the base URL tests check behavior with a flag explicitly enabled and again with that flag explicitly disabled:

class FlaggedWagtailOnlyViewTests(TestCase):

    @override_settings(FLAGS={'MY_TEST_FLAG': [('boolean', True)]})
    def test_flag_set_returns_view_that_calls_wagtail_serve_view(self):
        response = self.client.get('/')
        self.assertContains(
            response,
            'U.S. government agency that makes sure banks'
        )

    @override_settings(FLAGS={'MY_TEST_FLAG': [('boolean', False)]})
    def test_flag_not_set_returns_view_that_raises_http404(self):
        response = self.client.get('/')
        self.assertEqual(response.status_code, 404)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!