Tristan's Blog

At work, I inherited a legacy end to end testing setup based on wdio.js. It featured few tests, and wasn't run in CI, so it fell out of date pretty quickly. It also suffered from some of the usual challenges of using a non-integrated e2e testing tool, namely server setup/teardown and data fixtures. I decided to revamp the e2e test setup in order to solve these problems and encourage engineers to write more tests.

Playwright

Playwright ships with pytest integration, which made it an attractive candidate for testing our Django application with an existing pytest-based test suite. It's also a modern browser automation software, having learned from the painful experience of nigh-two decades of Selenium. Playwright features automatic waiting, a GUI-based test writing tool, a test inspector with play by play action, and more.

I strongly believe in using as few programming languages as possible within a project. This encourages and leverages language expertise for engineers, and reduces barriers to contribution for new team members. Writing tests in python, especially in pytest, meant that engineers could contribute to e2e tests with minimal friction. It also meant code reuse with our existing pytest fixtures.

With playwright in pytest, writing a test that opens a browser at a given page and runs assertions is this simple:

import re
from playwright.sync_api import Page, expect

def test_has_title(page: Page):
    page.goto("https://playwright.dev/")

    # Expect a title "to contain" a substring.
    expect(page).to_have_title(re.compile("Playwright"))

Amazing! Short, sweet and to the point.

For my e2e tests I want to open a browser at a local server. Certainly I could do this by opening a new terminal, running manage.py runserver, and testing with page.goto("http://localhost:8000/").. but that means I'm limited in a number of ways. One, state must be pre-configured in the server with some manage.py prepare_db_for_e2e script to add necessary data (user accounts, etc). And two, CI becomes more complex with server setup and teardown.

pytest-django to the rescue! The library ships with a live_server fixture, which "runs a live Django server in a background thread." In practice this means an e2e test running against the django server can be as simple as:

from django.urls import reverse
from playwright.sync_api import Page, expect

def test_load_home_page(page: Page, live_server):
    # Note navigating to live_server.url 
    page.goto(live_server.url + reverse("home"))

    expect(page).to_have_title("My Great Website")

Magnificent: no separate server to stand up/tear down, it's all handled automatically. Now, running my e2e tests is as simple as running pytest.

Test data can be loaded through the same mechanisms as other pytest tests, supporting strong code reuse. For example, at work I use a mix of django json fixtures and factory boy - all re-usable in this testing approach. Need to test viewing an order in a given state? Create the order with a factory and navigate to its url. Easy peasy.

Seriously. This setup rocks.

Challenges

Still, no software solution is ever perfect. This setup has a few challenges that are worth mentioning.

Shared global data fixtures

First, the live_server fixture running the server in a separate thread means that per-test db transaction rollbacks are not possible. In practice this means that after each test using the fixture, every database table is truncated. If your tests depend on global data fixtures (e.g. provided in an django_db_setup override), then you'll run into problems. To resolve this, I introduced a new command line flag to pytest to separate e2e from non-e2e test runs:

# in global conftest.py

def pytest_addoption(parser):
    parser.addoption(
        "--e2e",
        action="store_true",
        default=False,
        help="Enable when running e2e tests",
    )

def determine_django_db_setup_scope(fixture_name, config):
    """
    e2e tests use the live_server fixture, which will wipe
    the db after each test run, which conflicts with a 
    session scoped db setup.

    therefore, we use a function scoped db setup for e2e tests.
    """
    if config.getoption("--e2e"):
        return "function"

    return "session"

@pytest.fixture(scope=determine_django_db_setup_scope)
def django_db_setup(django_db_setup, django_db_blocker):
    ...

I also separated e2e tests into their own top-level test folder, excluded by default in pytest.ini. In practice this means running the full test suite requires two commands:

# run e2e tests
$ pytest --e2e tests_e2e/
# run standard unit tests
$ pytest

Ideally, this test classification would be unnecessary, but it's a minor setback.

Static assets

Another gotcha is configuring the static assets path, so that js, css and other static content is loaded from the test server correctly. To resolve this, I created a custom live server fixture:

@pytest.fixture
def my_live_server(live_server, settings):
    """
    Customized live_server fixture that configures STATIC_URL to point to the live server URL.
    """
    settings.STATIC_URL = live_server.url + "/assets/"

    yield live_server

And then replaced the test parameters with my_live_server.

Test teardown synchronization

The final issue that I've experienced in practice is a non-obvious test failure:

Exception Database test_xxxxxxxx couldn't be flushed. Possible reasons:
  * The database isn't running or isn't configured correctly.
  * At least one of the expected database tables doesn't exist.
  * The SQL was invalid.
Hint: Look at the output of 'django-admin sqlflush'. That's the SQL this command wasn't able to run.
The full error: deadlock detected
DETAIL:  Process 37253 waits for AccessExclusiveLock on relation 131803 of database 131722; blocked by process 37250.
Process 37250 waits for AccessShareLock on relation 132659 of database 131722; blocked by process 37253.
HINT:  See server log for query details.

It was discovered this is due to asynchronous requests in the browser not finishing before the test ended. For example, consider a React-based page that fetches data asynchronously on load. If the playwright test merely checks the page's title before ending, then the test may end before the asynchronous data has been fetched.

To resolve this issue, I typically add explicit waiting on the relevant network requests:

with page.expect_response(my_live_server.url + reverse("my_api")):
    ...

Ideally, pending network requests could be handled more gracefully.

Takeaways

The challenges with this approach are greatly outweighed by the benefits. I have yet to find an e2e test setup that doesn't have issues. Writing tests in the same framework (pytest) and with the same tooling (data fixtures, mocks and so on) is a major boon compared to a different language, framework and tooling.

I'll close out by saying that, almost a year after introducing playwright at work, few e2e tests have been written! It goes to show something very important about tests: the real challenge is team culture; no matter how easy it is to write tests there has to be buy-in from leadership and team for the importance and value of tests.