I've been bitten by the end-to-end testing bug! Now I envision a world in which every site has tests that tell stakeholders the moment something breaks. By the way, If you want to watch this information instead (plus lots more), watch Setting up a Comprehensive Automated Testing Regime for Drupal using Cypress and Playwright from the 2023 Stanford Web Camp.

The Value of End-to-End Testing

Why is E2E testing so valuable? It allows teams to find bugs early when the cost of fixing it in dollars and reputation is far less:

But E2E testing is just a part of a larger movement called Shift-Left. What's shifting? The commitment to building quality into a software project. And that usually means introducing testing earlier, like this:

The value of Shifting Left and earlier testing leads to:

• reduced costs
• faster feature rollouts
• higher quality and
• and ultimately a better user experience.

Running End-to-End Tests

Let's say you're convinced that end-to-end testing is right for your project. When would you run them? Here are the typical places:

• At the developer desktop during development (regression/functionality)
• When a merge request is created or before a commit (regression testing)
• During integration (regression testing)
• Before deployment (every sprint)
• After deployment (every sprint)
• Acceptance testing
• Continuous testing (nightly or even more often)

What about earlier testing, like unit testing? Definitely keep all your existing tests and if you don't have any yet, consider adding them in gradually. End-to-end testing is not a replacement for unit and integration tests—though with careful planning it can catch many of the same types of bugs.

Why Automated Testing Kit?

I started ATK because helping teams create a full suite of end-to-end tests is a great way to improve site quality. It will also improve team quality of life. It means fewer rollbacks, fewer hot patches, higher code quality and less stress.

Why Cypress and Playwright?

There is a new generation of testing tools available now that, in my experience, solve many of the issues of earlier tools like Selenium. Cypress was one of the first of this new generation tools. It was released in 2017 and showed how end-to-end testing could really be.

Playwright, which is forked from Puppeteer, solves the same issues as Cypress and a few issues Cypress has that make writing and debugging tests unnecessarily tricky (I'm looking at you, command queue and controlling multiple browsers). It also allows tests to be written in several languages and is significantly faster than Cypress. It is made and supported by Microsoft and we can expect it to expand its market share greatly in the future, especially since it supports more browsers than Puppeteer.

The browsers Cypress supports are Chrome, Chromium, Edge, Firefox and Electron. Playwright supports Chrome, Chromium, Edge, Firefox and Safari.

Cypress tests can be written in Javascript or Typescript while Playwright tests can be written in Javascript, Typescript, Python, C# and Java. Since both tools support Javascript, the Kit is written in Javascript to share code between the libraries.

What's In the Kit?

The initial release of the kit includes some basic tests and a series of useful utility functions. These early parts should be useful to all teams; see the documentation for more detail. Here are the categories of utility functions:

• logging in, logging out and registering new users
• user creation and deletion via Drush
• executing general Drush commands
• getting usernames, Drupal UIDs and 
• setting Drupal configuration.

The first batch of tests included are:

• registering a new user and testing the email at Ethereal.email by identifying a unique hash
• logging in via a login form or Drush user:login
• create a user via Drush
• delete a user via username, email or user id.

The way Drush is invoked can be easily modified through a configuration file, which makes it easy for Drush to operate locally, into a container (like, Lando, DDV or Docksal) or within a container. Targeting Pantheon is also built in. Note that running Cypress or Playwright inside a container requires extra setup. You'll have to install all the supporting files yourself or use an image. It's easy to install Playwright but you may want to use the Cypress image Tugboat provides.

Some future tests will include:

• exposing the Drupal dictionary
• confirming each entry in sitemap.xml
• triggering remote business logic
• object cache testing
• CKEditor testing
• and more.

Download the module at drupal.org/project/automated_testing_kit, follow the installation instructions and let me know what you think! And if you want support setting up a testing program for your project or help writing and maintaining the tests, check out the ways we help then reach out. We can tailor an engagement that works for you.