Testing

Introduction#

Testing is how a team builds confidence that each feature works as intended, keeps working as the system changes around it, and meets the needs of its users, including the non-functional ones such as speed, security, and accessibility.

The approach described here is continuous, mostly automated, and shared across the team. Testing happens as part of implementation, rather than in a phase of its own, and most of the automated tests are written by the developers who write the code. Specialist testers remain essential, but their effort shifts from executing tests to shaping quality: building it into the work before it starts, guiding how it is tested, and validating that the automation covers what it needs to.

This page explains the aim of testing, how it fits into the workflow, who does what and why, exploratory testing, the test pyramid, when tests should run, non-functional testing, and the practicalities of test data and test environments.

Prevention is better than cure#

A bug found in testing is a success of the safety net, but a failure of everything that came before it.

The aim of testing is not to find bugs, but to prevent them from affecting users, and the most efficient way to do that is to avoid them being created in the first place.

Finding a bug means an earlier opportunity to stop it was missed. Perhaps an edge case was never discussed, an AC was ambiguous, or a test that should have been written alongside the code wasn’t. Every stage of the workflow is a chance to prevent defects, and the earlier the chance is taken, the cheaper it is.

In refinement, a misunderstanding or missing point is quick to discuss and document.
During implementation, issues can be found and fixed in a few minutes, based on feedback from unit or other automated tests.
During exploratory testing, test gaps are found so they can be addressed before the developer moves on.
In a separate test stage, or later in UAT, issues can only be fixed by a developer switching context, and fixes take hours or more.
In production, fixes incur a full release on top of any fix work, plus the impact on users.

If there is a separate test stage, items that bounce back to development inflate cycle time and work in progress.

When testing focuses on prevention, developers and testers work together towards the same goal rather than one inspecting the other’s output. This is why testers are most valuable in refinement, why developers should write tests as they write code, and why manual testing should be used to check the safety net rather than to be it.

Testing as part of implementation#

Jonathan Smart ↗

Quality should be built in, rather than inspected in later.

Testing should happen in the Implementation stage, alongside writing the code, not in a separate stage afterwards. A separate test stage creates a handover, a queue, and a mentality that quality is somebody else’s job.

Ideally, implementation consists of many small commits, each containing a little production code and the tests that prove it. Testing is not something that happens once the code is finished; it is part of implementing the feature correctly.

Because testing is part of the work, an item isn’t done until it is tested. Cycle time should ideally be one to two days, including all the testing. This is only achievable with items that are sliced thinly. Story point estimates cover everything needed to finish the item, including testing. Planning poker includes testers, and their view of the testing effort often reveals complexity others had missed.

In most cases, tests should be written as part of the item they test, not deferred to separate “testing” items. Testing items tend to be deprioritised, are hard to value on their own, and let the original item appear done when it isn’t. Unit and API tests should always be implemented as part of the item.

But there are two cases where a separate test item is justified:

UI automation for very thin slices. When items are sliced very finely, a single slice may not deliver enough of a user journey to be worth testing through the UI. It can make sense to wait until a few slices together form a coherent journey and then add the UI automation, either as part of the slice that completes the journey or as a separate item. The individual slices are still covered by unit and API tests in the meantime.
Some non-functional testing. Load, soak, and resilience testing often need enough of the system in place to be meaningful, and that point rarely maps one to one onto backlog items. A dedicated “load test the checkout journey” item is sensible here. The non-functional ACs still belong on the items that introduce the behaviour, as described in Non-functional ACs. What is separate is the exercise of testing several items together under realistic conditions.

Avoid	Better
A “Dev done” column followed by a “Test” column, with items queueing between them.	A single Implementation stage where code and tests are written and reviewed together.
“Write tests for the address form” as its own backlog item.	Tests as part of the definition of done for every item.
Estimating the development effort and adding “testing time” afterwards.	Estimating the whole item, testing included, with testers in the room.

Automate by default#

Most testing should be automated, at unit, integration, API, and UI levels.

Automated tests should be run regularly and automatically. For example, a subset of tests may run on every pull request for quick feedback, with a fuller set running periodically, such as nightly (see Run tests at the right time). A robust set of automated tests makes frequent deployments safe, because every change is checked against everything that has been built so far, not just the part that changed.

Manual regression testing doesn’t scale. The product grows with every item delivered, so the regression pack grows too, and the time to run it by hand grows accordingly. Teams that rely on manual regression face a choice between slowing their release cadence and testing less thoroughly, and under deadline pressure they tend to choose the latter. Manual checks are also error-prone: repeating the same set of complicated steps in the same way many times over is exactly the kind of task that machines are better at than people.

Manual testing still has an important role, but it should be for exploration rather than regression.

Developers write the tests#

Developers write the unit, API, and UI automation tests for the items they implement, as part of implementing them. A specialist tester may guide, advise, or contribute to writing these tests, but the bulk of automated tests should be implemented by developers.

Developers also test their own work before asking a tester for input. By the time a tester looks at an item, the developer should be confident that it meets its ACs, and the tester’s time should go on what the developer couldn’t reasonably have caught.

There are several reasons for developers to write the tests:

It removes the test bottleneck. Developers typically outnumber testers. When testers are responsible for writing and running the tests, work piles up waiting for them. Developers don’t like to sit idle, so they start new items. Work in progress grows, and when bugs come back days later, the developer, who has since moved on, must switch context to deal with them. When a deadline looms, testing is the part that gets squeezed, because it’s the part at the end. Developers writing the tests means testing capacity grows with development capacity, and the queue disappears.
It builds a quality mindset and ownership. When someone else is responsible for finding the problems, it is natural to treat “done” as “handed over.” Writing the tests makes developers responsible for proving their own work is correct. This ends the “throw it over the wall” mentality, and quality becomes part of the craft of software development rather than a separate activity performed by other people.
It produces more maintainable code. Code that its author has to test tends to be written to be testable: smaller units, clearer interfaces, and fewer hidden dependencies. These are the same qualities that make code easy to maintain.
It gives the fastest possible feedback. A test written alongside the code catches a problem within minutes, while the developer still has the context fresh. A test run days later catches it after that context has gone.
The tests live with the code. They are written, reviewed, and merged in the same pull request, so they stay in step with the code, and a reviewer can check the tests prove the ACs at the same time as checking the code.

Exploratory testing#

With most testing automated, manual testing has a different and more focused purpose: verifying whether the automated tests cover everything they need to.

Exploratory testing is targeted. The tester draws on the acceptance criteria, what the automated tests cover, their knowledge of the system, and their sense of where risk lies, and probes the item for behaviour the automation doesn’t cover. This is not a scripted regression run; it is skilled, directed investigation, and it is where the tester mindset is applied most directly to working software.

Exploratory testing happens as the item takes shape during implementation, ideally as soon as there is something to explore, and is confirmed complete as part of UAT. Its findings feed back in one of three ways:

A gap in the automation: add the missing test, at the cheapest level that can prove the behaviour.
A defect in this item: fix it as part of the item, with a test that proves the fix.
Something outside the item’s scope: capture it as a new backlog item rather than letting the current one grow.

Exploratory testing is also a useful source of learning for developers. Pairing on it now and then helps developers absorb the tester’s way of thinking, which improves the tests they write.

The tester mindset#

Though developers may write most of the tests, specialist testers still play an essential role.

Developers naturally think in solution mode. Their job is to make something work, so their attention goes to how it should work: the route through the code that produces the right outcome. This is exactly the right mindset for building things, but it favours the happy path.

Testers think in problem mode. They consider all the ways something could fail, be misused, or surprise a user: the empty list, the second click before the first has finished, the network dropping mid-save, the customer with an apostrophe in their surname, the user who navigates backwards. This critical thinking is a distinct skill and discipline, developed through practice, and good testers apply it instinctively.

There is also a more fundamental problem with anyone testing only their own work. Confirmation bias means we tend to test what we expect to work, and the gaps in our own thinking are, by definition, invisible to us. A developer who didn’t think of an edge case while writing the code is unlikely to think of it while writing the tests.

Developers writing most of the tests doesn’t make testers less important, but it does change where their effort goes, towards the places where it has most leverage:

Refinement. Asking “how could this fail?” while the item is still being defined, so edge cases become acceptance criteria and bugs are prevented. This is the single most valuable use of a tester’s time. See Quality starts in refinement.
Test design. Guiding developers on what to test and at which level, reviewing test coverage in pull requests, and pairing with developers on tricky areas.
Exploratory testing. Targeted probing to validate that the automation covers what it needs to. See Exploratory testing.
Curation. Keeping the automated test packs lean, fast, and trustworthy. See Curating the test packs.
Leading practice. Setting the standard for good testing in the team, and identifying and driving adoption of new tools and techniques.

In this model, the tester becomes a multiplier. Instead of personally checking a fraction of the team’s output, they raise the quality of everything every developer produces.

Tester as gatekeeper	Tester as quality coach
Receives finished items and checks them.	Shapes items before they are started so problems don’t arise.
Writes and runs most of the tests personally.	Guides and reviews the tests developers write.
Is the bottleneck whenever development speeds up.	Scales with the team, because the testing effort is shared.
Measured by bugs found.	Measured by bugs that never happened, and by the quality habits of the team.
Developers see quality as the tester’s job.	Developers see quality as their own job, with an expert to help.

The Product Owner’s role#

Testing is a shared responsibility, and the Product Owner has a role in it too. (As elsewhere in the handbook, Product Owner stands for this and similar roles, such as Product Manager.)

The Product Owner contributes to the ACs during refinement, gives early feedback during implementation when there is something to look at, and makes the final call in UAT that the item can go live as it is. Their perspective is different again from both developers and testers: not “does it work?” or “how could it fail?”, but “is this what our users actually need?”

Common concerns#

The approach described above is not universally shared, and several concerns are commonly raised about it. We address each in turn:

“Developers shouldn’t mark their own homework.” They don’t do it alone. Testers shape what gets tested by contributing to the ACs, review the test coverage, and validate it with exploratory testing. Peer review provides a second pair of eyes on the tests as well as the code. The developer writes the tests, but the whole team contributes to what they need to prove.
“Developers don’t have the testing skills.” Many don’t yet, which is exactly why the tester’s coaching role matters. Testing skills develop quickly with pairing and review, and spreading them across the team is far more resilient than concentrating them in one person. A team where every developer thinks a little like a tester is better than a team where only the tester does.
“Developers writing tests slows delivery down.” It makes each item slightly slower to code but makes the flow of work sustainably faster, by removing the queue, the handovers, and the rework. Because story points include the testing effort, it is visible and planned rather than a hidden tax. Teams consistently find that the time they “lose” writing tests is repaid many times over in the bugs they don’t have to fix later.
“If developers write the automation, we don’t need testers.” This confuses writing tests with instilling quality. Teams that remove testers on this basis typically end up with test suites that confirm the happy path very efficiently, while bugs in the edge cases reach production. The automation is only as good as the thinking behind it, and critical thinking about failure is the tester’s core skill.
“Our testers can’t code.” They don’t need to write most of the automation. Their greatest value lies in the thinking, which applies in refinement, test design, and exploration regardless of coding skill. Coding skills are useful, particularly for reviewing tests and maintaining the test packs, and are worth developing, but they are not the core of the role.

The cheapest place to prevent a defect is before any code has been written, which makes refinement the most valuable tester activity of all.

Acceptance criteria are drafted in a Three Amigos ↗ conversation involving product, development, and test perspectives, as described in Acceptance criteria: Who and when. The tester’s contribution here is what stops the ACs being a restatement of the happy path. Each “what happens if…?” question either becomes a new AC, is explicitly ruled out of scope, or prompts the item to be split.

ACs in given, when, then form translate directly into tests. The given is the test setup, the when is the action, and the then is the assertion. Working through the ACs and asking “what proves this, and at what level?” produces the test approach, as described in From ACs to tests.

Time spent on good refinement is a sound investment. It feels slower in the moment, but it repays itself in fewer bugs, less rework, and items that move steadily across the board instead of bouncing back and forth.

The test pyramid#

The test pyramid describes the balance of automated tests across levels: many fast, focused unit tests at the base; fewer integration and API tests in the middle; and a small number of UI tests at the top.

Unit tests exercise a single piece of logic in isolation. They run in milliseconds, so there can be thousands of them, and when one fails it points at exactly one thing.
Integration tests at the code level verify that individual units combine as expected. These look like unit tests but with the subject under test integrated with some real internal dependencies rather than having every dependency mocked.
API tests exercise the system through its interfaces, including the database and other real components. They are slower than unit tests but prove that the pieces work together and that the behaviour visible to API consumers is correct.
UI tests exercise the system through the user interface, as a user would. They are the slowest and most brittle, and a failure can have many causes, so they should be reserved for proving that complete journeys hang together.

The aim is minimal overlap between levels. Each level should prove what the levels below it cannot, and no more. Every behaviour is tested once, at the cheapest level that can prove it.

Example: testing an alternative delivery address

Taking the alternative delivery address item used in definition of done and acceptance criteria, the levels of the test pyramid may look like this:

Level	What it proves	What it deliberately doesn’t
Unit	Postcode normalisation and validation: every valid format, lower case, missing or extra spaces, too short, too long, invalid characters. Dozens of cases, each running in milliseconds.	Doesn’t touch the database, the API, or the UI.
Integration	Verifying an address validates every part, with the postcode validator running alongside the validators for the other parts.	Doesn’t check all the permutations of invalid postcodes.
API	An address can be saved and retrieved. An invalid address returns a validation error, using one representative case. One customer cannot retrieve another customer’s address.	Doesn’t re-test every invalid address format, which the unit tests already cover. Doesn’t check anything about layout.
UI	One journey: add an address, select it at checkout, and see it on the order summary.	Doesn’t test validation rules or authorisation, which are covered below it. Just proves the pieces are wired together.

If the UI test fails but the API and unit tests pass, the problem is almost certainly in the UI or the wiring between UI and API, which narrows the search considerably.

Overlap has a cost. Every duplicated test is one more thing to run, maintain, and update when the behaviour changes. A suite where the same validation rule is tested at all four levels takes longer to run, fails in four places at once, and needs changing in four places when the rule changes.

The common anti-pattern is the ice cream cone ↗: an inverted pyramid with few unit tests, some API tests, a large suite of UI tests, and a big manual regression effort on top. It is slow, fragile, expensive to maintain, and produces failures that are harder to trace back to their causes. It usually arises when testing is done after development by people who can only reach the system through its UI, which is one more reason for developers to write the tests.

Run tests at the right time#

Automated tests should run as part of the build and deployment pipeline, and fast enough that the feedback arrives while it is still useful and nobody is tempted to skip them.

Not every test needs to run on every change. A sensible cadence is:

When	What	Why
Every pull request	Unit tests, API tests, a small set of critical UI tests, linting, static analysis, and dependency scanning.	Fast feedback on every change before it is merged. This should complete in minutes, ideally under ten.
After every deployment	Smoke tests against the environment just deployed to.	Confirms the deployment worked and the environment is healthy before anyone relies on it.
Nightly	The full UI suite, longer-running security scans, and performance baselines.	Broader coverage that is too slow for every change, while still catching problems within a day.
As needed	Load, soak, and resilience tests.	Expensive to run and need a representative environment, so they are run before significant releases or when relevant changes are made.

The exact split will depend on the team’s context, and the right answer changes as the suites grow. What matters is that each test runs as early as it can while still being fast enough for its trigger, and that the whole arrangement is automated.

Flaky tests must be addressed. A test that sometimes fails for no real reason teaches the team to ignore failures, which is worse than having no test at all, because it creates false confidence. Fix flaky tests promptly, or quarantine them until they can be fixed, but don’t tolerate them.

Curating the test packs#

Test suites tend to grow over time. Implementing every backlog item adds tests, and without deliberate attention the suites gradually become slower, more duplicated, and more burdensome to maintain.

Testers curate the automated test packs to keep them appropriately lean, working with developers to:

Remove duplication across levels, in line with the pyramid’s principle of minimal overlap.
Push tests down the pyramid where a cheaper level can prove the same thing.
Retire tests that no longer earn their keep, such as those for behaviour that has since changed or been removed.
Identify and fix slow and flaky tests.
Keep the split between pull request, nightly, and post-deployment runs sensible as the suites grow.

This is ongoing housekeeping, not a one-off exercise. Developers make many of the changes, but someone needs to be watching the shape of the whole, and that is a natural part of the tester’s role as the team’s quality lead.

Non-functional testing#

Non-functional needs such as performance, capacity, security, accessibility, and resilience are user and business needs, and they are expressed as ACs on the items that introduce the behaviour, as described in Non-functional ACs. Because these ACs are written from the user’s point of view, they say what needs testing and what “good enough” looks like.

The main types of non-functional testing are:

Performance: whether individual operations are fast enough, such as a page becoming interactive within 500ms. Automated tests that validate the baseline on an unloaded system can be run regularly to catch regressions early.
Load and capacity: whether the system copes with realistic and peak volumes of users and data, and where it starts to degrade.
Resilience: whether the system behaves acceptably when things go wrong, such as a dependency becoming unavailable, a database failing over, or a network becoming slow.
Security: static analysis and dependency scanning on every change, dynamic scanning with tools such as OWASP ZAP ↗ regularly, and specific tests for security ACs such as one customer being unable to access another’s data. Periodic penetration testing by specialists complements these.
Accessibility: automated checks can run on every change and catch many common issues cheaply, but they only detect a proportion of accessibility problems. Manual testing with a keyboard and screen reader is also needed.

Some of these, such as dependency security scanning and automated accessibility checks, are cheap enough to run on every pull request. Others, such as load testing, need a representative environment and enough of the system to be meaningful, which is why a dedicated backlog item for them can be sensible, as described in Testing as part of implementation.

Leaving non-functional testing until just before launch is a common and costly mistake. Discovering that the system can’t handle the expected load a week before go-live leaves few good options. A walking skeleton approach allows us to establish performance and security testing early, while the system is small and easy to change.

Test data#

Good tests need good test data, and the effort and complexity required to get it right are often underestimated.

There are two broad approaches, each with trade-offs:

Purely synthetic data is generated specifically for testing. It avoids any risk of leaking real customer data and can be designed to cover specific scenarios precisely. But it can be difficult to match the volume, distribution, and quirks of production data: the unexpected characters, the customers with forty addresses, and the records created by an ancient version of the system. Tests that pass against tidy synthetic data can still fail against the real thing.
Copied and anonymised production data has the realistic shape and volume that synthetic data lacks. But the anonymisation has to be robust, and that is harder than it looks: free-text fields can contain personal details, and combinations of innocuous-looking fields can be enough to identify someone. Data that is not properly anonymised is still personal data, with all the legal obligations that come with it. The copies also need refreshing to stay representative, and the process needs maintaining as the schema changes.

Hybrid approaches can combine the strengths of both, such as generating synthetic data from statistical profiles of production data, or using synthetic data for most testing and carefully anonymised data only for performance and load testing where volume and distribution matter most.

Whichever approach is used, test suites should own their data. Each test should create, or reset to, the data it needs, rather than relying on data that happens to be present in a shared environment. Tests that depend on someone not having changed a record by hand fail unpredictably and are hard to diagnose.

Tests should be independent, so they can be run in any order and in parallel. Each must avoid polluting data used by other tests.

Test environments#

Test environments are a common source of pain. Problems include environments that have drifted from production or that nobody knows the state of, and teams queueing to use the one environment where a particular thing can be tested.

Automate infrastructure and configuration. Environments should be defined as code and created by automation, so every environment is in a known, reproducible state and differs from production only in deliberate, documented ways. An environment that has been configured by hand will drift, and “it works in test but not in production” is the painful result.

Promote the same artefact through environments. Deployables progress through environments following the stages of the SDLC, and progress is gated primarily by testing. A typical arrangement is:

During Implementation, the code is tested locally and by the pull request pipeline, and merging is gated on those tests passing.
Merged changes are deployed automatically to a test environment, where smoke tests confirm the deployment worked and the item can be explored and accepted in UAT.
Items that have passed UAT move to Ready to deploy, and the artefact is promoted through any further environments, such as staging, with automated tests at each step.
The same artefact is finally deployed to production, where smoke tests confirm it is healthy and the item moves to Deployed.

The artefact is built once and promoted, not rebuilt for each environment, so what reaches production is exactly what was tested. Feature flags allow changes to progress to production before they are made visible to users.

Ephemeral environments. One technique that can sidestep some of these problems is to create environments automatically for individual pull requests or branches and destroy them when no longer needed. This removes contention for shared environments, gives testers and Product Owners something to explore and give feedback on before the code is merged, and keeps each environment in a clean, known state. Ephemeral environments depend on automated infrastructure, and the cost of running them needs managing, but where they are feasible they remove a whole category of environment problems.

Local testing. Being able to run and test the system on a developer’s own machine gives the fastest feedback loop of all. Containers make it practical to run databases, message queues, and stand-ins for third-party services locally, as in the containerised postcode service in the definition of done example. The same containers can then be used in the pull request pipeline, so tests behave the same way locally and in CI.

Hybrid local and shared. Running everything locally isn’t always practical: some systems are too large, and some dependencies can’t be containerised. A hybrid arrangement, where the component being worked on runs locally and connects to shared instances of everything else, keeps most of the speed of local testing without needing the whole system on one machine. The shared components need to be stable and in a known state for this to work well, which is another reason to automate them.

Key points#

The aim of testing is to prevent bugs, not to find them.
Testing is part of implementation, not a separate stage. Cycle time includes testing and should ideally be one to two days.
Tests belong in the work item they test, and story point estimates include them. The exceptions are UI automation for very thin slices and some non-functional testing that needs several items in place.
Most testing is automated, at unit, integration, API, and UI levels.
Developers write most of the automated tests, and test their own work before asking a tester for input. This removes the test bottleneck and builds ownership of quality.
Specialist testers remain essential. Their critical, failure-focused thinking complements developers’ solution-focused thinking, and their greatest leverage is in preventing defects during refinement.
Testers guide test design, validate coverage with exploratory testing, curate the test packs, and lead the team’s testing practice.
The test pyramid has many unit tests, fewer integration and API tests, and few UI tests, with minimal overlap between levels.
Tests run automatically and fast enough for their trigger: some on every pull request, some after every deployment, some nightly.
Non-functional testing is driven by user-oriented non-functional ACs and should start early.
Test data is a trade-off between realism and risk, and test suites should own the data they need.
Environments should be automated and in a known state, with the same artefact promoted through them, gated by tests. Ephemeral, local, and hybrid environments all shorten the feedback loop.

Mistakes to avoid

A separate test stage. Recreates the handover, makes testing a bottleneck, and teaches developers that quality is someone else’s job.
Separate unit/API test items. Tests deferred into their own items get deprioritised, and the original item looks done when it isn’t.
Throwing it over the wall. Developers asking for testing before they have tested their own work.
Testers doing all the testing. Creates a bottleneck and puts all the testing thinking in one person’s head.
Removing testers because developers write the automation. Loses the critical thinking that stops the tests being a restatement of the happy path.
Testers stuck executing scripts. Spending the tester’s time on manual regression wastes their most valuable skill.
Testing everything through the UI. Produces a slow, brittle suite that duplicates lower-level tests and gives vague feedback.
Tolerating flaky tests. Teaches the team to ignore failures, which is worse than having no tests.
Leaving non-functional testing to the end. Performance and security problems found just before launch leave few good options.
Using unmanaged production data. Real customer data in poorly controlled environments is a serious risk.
Hand-configured environments. Nobody knows what state they are in, and “works in test” stops meaning anything.