Continuing our trend of covering testing-related topics, today, we talk about test coverage techniques. What are these techniques, and why should you be aware of them?
Let’s face it: automated testing has won. Sure, manual testing still has its value and place. But having a proper automated testing strategy in place is vital for any modern software organization. The problem is that coming up with a sound testing approach is certainly easier said than done.
One of the biggest challenges of testing is knowing when to stop. How much testing is enough? Should we strive to test 100% of our application? Or should we prioritize the areas that are more likely to contain problems? Should we allocate our resources uniformly throughout the application or concentrate our efforts on the most critical modules?
In today’s post, we answer the questions above—and more. You’ll understand what the test coverage techniques you should add to your toolbelt are and how to use them to make the most out of your testing approach.
We start with fundamentals, by defining test coverage in detail and even clarifying a common misconception. After that, we go through our list of coverage techniques, defining each one and explaining how they can provide value to your overall testing strategy.
Let’s begin.
Test Coverage Techniques – Let’s Start With The Basics
We’re about to go through our list of coverage techniques. Before we do that, though, we’ll cover some basics. We’re going to define what we mean by “test coverage,” explaining how it differs from “code coverage” and what benefits we can obtain by adopting it.
Defining Test Coverage
It should be clear that test coverage techniques are techniques we can use to improve or maximize our test coverage. So, what’s left for us is to define “test coverage” itself.
Test coverage is the process you use to determine whether you’re testing everything you’re supposed to test. It allows you to check the quality of your testing. By finding areas that aren’t currently covered, test coverage helps you develop tests for those areas, increasing the overall quality of your test suite.
Test Coverage vs. Code Coverage
People often confuse Test Coverage with Code Coverage. That’s understandable since they’re very close terms. However, despite there being similarities, these two terms refer to different concepts, and you shouldn’t use them interchangeably. Let’s first see code coverage.
Expand Your Test Coverage
Code Coverage
Code coverage is a metric related to unit testing. The idea is to measure the percentage of lines and execution paths in the code covered by at least one test case. To measure code coverage, you’d typically need a dedicated coverage tool, besides the unit test framework you already use.
For instance, if Mocha is your test framework of choice, you can get code coverage by leveraging Istambul. However, there are unit test frameworks that provide coverage themselves, such as QUnit.
The most common type of code coverage is line coverage. Line coverage measures the ratio of lines of code that are covered by at least a single test. This metric, despite being useful, is often not enough.
That’s because you can have 100% line coverage but still have untested scenarios in your application. Consider the following excerpt of code:
if (condition && condition2) {
// important stuff happens here
}
// more code here
The example above is super silly, but I think it gets the point across. Imagine we have a test case that passes true as the value for both parameters. In such a scenario, the line coverage would be 100%.
However, the test wouldn’t have exercised the scenarios in which either—or both—of the conditions is false and the if block isn’t executed. The branch coverage, on the other hand, would be able to detect that test omission.
Regardless of the variety of code coverage in use, there’s an important caveat you need to be aware of: the famous Goodhart’s Law. This “law” says that when a metric becomes a target, it ceases to be a valid measurement. In other words, if you define goals around the metric and attach outcomes to it, people might be tempted to game it.
Who’s responsible for code coverage? Since code coverage is a unit test metric, developers should take care of it.
Test Coverage
On the other hand, test coverage is a job for the QA department. As mentioned in the previous section, it means making sure everything that needs to be tested, is. What is the difference between code coverage and test coverage?
Code coverage is a lower-level metric. Its only concern is measuring how thoroughly the unit tests cover the existing code. Since it’s a technical concern, the ones who care the most about test coverage are developers.
On the other hand, you can think of test coverage as a higher-level technique, whose goal, in short, is to determine how well the application is tested when it comes to its requirements. As such, test coverage isn’t a concern only of engineers, but of all people who perform testing in the organization, especially the QA professionals.
Benefits of Test Coverage
Before we go on to cover—no pun intended!—the actual test coverage techniques you need to know, we’ll briefly mention the main benefits of test coverage. After all, learning about specific techniques is a moot point if you’re aren’t convinced about the whole enterprise to begin with.
So, why should you care about test coverage? Here are a few reasons:
- Test coverage provides accountability when it comes to testing and QA as a whole. Measuring test coverage makes it possible to share the progress made with your QA strategy to all relevant stakeholders.
- Test coverage helps you improve your software testing strategy. You can’t improve what you don’t measure: measuring test coverage is the first step in increasing it.
- Test coverage helps you reason about the ROI of automated testing. By tracking your test coverage over time and correlating it with quality measurements, you can get a really great idea of how investments in software testing translate into a better product.
Test Coverage Techniques You Should Know
With the basics out of the way, it’s finally time for us to dig deeper into our list of test coverage techniques: product coverage, risk coverage, requirements coverage, boundary-value coverage and AI-aided test automation. Let’s do it.
Product Coverage
The first technique we’re covering is product coverage, which simply means looking at test coverage from a product perspective. Product coverage means to answer the questions: which areas of the product have you tested?
Let’s say the application you’re testing is a simple calculator (only the four arithmetic operations available.) Of course, you should verify that the operations work as intended. But only that is not enough. There are other factors you must take into account.
How well does the calculator handle really large numbers? (In other words, what’s the range it supports.) What’s its behavior when the user does something unexpected, like pasting non-numerical characters into the text field or trying to divide by zero?
You can’t claim your application has complete product coverage unless you verify the scenarios described above and more like it.
Risk Coverage
The second item in our list is risk coverage. This test coverage technique, as its name suggests, is all about assessing the risks inherent to an application and testing them thoroughly. In short, the technique consists of listing the risks involved in the application and making sure they’re covered by tests.
For instance, imagine you’re testing a mobile app that helps users manage their personal finances. The application queries a third-party API to search for financial data such as currency exchange rates or stock market share prices. One obvious risk here is that the third-party API could go offline.
How is the application supposed to behave in a scenario like this?
If you want to say that your application is covered, you have to list all of its relevant risks and answer the questions they elicit.
Requirements Coverage
Requirements coverage is arguably the most important of the techniques we’re covering today. After all, your application might work really well, but if it doesn’t meet the users’ requirements, it’s useless.
Let’s go back to our previous example. Suppose our personal finances app was comprehensively tested, at least in regards to risk coverage, and then released. The unavailable API scenario was gracefully handled, and everything works nicely.
Or so it seems. Then the first reviews start coming. They demolish the application. What happened?
In short, the list of requirements wasn’t taken into account when creating the test plans. The requirements clearly stated that the app should offer automatic sync with the users’ bank accounts. That feature was never implemented, but it was certainly marketed.
So, requirements coverage is both the most basic and the most essential of the techniques so far. It helps ensure the requirements are fulfilled, which is the whole raison d’être of software development.
Boundary Value Coverage
Suppose you have a feature that can take numbers as input, but it should only accept numbers from 0 to 100, inclusive. In scenarios like these, to ensure the boundaries are properly tested, you’d typically test at the boundary, and immediately below and above the boundary.
So, for the minimum value (0), you’d need to test with -1, 0, and 1. For the upper limit, you’d test with 99, 100, and 101. That way, you reduce the likelihood of off-by-one errors in your application.
Boundary-value coverage consists of ensuring such boundaries are tested across the application.
Compatibility Coverage
This one is closely related with the concept of cross-browser testing. In short, you must make sure your tests cover your application across different browsers and operating systems. Sure, you don’t need to test across all possible combinations—that would be impractical. But research what the three most used browsers are across Windows, macOS, Linux, iOS, and Android, and then make sure to cover those platforms.
AI-Aided Test Automation
Last but not least, we have AI-aided test automation.
The main difference between this technique and the previous ones is the total reliance on automation. Over the previous items, we’ve presented techniques that can make use of automation but are still heavily reliant on manual processes.
AI-Aided Test Automation, as its name makes clear, is all about automation. It consists of the use of test automation tools that rely on AI to take your testing approach to a whole new level. Such tools can enable self-improving test suites, that apply machine learning in order to learn with each execution and improve themselves.
That way, you end up with a robust and stable test suite that not only provides your app with test coverage but also test coverage that keeps on improving itself, over time, on its own.
How to Calculate Test Coverage
Calculating test coverage is actually fairly easy. You can simply take the number of lines that are covered by a test (any kind of test, across your whole testing strategy) and divide by the total number of lines in your application. So, if your app has 1000000 lines of code, and 380000 of those are exercised by at least one test case, your test coverage is 0,38 or 38%.
We could represent this calculation with the following formula:
Even though the calculation itself is trivial, figuring out the number of lines in the app that are currently under test can be quite a challenge. That’s because there are many different types of software testing, which use different tools—or simply no special tools, in the case of manual testing.
However, much more important than coming up with an exact figure is adopting techniques that ensure you test what matters the most in your application.
Test Coverage Techniques: Learn Them and Put Them to Good Use
Automated testing is essential for any modern software organization. Its benefits are well-documented, as well as its return on investment.
Putting a sound testing strategy in place isn’t the easiest of tasks, however, as there are many challenges involved. One of those challenges is simply to decide what and how much to test.
Fortunately, there are techniques you can use to help you make the most out of your tests, which we call test coverage techniques. As you’ve seen in this post, people often confuse test coverage with code coverage, but it does well beyond that. Despite the similarities, these are two different concepts, and the terms shouldn’t be used interchangeably.
Testing coverage techniques help you track the quality of your tests and cover areas currently not validated. They might help you decide what to test based on risk, prioritizing the most critical and likely to break areas, which allows you to allocate your resources more efficiently. With the help of AI, they can even enable self-correcting test cases, that learn from each execution and use that knowledge to improve themselves, easing the burden of test maintenance.
Is testing coverage the silver bullet of testing? Of course not. Software testing—and software development in general, for that matter—is an inherently difficult activity. It’s challenging, and it will always be. But the rewards are worth it, so that’s why we keep doing it.
Thanks for reading, and I see you next time.
Carlos Schults wrote this post. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.
