No company would release a piece of software without testing it. But there are many ways of testing software. End-to-end testing is one way. It might even be the most important way, though often the most time- and resource-intensive one. Luckily, there are ways to automate end-to-end tests.
In today’s post, we’ll clarify not only what end-to-end testing is, but how it fits into an overall testing strategy that can benefit every organization. You’ll see that the benefits of test automation go way beyond than just saving time. Here’s a summary of what we’ll cover:
- What is end-to-end testing?
- Why do we need end-to-end tests?
- The overall testing strategy
- Using end-to-end tests
- End-to-end testing examples
- The Importance of Automation in a Software Testing Strategy
- Which Layers of an App Are Involved In E2E Testing? A Simple Checklist
What Is End-to-End Testing?
In theory, end-to-end testing (E2E testing) is the process of testing a piece of software from start to finish as it will be used by the actual users. For a web application, you would start a browser, navigate to the correct URL, use the application as intended, and verify the behavior. In the case of desktop application, you would start up the application, use it, and verify its behavior too.
If you’re testing an API, you would make calls just like the real clients would do.
Whatever your piece of software, the idea is to run the application with all of its dependencies: database, external services, metrics, logging, etc. This can still be in a test environment, separate from the production environment. But ideally, this test environment represents the production environment as closely as possible.
Expand Your Test Coverage
You may have noticed that I started this section with the words “in theory.” This is because it might not always be possible to run the application without real side effects, even in an isolated testing environment. This can be the case when external dependencies only exist in the production environment.
For example, in an industrial setting, it’s not uncommon to have only production PLCs (computers that drive industrial machines). You don’t want your end-to-end tests influencing the machines in the production environment. So in this case, you’ll have to be creative and stop the commands from going to the PLCs, while still verifying their correctness.
There are many situations where you might want to “fake” certain components of your overall application for your end-to-end tests. But in end-to-end tests, you want to reduce these fake components to a minimum.
Why Is It Called End-to-End Testing?
One thing that might be confusing about end-to-end testing is the name. Why do we call it that?
Picture a simple web application. When interacting with the application, the user performs an action—for instance, enters information into a form and presses a button—which then triggers the frontend of the application to send a POST request to a RESTful API, which does some processing and finally adds some row to a table in the database.
If you imagine this whole process as a line, one end of the line would be the “place” at which users interact with the application—in our example, this would be the user interface. The other extreme would be the database. So, that’s the reason why end-to-end tests are called that: they start at one end of the application and go all the way to the other end, exercising all of the layers that exist in between.
Why Is E2E testing Important?
Software can and should be tested on many levels. At the lowest level, we have the unit test. This is where programmers write a small routine to test the routine they are actually writing. All these small routines together will form your application.
But it’s useful for the programmers to write these small test routines so they can have fast feedback on whether or not they are implementing a feature correctly.
However, just verifying that all those small pieces work isn’t enough. It doesn’t ensure that they work correctly together. This is where integration tests come into play. In integration tests, two or more components are used together in a test and the result is verified.
But even if all these tests run perfectly fine, you could still have bugs or unmet requirements.
First of all, if you have a UI, unit and integration tests don’t cover the UI. Is the UI structured correctly? Is it a pleasant experience? Are all the necessary elements in place? Can the user achieve what they want to achieve? These types of questions can only be answered by an end-to-end test.
A second reason why end-to-end tests are necessary is that there are so many moving pieces in modern software: servers can cause applications to behave differently, configuration must be set up correctly, and external dependencies influence our application. Testing separate components will not cover all this.
Does this mean unit and integration tests don’t have value? Definitely not. Unit, integration, and end-to-end tests all play an important role in an overall testing strategy.
What Are Different Types of Testing? The Overall Testing Strategy
Testing software is an activity that happens on multiple levels, by multiple people in different roles. Developers test their code with unit and integration tests, while testers work together with developers to define test scenarios that they’ll run manually or automatically.
End-to-end testing is the category of tests that’s drawn at the top of the testing pyramid. The testing pyramid gives us a guideline of how many types of tests we should have. Because they are easy to write and maintain, and fast to run, it’s valuable to have many unit tests.
As we move up the pyramid, our tests start using more pieces of our application. These tests become slower to execute and harder to maintain.
That’s why there are usually fewer end-to-end tests than there are unit tests. But the end-to-end tests are still featured in the testing pyramid. In the section above, I’ve explained why they’re still valuable.
The different categories of tests each have their unique value in the overall strategy of testing your software. But many organizations still perform the end-to-end testing manually—even when there are tools that can automate end-to-end tests. With automated end-to-end tests, you can add and maintain more tests in less time.
Using End-to-End Tests
It isn’t always clear how to start with end-to-end tests. Many companies manually test new features and leave it at that. But this allows bugs in features that previously worked to go unnoticed. Here are some practical steps to get started with E2E testing:
- Identify test scenarios.
- Map the steps in each scenario.
- Use the mapped steps to perform manual testing.
- Automate the tests.
- Add your tests to the CI pipeline.
Identifying Test Scenarios
Get together with business owners, developers, and testers and think about some scenarios that you want to test before every release. This doesn’t have to be an exhaustive list. It’s better to have a minimal set running first, and then expand on it later.
Mapping Test Steps
Using this list, you can now map out the different steps in each scenario. Write down the steps that need to be performed and the expected results.
Optionally, Execute Mapped Steps as Manual Tests
Notice that we still haven’t automated anything yet. You can already use these scenarios to perform manual end-to-end tests. Of course, the real value lies in automating these tests.
But if, for some reason, it can’t be automated (yet), at least you have some end-to-end tests to get started with.
Automating The Tests: How Do You Do End-to-End API Tests?
However, a next step will be to automate these tests. An application without a UI like a REST API can be tested by executing a series of calls to the API and checking the results. Tools like Postman or a Cucumber library are useful here.
UIs are traditionally harder to run in automated tests. Tools like Selenium can run automated UI tests, but easily lead to fragile tests. They also require a decent amount of work to build and maintain.
With Testim, you can create tests just by using your application. Testim records your actions and can then run those actions as an end-to-end test. Testim is unique because it will learn about your application.
If you change your UI, Testim is able to notice this and adapt the test. This leads to less fragile tests, allowing you to focus more on improving your product.
Adding The Tests To The CI Pipeline
Once you have a few end-to-end tests set up, make sure they run in your CI process. This provides valuable feedback to developers so they know if they broke anything with a code change. As you experience the value of end-to-end tests, you can then repeat the process: identify test scenarios, list the steps, and automate the test.
What’s an End-to-End Testing Example?
What would an example of (manual) E2E test look like? That’s what we’ll cover now. To do that, we’ll use the website demoblaze.com, which is a demo site for testing purposes. To perform the test, you can follow the steps below:
- Go to https://demoblaze.com.
- Click on the first product (the Samsung Galaxy s6 phone, with a price of $360.)
- On the product’s page, ensure the name and the price of the product are correct.
- Then, click on the “Add to cart” button.
- Ensure you see an alert with the message “Product added.”
- Click on the “Cart” link, in the main navigation menu.
- Ensure the cart has a single product with the correct info, and the total is $360.
- Finally, click on the “Delete” link and verify whether the product got deleted.
This is a simple test case for testing whether the cart functionality of an e-commerce site works as intended. With a robust codeless test automation tool like Testim, automating a test like this one would be a breeze.
The Importance of Automation in a Software Testing Strategy
In the previous section, we’ve offered you an example of what a manual end-to-end test case could look like. However, there’s great value in automating your tests. We’ll now delve into that topic in a deeper way.
Automated Testing Helps You Not Only To Save Time
So, why is automated testing so important? A correct but incomplete answer would be: “to save time.” Sure, automated testing does help you save a lot of time, which is already a huge benefit in itself. But the real value of automated testing is way deeper than that. We can summarize such importance in a single sentence: without automated testing, you can’t deliver high-quality software quickly. Period.
That’s not to say that automated testing is the only factor that leads to software quality. You should also be aware of the main code smell, adopt coding best practices, and use processes—such as code reviews—and tools—for instance, linters—that can help improve the quality of your code.
But without automated testing, you’re pretty much hopeless in your struggle against software quality issues, particularly the ones we call regressions.
Software Suffers From Regressions
What are regressions? Well, do you know those situations when you fix a defect—or add a new feature—and then another thing stops working? That’s a regression.
Though physical systems can suffer from regressions, software is particularly prone to them, due to its very nature. Since any part of the application can interact with any other part, even the smallest change to a codebase can, in theory, cause unintended consequences to another seemingly unrelated part.
Automated Tests Can Save You From Regressions
Even though we could list many reasons why your software testing strategy should include automation, protecting your apps against regression is certainly one of the most important ones. And why is that?
- Software applications are so susceptible to regressions, there’s really no way to have the bare minimum of confidence in the quality of your app unless you completely test it after each and every change.
- Doing this manually would be a sishypeahn task. Even if you could manage to pull it off, the process would be so slow that all of your competitors would beat you to market.
- Thus, aggressively pursuing automation in your test strategy is the only way you can have confidence in the quality of the code you ship while maintaining the ability to ship it as fast as you need.
Which Layers of an App Are Involved In E2E Testing? A Simple Checklist
You’ll often notice that there seems to be some overlap between different kinds of testing. Though in definitions you might find on Wikipedia and even blog posts like this one things might look very clear, the reality is often messy.
When it comes to E2E testing, you might discover that the lines between it and other forms of testing might often be blurred. For instance,
- How does E2E testing differs from UI testing?
- How does it differ from manual exploratory testing?
- And what about integration testing?
Even though we’ve already talked about these other types of testing, things might not be entirely clear yet.
A good rule of thumb to determine whether a given test is an E2E test is to check which layers of the application it interacts with. For that, the following checklist might come in handy:
- Presentation (or UI) layer. All E2E tests necessarily interact with the UI layer. Keep in mind that this isn’t necessarily a graphical user interface (GUI.) It could be a command-line interface, a text-based interface. The endpoints of a REST API could also be considered a type of user interface.
- Domain logic layer. A true E2E must interact with the app’s real domain logic. Imagine an application that consists of an Angular frontend that consumes a REST API. Now suppose you have some tests that drive the application through the frontend while mocking the API with a fake one for testing purposes. Though such tests would be valuable, they would not be considered E2E tests.
- Persistence layer. Unlike other types of tests such as unit testing, end-to-end tests talk to the real persistence solution. The persistence solution doesn’t necessarily have to a database; we could be talking about session/cooks or even the browser local storage. All that matters is that it’s the real storage solution used by the application.
How would our example from the previous section fare against the checklist above?
- Does it interact with the presentation layer? Yep, definitely.
- Does it involve domain logic? Sure. Had we added another product worth 140 dollars, the total amount would have to be updated to $500. That’s domain logic.
- What about the persistence layer? The site does keep the products we’ve added to the cart while navigating between pages, so there’s definitely some persistence happening.
Test Your Software, Measure Your Results
This article aimed to clarify not only what end-to-end testing is, but also why it’s necessary. End-to-end testing accompanies other types of tests like unit and integration tests. It fits in an overall testing strategy that every organization will benefit from. This rings especially true if you automate your end-to-end tests.
So, I’ve managed to convince you of the importance of end-to-end testing—at least, I hope so! What should your next steps be?
Well, testing is an investment as any other, and you’ll want to ensure its ROI is as high as possible. But you know what they say: you can’t improve what you don’t measure. It’s essential that, in order to improve your end-to-end testing approach, you track the most valuable metrics.
Those metrics include the percentage of passed test cases and test suites, the amount of active tests and the average duration for test runs. Lucky for you, these are some of the metrics you can track and improve with Testim’s new managerial test automation reports.
This post was written by Peter Morlion. Peter is a passionate programmer that helps people and companies improve the quality of their code, especially in legacy codebases. He firmly believes that industry best practices are invaluable when working towards this goal, and his specialties include TDD, DI, and SOLID principles.
