Testing processes have evolved significantly over the past 40 years. As software becomes more complex, traditional software development processes are beginning to fall apart. Manual testing by individuals is forcing organizations to choose between cost, user experience, and time to market.
Luckily, there’s an answer. Continuous integration (CI) and continuous delivery (CD) are essential practices in modern software development processes. CI and CD work together to ensure successful development operations. Automating build, integration, and testing processes can help you optimize your development pipeline to weed out bugs early on and fix them as quickly as possible at a lower cost to your business.
In this guide, we’ll talk about the real cost of quality and the many ways in which automated testing can drive those costs down. We’ll also dive into some key testing strategies, including how to decide what to test, who should be in charge of testing, and how frequently you should execute different types of tests.
How much does quality cost?
Every developer knows that no bug fix is free. From the cost of labor to the costs of licensing, lab, energy, and deployment, ensuring the quality of an application can send thousands of dollars and hours of productivity down the drain. According to the 2018 Global Developer Report, more than half of the respondents said that testing is the most significant source of delays in the development lifecycle. In the 2019 report, testing remained the top choice at 49% of respondents.
These delays in testing can be expensive. The closer to production that bugs are found, the higher the cost of fixing them. Multiple studies by software companies, academics, and large banks have revealed the staggering financial differences in discovering bugs in production and discovering them in development.
- Discovering a bug in design is the least expensive. If a test fails in development, the code and its implications are still fresh in the developer’s mind, allowing them to solve the issue quickly and continue pushing toward the release date.
- Discovering a bug in quality assurance (QA) according to the IBM Systems Sciences Institute, is about 15 times more costly to fix than if discovered in the design stage. This massive jump in cost is due to the extra time and effort it takes to fix. The QA person has to report the bug to the senior manager, and the senior manager has to assign the fix to a developer. Taking the developer away from their current sprint and back to a feature they previously wrote is time-consuming and can potentially impact the current sprint.
- Discovering a bug in production is up to 7 times more expensive to fix than when detected in the QA phase, according to the same IBM study from 2010. This considerable expense is due to the cost of triaging, reproducing, setting up the environment all over again, and developers stepping back into code, potentially written by another developer. It can take QA days to understand what’s causing these bugs and the developer now has to switch context back to a feature they forgot about long ago. All while your business suffers.
One way to battle these costs is by testing as close to development as possible with a fast and continuous feedback loop. Continuous testing (CT) is the key to achieving that. Continuous testing ensures fast builds, quick tests, and shorter feedback between the developer writing the code and the developer testing it.
Organizations that have deployed continuous testing have noted a decrease in the cost of quality, an increase of agility, and an increase in on-time delivery. The more quickly developers are alerted of bugs, the faster they can fix those bugs, and the lower the cost of quality.
The role of CI/CD
CI tools allow you to automate the entire process from development to production and test throughout the process. They also allow you to compile all of the changes made by different developers into a master version of the application. Finding bugs early on requires testing each of these commits against the master branch.
By automating the testing process with CI, you can run multiple concurrent tests across browsers. Furthermore, CI provides you with the desired capabilities remotely, rather than being confined locally to one machine. It’s more scalable, more agile, and more cost-effective.
CI gives the developer the ability to test and automatically deploy their changes into the master branch without breaking the entire code. Before any change is committed and merged into the master environment, CI/CD tests for bugs as early as possible, reducing the amount of time and money it takes to fix them — and ensuring a quality application every time.
Challenges of automated testing
Traditional software development processes require months of testing before seeing any results. As software becomes more complex, it’s becoming more difficult to keep up with the amount of testing required. But test automation can drastically reduce the amount of time needed to test and fix bugs before deployment. But many organizations haven’t been able to live up to that. There are several challenges you may run into when trying to shift left, including:
- Skillset: Generally speaking, automated tests require coding. Just like coding your app, automated tests require organizing your code, setting data, importing libraries, defining classes, etc. An automation engineer is a developer by skillset. On the one hand, transforming manual testers into automation engineers is a tough transition and is likely to fail. On the other hand, automation platforms are built on open-source frameworks that your front-end developers most likely already know.
- Authoring: Authoring good tests take time, and even skilled developers may take days to write a single test. While authoring tests, developers need to think about how to handle element waits, make reusable components, and parameterize the tests.
- Initializing: Many companies hit a wall in the process of initializing data and preparing it for the tests. Automated tests depend on the integrity of their initial state, so if you can’t break down this wall, you can’t successfully implement automated CI testing.
- Maintenance: The stability of a test depends on the stability of the application. If you change the screen of an application, the tests start to break, so you have to change each test to accommodate those changes. Maintenance can add up to 30% of a tester’s overall work, but automated tests can streamline this process.
- Scalability: Some companies run into issues when they try to scale their businesses. As the application becomes more complex and you need to release more often on a larger scale, you’ll need automation.
To overcome these challenges and begin using CI/CD properly, you need to transform your company’s testing mentality and development processes.
4 essentials for continuous testing
TESTS ORGANIZATION
To optimize your QA processes, segment your tests into suites, and run different suites at different stages of your development cycle. Full regression might include thousands of tests, but running those tests every time a developer pushes code costs an enormous amount of time and resources. If two to three key tests failed, there’s no point in waiting for the full suite to complete.
Most organizations run a subset of their scenarios on every commit with a full regression each night. Figuring out which scenarios you should include in your sanity check and which are full regression is not an easy task. High-value scenarios, ones that are prone to bugs and parts in the code that frequently change, are good candidates for your sanity suite. These should be short, concise, and specific. The more extensive regression suites can be saved for after a successful sanity suite run.
TEST AUTOMATION
Using manual testers ensure the quality of rapidly changing applications is a short term solution at best—it will slow your release cycles and hinder your competitiveness. The best way to speed up the authoring, execution, and maintenance of tests is to automate them. Automated tests can run 24/7 across multiple projects, providing optimal coverage for QA. Ultimately, this will increase on-time deployment and ensure quality, all at a lower cost to your business.
REMOTE RUNS/GRID
Once you have dozens of tests, you will want to run them across browsers to ensure that your application works properly in your users’ environments. A Selenium grid can spin up and down different combinations of browsers and operating systems. A large number of such instances is required to run tests in parallel and provide a short feedback loop. The time it takes for the dev team to get feedback is a combination of the number of tests that you run as part of your sanity check, the length of those tests, and the number of tests you can run in parallel. By increasing your levels of parallelism, you shorten your feedback loop — ultimately increasing on-time deployment and reducing overall development costs.
BUILD-TEST-DEPLOY AUTOMATION
The software development cycle includes dozens of handovers across different stages. Human intervention, of course, can always delay this process. By shifting left, you can automate this handover and optimize the entire process.
Continuous integration (CI) and continuous delivery (CD) servers can help automate handoffs between steps and accelerate release cycles. As you know, CI enables developers to monitor events in the development cycle and trigger a set of actions once certain events have occurred. For example, CI can enable developers to merge their changes back to the main branch of an application as often as possible. Then, they can program the CI server to trigger automated tests against this new build and that every new commit integrated into the main branch doesn’t break the application.
While CI deals with the build-test part of each development cycle, CD focuses on what happens with a committed change after that point. CD allows developers to release these new changes quickly. This allows you to deploy in small batches that are easy to troubleshoot in case of a problem.
But to work in a CI/CD environment, you need extremely stable tests that you can trust to run quickly and frequently. You need to have the proper infrastructure of automation to minimize false and flaky tests.
So how often should you test, anyway?
How often you should test depends mainly on the complexity of your application, the frequency of new releases, and how critical the application is to the business. As these factors rise, you should increase testing frequency.
According to the 2018 Global Developer Report, almost half of organizations deploy code multiple times a day, requiring a greater frequency of testing as well.
HOW OFTEN DOES YOUR ORGANIZATION DEPLOY CODE?
Ideally, developers should run tests on every commit. Segmenting them into automation suites helps to ensure the right set of tests are triggered each time. The really innovative companies are running between five and 100 tests a day — that’s a lot of tests to juggle without automation. Using CI/CD, you can shrink your testing cycle and free up developers to focus on sprints.
Keep in mind that tests that fail require attention. Testing too frequently without allocating time and resources to repair failed tests diminishes their effectiveness and can lower confidence in the entire QA process.
Achieving proper coverage
When you’ve decided to commit to continuous testing, it’s important to define optimal coverage. As you know, applications can have thousands of different scenarios. Some are more common than others and some are more critical to the business. In an effort to ensure quality, some organizations may try to test every possible scenario. But this leads to hundreds of lower quality tests that have little value.
What you test should differ depending on the type of application and what drives your business. Do you care more about the functionality or the way the user interface looks? You have to think about how critical certain scenarios are to the application, and then segment those scenarios to ensure you’re getting the right coverage with each test.
If you’re a company like Coca-Cola, for example, you definitely can’t afford a bad visual experience — so maybe you’d do a UI validation down to the pixel of your logo. But if you’re developing a business application, then you probably care more about the functionality of the application than the way it looks. Achieving proper coverage Regardless of what you decide, there are three areas that every organization should test:
- Critical paths
- Stable parts
- Revenue generating flows
These are the essential parts of your application, the paths that current paying customers move through most frequently, such as the login. If the login fails, you’ll lose business and take a hit on your revenue. In other words, this path is critical to your business. You can use analytics tools like Google Analytics to see which of these user flows are most common and start listing out the ones that are most important to user experience and revenue.
Use regression testing throughout each stage of development to make sure specific, critical functionality always works. Exploratory testing can be used on a lower frequency by manual QA to find bugs in other areas.
Remember: Your tests should be short, concise, and specific. With automated CI/CD, you can ensure you’re squeezing maximum value, coverage, and efficiency from each test.
Don’t overlook debugging
Too often, when choosing a test automation solution, organizations don’t factor in the time and cost of fixing tests. Yet, the real work starts when a test fails. Reducing cost and feedback loop requires rich information to quickly determine the root cause. Screenshots, log files, detailed error messages and the ability to reproduce can reduce triaging time. Integrating your tests to your bug reporting tool such as Jira, Trello, or Slack can help teams communicate better, leading to shorter time to resolution.
Quality is everyone’s responsibility
As modern software teams adopt more fluid development methodologies, the role of testing is shifting from the QA team to everyone. Leading software-defined companies are acknowledging the significant role developers can play in ensuring quality. Some organizations are shifting left, having their developers take a significant part in testing.
Why? Because developers know what changes they are making to the code and therefore know the critical paths that need to be tested in a given timeframe. It’s easier for them to make changes to the tests as they make changes to the application.
Traditionally, developers have taken a reactive approach to QA. They make changes to the code, the QAs’ tests fail, and the developers go back to fix the bugs. But testing can no longer be a secondary issue. By taking a more proactive approach to testing with CI/CD tools, developers can eliminate many challenges of the testing process and drastically improve quality, all while reducing the overall cost of the DevOps pipeline.
Proactive testing means making changes to the tests as changes are made to the application. Alternatively, organizations can take a hybrid approach to this, where QA creates the first batch of tests, and the developers each maintain their own area of the application.
Regardless, QA and Development should work together to design and implement tests that efficiently validate the application’s most important user flows.
The future of testing
While most IT organizations understand the critical need for and importance of DevOps, the 2018 Global Developer Report reveals few have completely adopted the methodology. A study of more than 5,000 responses from software professionals around the world, the report uncovers that although the majority believe DevOps improves cycle time, only 23% say they have adopted a DevOps workflow.
However, companies that are still relying on manual testing and traditional development processes are severely lagging. They’re spending thousands of dollars and hours of productivity on bugs that could’ve been fixed months beforehand for a fraction of the cost.
The importance of CI/CD is no longer in question — continuous testing is the way of the future. To keep up with leading companies, you have to integrate automation into your DevOps process. With skilled developers managing your CI/CD servers, you can author short, concise tests that run continuously with every commit and deploy successful changes automatically, cutting out unnecessary costs and ensuring on-time deployment.
But it’s important to get everyone on board. The first step to successfully integrating automated CI/CD into your DevOps process is to communicate its value to everyone in your organization—developers and QA engineers alike.
With smart, automated testing, you will increase on-time deployment, reduce the overall costs of production, and increase the agility of your applications.
Quality as an organizational initiative
The way in which a company’s R&D efforts are structured will have a direct impact on the quality of their products, services, and procedures. For software specifically, the question revolves around ownership of the development and QA process. With more tools becoming available to developers and QA testers, and with the integration of AI, companies can customize the structure in any way they see fit.
Companies like Engagio, Wix, JFrog, and NetApp are already using automated testing solutions, and they all have a unique organizational structure for their R&D activities. A solution, like Testim, that makes use of artificial intelligence and allows anyone to easily author, execute, and optimize tests will bring value to any business.
