The process of building and deploying software is undergoing a revolution, not unlike the invention of the assembly line. That seminal development allowed Ford Motor Company to crank out consumer vehicles so fast that it positioned them as the market leader almost immediately.
Today, old manufacturing processes have been replaced by the knowledge-worker-driven software development process. Unfortunately, the same traditional wisdom that applies to Six Sigma style manufacturing cannot be applied to software development. It is an entirely new discipline that has existed as a major, widespread career only for the past 30 years.
In manufacturing, more bodies on the line mean more products out the door. But in the software development world, this has traditionally been an inverse ratio: The more people who work on a project, the slower it moves overall. That’s because software development is mental gymnastics. Developers must keep the whole model of the program in their minds while they work, and once things get too big to keep in a single person’s brain, inter-personal communication and coordination become a major bottleneck.
That’s why agile software development was invented. By cutting software into smaller bits, or “features,” teams can focus on the same task for shorter periods, ensuring that new ideas make it into software projects faster.
Agile processes took enterprise development from releasing annually or every six months to releasing once a quarter, or even once a month. That, in turn, drove up business revenues, as new features drew new customers, and software increasingly became a line of business, rather than simply a business tool.
Today, this increasing velocity of software development has led to continuous integration and continuous deployment, practices that allow the software to be updated every week and sometimes every day. Teams with the skills and processes in place to do it properly can even ship new code every hour.
That requires a careful balancing act, however. Teams must be disciplined, processes must be optimized, and above all else, tests should be automated and easy to maintain. Specifically, key verifications should be automated and available on demand.
Testing as Enabler
Test automation is the single most important factor in maintaining software quality at high velocity in CI/CD. In order to enable testing at this high rate of release, automation and long-term management of tests are top priorities for any manager to consider. That is to say, these should be at the top of their to-do list, but are not a must — sometimes manual testing remains the best choice.
Test automation is easily seen in the wrong light, however. Managers can look at the idea of automation and see it as a new way to reduce headcount or to save money on overall development costs. This is incorrect, however, and pursuing test automation for these reasons alone is to miss the forest for the trees.
The real reason test automation enables CI/CD is that it shortens the feedback loop between developers and test results. With proper automated tests in place and running at scale, developers can see the results of their work in minutes, allowing them to quickly return to fix obvious problems.
When a developer runs a build, they are no longer working. Historically, these developers would get coffee, play ping-pong, or even go to lunch. If a build and test process takes an hour or, worse yet, requires an actual person to do the work, your developer is burning cycles when they’re waiting.
If you’re running 20 builds a day at 10 minutes each, that’s 200 minutes wasted. That’s not only going to put your developers behind schedule, but it’s also going to hamper their attempts to push changes to production and bolster the company’s bottom line with new products and features.
If those 20 builds-per-day can be reduced to one minute each, you’ve just saved every developer three hours a day. That means they can cut more code, ship more features, and fix more problems.
Of course, making builds run quickly requires tests to run quickly as well. Regression tests are repetitive and must be maintained over time when buttons move or workflows undergo simple changes. Regression tests cannot be left out of your testing suite, as they can catch showstopper bugs that were previously fixed but became live again due to code updates. Behind the UI, functionality tests are perhaps your most important path to verification. In essence, you should be testing the business itself when testing functionality. If your application’s use case is directly tied to business revenue intake, making sure it’s stable while developing at velocity requires tightly managed tests across the range of the application.
When business value derives directly from the throughput of customers in an application, testing performance and security are also must-haves for every build cycle. Thanks to the modern business web, customers can quickly switch to a competing product, and the never-ending deluge of security concerns online also means that your data must be protected during testing. This adds another layer of complexity to the testing toolchain, as live data must be scrubbed of identifying information before being used in a test environment.
With that many tests required to run on every build that goes to production, and with multiple builds being pushed every day (in the ideal scenario), management of those tests must not be cumbersome or difficult.
UI testing, for example, is the most difficult type to get right consistently and in an automated fashion. If this is where your testing plans begin, you’re building a skyscraper from the top down. Instead, unit testing should be the fundamental, well, unit of your testing regimes. Building up from those foundations, a full battery of tests can be constructed to ensure pushing to production is only done when things are known to be stable.
Every testing environment is different and will require a mix and match of the various levels of tests. From inside the code to inside a harness around the code, every test is a unique component in your project. For this reason, test automation requires careful planning and a distinct roadmap to build out, based on your team’s needs, infrastructure and tooling.
Automated Testing: 8 Strategies and Tactical Plans
As Donald Rumsfeld once said, “You go to war with the army you have, not the army you wish you had.” That applies in software development, as well. You’re likely already working with a selection of tests, and they’re likely not as good as you want them to be.
But you’re going to start there no matter how you feel about those tests. The first step on the path to true CI/CD integrated test automation is to assess your tests. What do you have, and what do you need? Are you short on UI tests? Are there separate mobile tests and web tests? Is there a backend API you can test directly, without an obfuscating UI layer?
If you have an API layer, that’s a great place to begin automation. You want to find the spaces in the application that tie directly to the business logic and the business use case. The core of the application or service is where the most important functionality resides. Don’t try to automate everything; instead, focus on the most important bits of the application, and automate the tests as close to the core as possible.
Just automating these tests isn’t going to be the whole job, anyway. Once you begin, you’ll also want to ensure that the test environment, data, and reporting are also in place and serve in a reliable fashion for on-demand testing. The goal here is to get the infrastructure for automated testing at scale into place before you try to automate everything.
Once you have these infrastructure components in place to support that small set of core tests, and once that pipeline is running smoothly, you can begin to add in the other tests in your arsenal and automate them as appropriate.
Now that you’re into the meat of test automation, here are some tips and tricks for success.
01 PRIORITIZE
Make a list, prioritize it, and focus on the functionality related to your most important metric. For example you might focus on the functionality that converts users.
02 REUSE
Build for reusable parts, with atomic, clear naming conventions that tie back into the vocabulary of your source code, variables, and classes. Reuse comes from proper architecture, however, not from copying and pasting.
03 TWO LEVELS OF ABSTRACTION, MAYBE MORE
If you have clicked on your tests, you are doing it wrong. Those clicks need to be driven by an automated system that can handle changes in UI and functionality. Otherwise, your teams will spend all of their time updating tests as the target changes. Think about your average code change or addition of a new feature. If it was a major change to the UI or functionality, it likely broke a lot of tests, too. Automated test management and design can help to alleviate the need to rewrite tests with every passing software update. Use the Page Object pattern: Wrap your HTML in an API that will allow you to directly access elements with tests.
04 SINGLE-PURPOSE TESTS
Keep tests scoped to a single purpose. When things go wrong, the last thing you want to be wondering about is the test layer. If your tests target multiple things, they’re single points of failure for multiple points of contact. This is exactly why converting manual test cases to automated tests, one-to-one, is a very bad idea.
05 USE A CONSISTENT INITIAL STATE
Variables are bad. This is a science, not a college final. This is a marathon, not a sprint.
06 NEVER SLEEP A PROCESS
This is the source of all evil. When things don’t wake up, processes stall. Keep all processes awake during testing. Timeouts are another source of common problems. Keep a close eye on them.
07 MINIMIZE THE NUMBER OF CONDITIONS
The more moving pieces you have in a piece of software, the more things there are that can go wrong. Setting default states for applications in testing environments can be a useful way to minimize variance. Certainly, there is a time for fuss and for testing every string length under the sun, but when testing a specific case, keep the moving variables to a minimum and ramp them up as the application proves itself to be stable in simpler tests.
08 PREPARE FOR CHANGE
Some people assume that unit tests are whitebox, but any time you make changes to your application, you have lots of refactoring to do. Automated testing and automated test management will help you deal with the change. Do not fear it; expect it, prepare for it, and get it under control.
Who Can Do Test Automation?
Test automation requires software development skills. This is not a pursuit for unskilled workers, as has been thought in the past. The days of teams clicking on screens and recording their mouse movements are over. Test automation is a discipline akin to any other development or administrative role in software.
Testers require software development knowledge, scripting expertise, testing skills, and above all, excellent communication skills. Since they’ll be in charge of testing other people’s work, it’s key that they learn a common vocabulary for discussions and the naming of components and requirements.
Having a shared language makes it easy for everyone on the project team to stay in sync. DevOps is all about breaking down the silos between ops, development, and testing, so communication lines must flow just as easily as the software moving between these groups.
The tester’s job, in automating, is to shorten the feedback loop to the developer and to maintain those tests over time so that the loop remains short. This can be the most difficult part of the job, in fact, as tests break when changes are made. In the end, the only real solution is automated test maintenance using artificial intelligence and machine learning.
These, however, are the F-1 cars of testing. While they can help anyone go faster, ML- and AI-driven testing enable professionals to win races. And that’s the end goal of any software team these days: to beat the competition to implementing the latest feature.
The key to doing that is to ensure that your fast-moving development team, isn’t breaking the line of business applications in order to beat the other guy. And the only way to do that is to implement better testing through automation.
