It’s official: automated testing has won. Not that long ago, it wasn’t hard to find developers and other tech workers who were either skeptics of automated testing or totally unaware of its existence. Sure, there are still skeptics among us, but their numbers have been dramatically reduced. Automated testing is now an everyday reality in many software organizations. Tools for performing automated testing have exploded, as well as the number of types of automated testing. Today we’re going to talk about a super-specific category of automated testing: Artillery load testing.
By “Artillery” load testing, we mean load testing using Artillery. You might be wondering what “Artillery” is, and also “load testing.” In this post, we’ll answer those questions and more.
The short answer is that “load testing” is a type of automated testing, and that “Artillery” is a testing tool you can use to perform load testing. I’m afraid that isn’t the most useful of answers, though. It doesn’t address in detail what load testing really is, how it works, and what its benefits are. It also doesn’t do anything to explain what Artillery is, how to download and install it, and so on. Do you want to know the answer to those questions as well?
If your answer is “yes,” then keep reading this post. We’ll start by defining load testing, explaining what it is, and what benefits this testing technique provides. Then, we’ll go into Artillery itself. We’ll explain what it is, proceeding to show its installation. (Spoiler: it’s super easy!)
We then wrap up with a simple demo showing how you can write and run your first Artillery load test.
Load Testing Basics
We’ll start by offering a general overview of load testing. You’ll learn what this type of testing is about, how it differs from other types of testing, and how you can profit from it. Let’s get started.
Defining Load Testing
The first thing you need to understand about load testing is that it’s a particular type of performance testing. And what is performance testing? Let’s see Wikipedia’s definition:
In software quality assurance, performance testing is, in general, a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage.
With this definition, we can see that performance testing is a peculiar type of testing, in that it’s not meant to help you find bugs like most other types of testing. Rather, its goal is to verify whether the system can perform well under a certain load. This fact has a serious implication: performance testing isn’t interchangeable with other types of testing. When it comes to other categories of automated testing, there is a certain degree of overlap between them. It’s possible that a given bug is detected both by your unit and your integration tests, for example. That doesn’t apply to performance testing: other types of testing won’t usually detect stability problems.
And what about load testing? As we’ve said, load testing is a specific type of performance testing. You conduct load testing to check how your application behaves under a certain load. You’d usually perform load testing to verify your app’s behavior not only under normal expected conditions but also on peaks. That way, load testing is essential to verify the reliability and stability of your application, and also whether it’s capable of scaling.
Why Should You Care About Load Testing
Web applications often struggle with downtime when the traffic they receive increases too rapidly. Even popular, well-maintained sites are susceptible to this problem.
Of course, you don’t want your applications to suffer from this problem. One hour of downtime might come at a huge cost for any reasonably large website. So you have to make sure your application is ready to handle an increase in traffic demand, and that’s where load testing comes in handy.
This type of testing enables developers, testers, and other tech workers to ensure their applications are reliable. Load testing uses test scenarios featuring increased traffic before such an increase happens in production in real life. That way, it becomes easier to detect and fix potential bottlenecks, way before they have the opportunity to cause downtime.
Another reason why load testing is crucial is what you’ve just read in the previous section. Namely, that load testing—and performance testing in general—can’t be replaced with other types of testing, since it offers a particular kind of feedback that no other type of testing does.
Getting Started With Artillery Load Testing: The Simplest Possible Guide
We’re now ready to start covering Artillery. We’ll give you a quick and simple guide on this framework. You’ll learn what it’s about, its features, how to install it and get started using it.
What Is Artillery?
Artillery is an open-source testing toolkit, written in Node.js, that can help you with load and functional testing. Interesting, but what does it actually do? In a nutshell, Artillery is a load generator. That is, it sends a lot of requests—i.e., the load—to the server you specify, very quickly.
What follows is a non-exhaustive list of Artillery features:
- Easy installation with npm. Since Artillery is a Node.js tool, you can easily install it using the npm utility.
- Simple CLI interface. Unlike other load testing tools that have clunky GUI, Artillery’s interface is a simple CLI.
- Support to HTTP, Socket.io, WebSockets, and AWS Kinesis out of the box. Artillery allows you to test virtually any back-end service.
- Easy Usage. Since you’d use Artillery to describe Artillery’s test scenarios, it’s easy to learn, even for non-technical people.
- Easy Automation. You can easily integrate Artillery’s CLI into other scripts and CI solutions.
- Easy Extension. You can extend Artillery by writing custom engines, plugins, or custom reporters using JavaScript.
Installing Artillery
Now you’ll see how to install Artillery, and it’s going to be the quickest section in the whole post. Yep, it’s that easy.
As we’ve said before, Artillery uses npm for installation, which means you must have Node.js installed. If you use Linux, just google how to install Node.js using the preferred way of your distro (e.g., apt-get on Debian/Ubuntu). On the other hand, if you’re a Windows or Mac OS person, go to the download page to get an installer for your platform.
After you’ve installed Node.js, you’ll be able to use npm to install Artillery, by running the following command:
npm install -g artillery
You can then check whether the installation was successful by running the following command:
artillery -V
If everything went well, Artillery would display its version number.
The Quickest Possible Artillery Demo
Now you’re ready to start using Artillery. We’ll show you the quickest possible test you can do, by running the following command:
artillery quick --count 10 -n 20 https://artillery.io/
The “quick” option allows you to run a simple, quick test without having to create a config file. But what does the line above actually do?
The “count” option allows you to create a certain number of “virtual users” ten, in our example. With the “-n” option, we define the number of requests each virtual user is sending. So, we’ll have ten virtual users, sending 20 HTTP GET requests to the specified URL.
I’ve just run that command, and that’s the result Artillery showed me:
Started phase 0, duration: 1s @ 08:53:00(-0300) 2019-10-28
Report @ 08:53:06(-0300) 2019-10-28
Elapsed time: 5 seconds
Scenarios launched: 10
Scenarios completed: 10
Requests completed: 200
RPS sent: 37.04
Request latency:
min: 135.8
max: 593.2
median: 137.9
p95: 367.8
p99: 586.7
Codes:
200: 200
All virtual users finished
Summary report @ 08:53:06(-0300) 2019-10-28
Scenarios launched: 10
Scenarios completed: 10
Requests completed: 200
RPS sent: 36.9
Request latency:
min: 135.8
max: 593.2
median: 137.9
p95: 367.8
p99: 586.7
Scenario counts:
0: 10 (100%)
Codes:
200: 200
The next step would be learning how to run a test using a YAML configuration file.
Use Artillery In Your CI/CD Environment
With any type of software test, you want to ensure they’re included in your CI/CD pipeline, so you don’t ship code to production if it doesn’t meet your quality criteria. This is true for unit testing, linters, and load testing is certainly no exception.
So, how would you include Artillery as a quality gate to your pipeline? In short, you should define performance expectations that, if not met, cause the pipeline to stop. When configuring Artillery, there’s a property you can use for that: config.ensure.
With this property, you can verify, for instance, that a given percentage of requests have a response time equal to or less than a threshold you define. If this condition isn’t met, the script fails, interrupting the pipeline and preventing the deployment of the non-performant code.
Here is an example straight from Artillery’s documentation:
config:
ensure:
thresholds:
# p99 of response time must be <250:
- "http.response_time.p99": 250
# p95 of response time must be <100:
- "http.response_time.p95": 100
Artillery Load Testing: Expect the Unexpected, and Test for It
When you have a web application that’s vital for your business, you can’t afford to risk downtime. Unfortunately, from time to time, you’ll suffer from those, even with your best efforts. But you have to use every tool that’s within your reach to prevent such downtimes. For instance, if you have an e-commerce, then you must be prepared for the peaks during the holiday season.
One of the best ways to make sure your application can handle such peaks is by employing performance testing, especially the load testing variety. In today’s post, we’ve offered you an introduction to load testing, explaining what the technique is and why it’s important. We’ve also given you a brief starting quite to Artillery, which is a tool you can use to perform load testing.
Thanks for reading. And happy testing!
This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.
