Month: May 2023

Stay awhile and listen… load testing is a crucial step in ensuring the performance and scalability of REST-like web services. Artillery is a popular open-source load testing tool that allows you to simulate high user traffic and measure the system’s response under different loads. In this guide, we’ll walk through the process of creating an artillery load test for a REST service. You find the Artillery docs here.

---

Key Concepts

• Test Script: Artillery test scripts or test definitions are usually written as YAML (with the option for near-infinite customization with Node.js code, which may use any public or private npm packages). A test definition can have one or more scenarios.
• Virtual Users: A virtual user emulates a real user interacting with your app. A “real user” may be a person, or an API client depending on your app. A virtual user executes a scenario. Every virtual user running the scenario is completely independent of other virtual users running the same scenario, just like users and API clients in the real world. Virtual users do not share memory, network connections, cookies, or other state.
• Scenarios: Each scenario is a series of steps representing a typical sequence of requests or messages sent by a user of an application. Usually is a sequence of actions, such as making a HTTP GET request, followed by a HTTP POST request and so on.
• Phases: A load phase tells Artillery how many virtual users to create over a period of time. A production-grade load test will usually have multiple load phases, such as: A warm up phase, which will gently increase load over a period of time; One or more heavy load phases; Load phases are expressed as duration + an arrival rate. Each arrival is a new virtual user.

---

Installing Artillery

Node.js is a prerequisite for Artillery, you can find the installation procedure here.
To install Artillery globally open a terminal or command prompt and run the following command:

npm install -g artillery

To test the installation run:

artillery

You should get something like this

You can test the installation running the dino script:

artillery dino -m "Kernel Panic!" -r

And you should get a nice dinosaur like this one:

---

Test Script

Artillery use YAML files to define test scripts and its scenarios. Create a new file with a .yml extension (e.g., asciiart-load-test.yml).
In this file we have some root properties like config, scenarios, plugins and many others.

• config defines how the load test runs, e.g. the URL of the system we’re testing, generated load, any plugins we want to use, and so on.
• scenarios is where we define what the virtual users created by Artillery will do. A scenario is usually a sequence of steps that describes a user session in the app.
• plugins is where you define the plugins. there are build-in and third party plugins available

At the config section you should define the target url, this means that all requests will use that base URL by default.

target: "http://asciiart.artillery.io:8080"

Also at the config section you’ll need to define the phases. Load phases tell Artillery how many virtual users to create, and describe the shape of the load we want.

phases:
  - name: Warm up the API
    duration: 60
    arrivalRate: 5
    rampTo: 10
  - name: Ramp up to peak load
    duration: 60
    arrivalRate: 10
    rampTo: 50
  - name: Sustained peak load
    duration: 300
    rampTo: 50

In this test we’ve defined three distict phases:

• Warm up the API – this phase will run for 60 seconds. Artillery will start by creating 5 new virtual users per second, and gradually ramp up to 10 new virtual users per second by the end of the phase.
• Ramp up to peak load – this phase will also last for 60 seconds. Artillery will continue ramping up load from 10 to 50 virtual users per second.
• Sustained peak load – this phase will run for 300 seconds. Artillery will create 50 new virtual users every second during this phase.

Now we’ll define our scenarios.

scenarios:
  - name: Get 3 animal pictures
    flow:
      - get:
          url: "/dino"
      - get:
          url: "/armadillo"
      - get:
          url: "/pony"

Artillery creates a virtual user that will execute this scenario with three actions, in this test.

---

Running the Test

Execute the following command to run the test:

artillery run asciiart-load-test.yml

Artillery will run the test as described in the asciiart-load-test.yml. As virtual users run their scenario and collect performance metrics, Artillery will print a report every 10 seconds with a summary of collected metrics for that time period.

The output will look similar to this, with reports describing the number of virtual users created, HTTP response codes, and response times from API endpoints we’re testing:

---

Reports

Artillery can create nice HTML reports. Add the parameter --output asciiart-load-test.json to the run command to generate such reports and when the test is finished, you have to execute:

artillery report asciiart-load-test.json

This will generate a file called asciiart-load-test.json.html which you can browse. The output contains a summary and many charts like the http.codes.200 and http.response_time, shown bellow.

---

Example of Functional YAML File

config:
  target: "http://asciiart.artillery.io:8080"
  phases:
    - duration: 10
      arrivalRate: 5
      rampTo: 10
    - duration: 20
      arrivalRate: 10
      rampTo: 20
    - duration: 40
      rampTo: 20
scenarios:
  - name: Get 3 animal pictures
    flow:
      - get:
          url: "/dino"
      - get:
          url: "/armadillo"
      - get:
          url: "/pony"

---

Example of Functional YAML File

Sometimes we need to execute routines before run the test or maybe generate some random data to use during test. To achieve this task we can extend our test script with external custom javascript functions.

Now we create a javascript file called testUtils.js and write our functions in it.

I had to generate a random number with a specific format, and send it on a POST request for a particular test I was running so I created the following functions.

function generateNumber() {
    let c = 8;
    let l = ['55', '00', '9'];
    while(c) {
        l.push(`${Math.floor(Math.random() * 10)}`);
        c--;
    }
    return `${l.join('')}`;
}

function modifyRequestBody(requestParams, context, ee, next) {
    requestParams.json.myRandomNumber = generateNumber();
    return next();
}

module.exports = { modifyRequestBody }

To use this function we need to configure the property config.processor with the path of a CommonJS module which will be require()d and made available to scenarios.

config:
  target: "https://myservice:44334"
  phases:
    - duration: 10
      arrivalRate: 5
      name: Warm up
    - duration: 30
      arrivalRate: 5
      rampTo: 80
      name: Ramp up load
    - duration: 60
      arrivalRate: 80
      name: Sustained load
  processor: "./testUtils.js"

After that, you can use the functions inside the scenario flow, in one of the available hooks which are: beforeScenario, afterScenario, beforeRequest and afterResponse.

At this particular example, I called the custom function modifyRequestBody on the beforeRequest hook

scenarios:
  - name: "simple POST test"
    flow:
      - post:
          url: "/myapicontroller/mypostaction"
          headers:
            Content-Type: "application/json"
            api-token: "mysecretkey"
          json:
            myOrigin: "Artillery Test"
            myRandomNumber: "modified by the custom function"
          beforeRequest: "modifyRequestBody"
          capture:
            - json: "$.success"
              as: "success"
            - json: "&.message"
              as: "message"
              strict: false

Capture Property

Use the capture property to retain data of the request. The data can be used in any other request of the same virtual user.

The capture option must always have an as attribute, which names the value for use in subsequent requests. It also requires one of the following attributes:

• json – Allows you to define a JSONPath(opens in a new tab) expression.
• xpath – Allows you to define an XPath(opens in a new tab) expression.
• regexp – Allows you to define a regular expression that gets passed to a RegExp constructor(opens in a new tab). A specific capturing group(opens in a new tab) to return may be set with the group attribute (set to an integer index of the group in the regular expression). Flags(opens in a new tab) for the regular expression may be set with the flags attribute.
• header – Allows you to set the name of the response header whose value you want to capture.
• selector – Allows you to define a Cheerio(opens in a new tab) element selector. The attr attribute will contain the name of the attribute whose value we want. An optional index attribute may be set to a number to grab an element matching a selector at the specified index, “random” to grab an element at random, or “last” to grab the last matching element. If the index attribute is not specified, the first matching element will get captured.

By default, captures are strict. If a capture rule fails because nothing matches, any subsequent requests in the scenario will not run, and that virtual user will stop making requests. This behavior is the default since subsequent requests typically depend on captured values and fail when one is not available.

In some cases, it may be useful to turn this behavior off. To make virtual users continue with running requests even after a failed capture, set strict to false:

- get:
    url: "/"
    capture:
      json: "$.id"
      as: "id"
      strict: false # We don't mind if `id` can't be captured and the next requests 404s
- get:
    url: "/things/{{ id }}"

---

Multiple Environments

Typically, you may want to reuse a load testing script across multiple environments with minor tweaks. For instance, you may want to run the same performance tests in development, staging, and production. However, for each environment, you need to set a different target and modify the load phases.

Instead of duplicating your test definition files for each environment, you can use the config.environments setting. It allows you to specify the number of named environments that you can define with environment-specific configuration.

A typical use-case is to define multiple targets with different load phase definitions for each of those systems:

config:
  target: "http://service1.acme.corp:3003"
  phases:
    - duration: 10
      arrivalRate: 1
  environments:
    production:
      target: "http://service1.prod.acme.corp:44321"
      phases:
        - duration: 1200
          arrivalRate: 10
    local:
      target: "http://127.0.0.1:3003"
      phases:
        - duration: 1200
          arrivalRate: 20

When running your performance test, you can specify the environment on the command line using the -e flag. For example, to execute the example test script defined above with the staging configuration:

artillery run -e staging my-script.yml

When running your tests in a specific environment, you can access the name of the current environment using the $environment variable.

For example, you can print the name of the current environment from a scenario during test execution:

config:
  environments:
    local:
      target: "http://127.0.0.1:3003"
      phases:
        - duration: 120
          arrivalRate: 20
scenarios:
  - flow:
      - log: "Current environment is set to: {{ $environment }}"

Plugins

Artillery has support for plugins, which can add functionality and extend its built-in features. Plugins can hook into Artillery’s internal APIs and extend its behavior with new capabilities.

Plugins are distributed as normal npm packages which are named with an artillery-plugin- prefix, e.g. artillery-plugin-expect.

You can use built-in plugins like expect to check assertions on HTTP response.

To enable expect, add it to the config section:

config:
  target: "https://myservice:44334"
  plugins:
    expect: {} //usually you can pass parameters in braces. expect do not require any parameter

Adding expectations to HTTP response:

scenarios:
  - name: Get a movie
    flow:
      - get:
          url: "/movies/5"
          capture:
            - json: "$.title"
              as: title
          expect:
            - statusCode: 200
            - contentType: json
            - hasProperty: title
            - equals:
                - "From Dusk Till Dawn"
                - "{{ title }}"

This is a simple example, but Artillery has great powers and you can easely extend it! Check out the documentation.

Artillery can be used together with some trace application like newrelic, datadog or dynatrace to deal with bottlenecks and other performance issues.

Conclusion

In conclusion, the artillery test tool proves to be an invaluable asset in the world of software development and performance testing. Its versatility, ease of use, and powerful features make it an essential tool for any development team looking to ensure the reliability and scalability of their applications.

By simulating realistic user traffic and behavior, the artillery test tool provides insights of application’s performance under various scenarios. Working together with a trace service like newrelic, datadog or dynatrace allows to identify and rectify potential bottlenecks, optimize resource allocation, and fine-tune code for optimal efficiency.

See ya!