Test Studio Tests and Azure DevOps Build Pipeline

Test Studio tests can be successfully integrated for execution with the Azure DevOps Build pipelines. Below you can read details for the prerequisites and necessary configurations to apply.

Environmental Prerequisites

The build machine can be any virtual or physical machine to host the Test Studio Runner. Please refer to the system requirements for the required machine configuration.

The components to be installed on the machine:

  • Test Studio Run-time Edition (ArtOfTest.Runner.exe must be available): mandatory
  • Internet Browser (IE latest, Chrome latest and Firefox latest, Edge latest): mandatory
  • Visual Studio 2019 Professional: optional, required only for additional functions like build tasks and other non-directly related to running tests with Test Studio

Configuring the Azure DevOps Build Pipeline

Below are listed the steps to configure the build pipeline in Azure DevOps.

Create a new project in Azure DevOps

Start with creating a new project in Azure. The one used in this example is named Test Studio 1.

Create New project

Create a Self-hosted Agent

Depending on whether tests should be running as a part of the app under test deployment to Live/Test environments or as part of daily builds there are two types of agents. If tests are supposed to run as a part of the release, the agent executing the tests should be registered in a Deployment group. If tests are supposed to run on demand after each build of the source code, the agent executing the tests should be registered to an Agent pool.

The tests should be run using self-hosted Windows agents in order to be able to use the corresponding Test Studio installation.

To create such agent, follow the steps listed below

1.  Open the current Project settings.

Open the current project settings

2.  Click on Pipelines -> Agent Pools.

Agent Pools

3.  Add a new pool or open the default one.

New Agent Pools

4.  Click New Agent,

New Agent

5.  Download the Agent to the corresponding physical or virtual machine as per the installation requirements above.

6.  Extract the Agent and run as Administrator.

Start Agent

Note! In order to avoid any permissions issues, please make sure that the user configuring the Agent is part of the following Permission Groups: Build Administrators, Release Administrators, Project Administrators, ProjectName Team.

Configure a Self-hosted Agent

1.  Enter server URL: i.e. https://dev.azure.com/OrganizationName.

2.  Create Personal Access Token (PAT) as described here and save it.

3.  Choose PAT for authentication type and paste yours.

4.  Choose the Agent pool you created earlier.

5.  Choose Agent name, the default name is the machine name on which it is hosted.

6.  Choose a work folder or press enter for the default one _work sub-folder.

7.  Select N when asked to enter agent as a service.

8.  Enter configure autologon and run agent on startup Y/N is optional.

Configure Agent

9.  Upon successful configuration the config console will close, so execute run.cmd in console to trigger the Agent.

Trigger Agent

Run Tests on Agent

1.  Go to Pipelines >> Builds >> Create new.

Create Pipeline

2.  It is important to select Use the classic editor in order to create a pipeline without YAML.

Create Pipeline

3.  Choose the repository where your code is and click Continue.

Choose repo

4.  Choose Empty job as a template for the build

Choose empty job

5.  Select a pipeline and choose the Agent pool configured earlier.

Select pipeline and agent pool

6.  Create a new task on the Agent -> Utility -> Command line.

Command line new task

7.  Modify the Command line script, so that it executes tests using the CLI Test Studio Runner and make sure you add the junit option to the CLI test runner in order to output the results in junit format (junit is supported by the publish results task). The script points to the location of the ArtOfTest.Runner.exe combined with the desired test or test list (see the syntax options here). In the example below, we are executing a test list.

Command line script

8.  Select Continue on error from the Command line script Control Options, if you’d like to complete a results' upload task (see next list point) even if the test execution task fails.

Control Options

9.  To add a Publish Test Result task, select the folder to which the test results are deployed and choose to upload all .xml files **.xml for file search for example.

Publish Test results task

Something to consider here is whether results are piled together in this folder, if that’s the case there may be a need to make sure the results folder is empty before each run. If the folder is part of the build folders managed by the Azure agent, then this is not a concern because they are recreated for each run.

Publish Test results

Results from Test Execution

Once the test run finishes, you can see the results in the Test tab.

See test results

In this article
Not finding the help you need? Improve this article