Test Studio Tests in Azure DevOps CI with Microsoft-Hosted Agent
Test Studio test lists can be executed in Chrome Headless mode within a Microsoft-hosted agent. This article guide you trough the process of setting up the Azure pipeline to run in a Microsoft-hosted agent.
Tip
Check out our blog post about Tesitng in Microsoft-hosted agents.
Environmental Prerequisites
The prerequisites from Test Studio and Azure DevOps side are listed below. Make sure that you comply with them, before proceeding with the next steps.
- Azure DevOps account enabled to configure Microsoft-hosted agent VMs with latest Chrome version.
- Active Test Studio Run-Time license and .msi installer for it. The minimum version of Test Studio is 2021 R3.
Disclaimer
Executing tests in a Microsoft-hosted agent has its specifics and limitations. It is better to know about them before you proceed.
- Tests in a Microsoft-hosted agent can be executed only in Headless browser mode, because the container does not have UI. This is the Chrome Headless browser type in Test Studio.
- The images on failure are not available in the results generated from the test runs in Microsoft-hosted agent.
What is Microsoft-Hosted Agent
The Azure pipelines can be configured to run their tasks with different type of agents. The Microsoft-hosted agent is one of these options - this is a VM created in the pipeline run-time and is based on OS image granted in Azure DevOps by Microsoft. The running VM has the Azure agent, which gets automatically connected to the Agent pool in the Azure project, latest Chrome and Edge browsers, Visual Studio and other tools, all set to run as Administrator by default. The management of this OS image is supported by the Microsoft Azure DevOps team and they commit it will be updated once per week.
Note
Microsoft-hosted agent VMs use no GUI session or UI. Therefore it can be used only for headless browser testing supported from Test Studio for the Chrome browser.
Setup Azure Pipeline to Execute Headless Tests on Microsoft-hosted Agent
Running Test Studio tests requires installation of the Test Studio Run-time edition - this is a separate product package dedicated for test execution only. One license for the Run-time edition is distributed with the Test Studio Ultimate bundle and, alternatively, it can be purchased separately.
Once you have an active Test Studio Run-time license, you can download its *.msi installer from your Telerik account here and add it in the Azure Pipeline as an Artifact related to the Azure project. That way it will be available for deployment on the Microsoft-hosted agent, when this is created.
Add Run-time Installer as Artifact
Open the Artifacts view in the Azure platform and choose the feed in which you want to add the Test Studio Run-time installer. Then select the Connect to feed option to add an artifact in the feed. If you need further information about Feeds in Azure DevOps, you can visit the Microsoft docs page on the topic.
Choose the Universal Packages option from the list of artifacts. The instructions how to publish a package are listed on the right side of the list. It is necessary to use the Azure CLI to login and publish the installer file - you can find example commands, which need to be adjusted for your project specifics. If you need further information about Publishing Universal Packages, you can visit the Microsoft docs page on the topic.
After the package is successfully uploaded, it is listed as an artifact and can be used in the Azure pipelines for that project.
Create a Free Form Pipeline
To create the Azure Pipeline, which will run tests in the MHA, you can start with the classic editor.
Choose the repository from which the Test Studio project will be deployed to the Microsoft-hosted agent machine. In this example I used the Azure Git repository. Be sure to use test lists, which are set to use the Chrome browser in headless mode for this pipeline test runs.
When the source location is defined, choose the Empty pipeline template and Apply to the pipeline.
Set the Pipeline to Use the Microsoft-hosted Agent Pool
Once the pipeline is created, you can change its name and set the Agent Pool to be used. For Microsoft-hosted agent you need to choose the Azure Pipelines pool. When this option is selected, you can select a default image to be installed on the VM when running the pipeline.
Download and Install the Test Studio Run-time on Microsoft-hosted Agent
You can use the Universal Package task to download the Test Studio Run-time installer artifact.
Ensure to list the feed, name and version of the artifact as you have entered these when publishing the package.
To trigger the installation of the downloaded package, add the next task and choose the Command Line option.
The installation can be triggered with a msiexec command in passive mode due to the lack of UI session. Find below an example command to use in the task.
msiexec.exe /i c:\TestStudio_Runtime_2021_3_1103_1.msi /passive /le c:\errorlog.txt
Execute the Tests on Microsoft-hosted Agent
The next task in the pipeline can be set to execute the tests. To trigger the test list run, you can again use a Command Line task. In it you can list a command using the options of the Test Studio CLI runner called ArtOfTest.Runner.exe.
The example in the screenshot uses the options to set custom name for the result file and to generate junit formatted result for the run. In this command I use the $(Build.Repository.LocalPath) variable provided from Azure. It is recommended to use such environmental variables as hard-coded paths may cause some false failures.
Another important setting for this task is to change its Control Options to "Continue on error". This way the test result publish tasks will get executed regardless of the test execution outcome.
Publish Results Generated on Microsoft-hosted Agent
It is useful to publish both the Test Studio results file and the junit formatted file. These require setup of separate tasks.
Create a Publish Pipeline Artifact task to upload the Test Studio results file as an artifact in the pipeline.
The junit formatted results can be published to the pipeline build summary with the Publish Test Results task.
For both these tasks you can use the variable $(Build.Repository.LocalPath).
Run the Azure Pipeline
With the pipeline set in this way, you can trigger the Test Studio headless test execution on a Microsoft-hosted agent VM.
In the Run Summary you can find useful details for the run and access the Test Studio result file for download.
In the Tests section you can see a summary of the test results based on the junit formatted results.
