Test Studio CLI Runner ArtOfTest.Runner.exe
The CLI runner ArtOfTest.Runner.exe is deployed with any of the Test Studio editions in the Test Studio installation directory. The default install folder where you can find the runner is C:\Program Files (x86)\Progress\Test Studio\Bin.
This article guides you trough the options of the Test Studio CLI Runner for building commands to execute tests.
- How to Call the ArtOfTest.Runner.exe in Command Prompt
- Help Screen
- Options to Specify Which File to Execute
- Options to Customize the Results File
- Option to Use Custom Settings for the Execution
How to Call the ArtOfTest.Runner.exe in Command Prompt
Similar to any other executable files you have few options to call the ArtOfTest.Runner.exe in the command prompt.
- Use the full file path to call the *.exe.
- Change the context root directory in the command prompt window to the one containing the *.exe file and use the *.exe name directly.
- Set the *.exe directory as environmental variable in the OS and use the *.exe name directly from whichever directory is the current context.
Help Screen
Use the help option to get a list of all options supported by the CLI runner. The short form to use is /? or /h. Below is the Help Screen for ArtOfTest.Runner.exe:
Options to Specify Which File to Execute
The ArtOfTest.Runner.exe supports the execution of a single test or a list of tests. Choose the option suitable for your needs.
test Option
The test option accepts full file path to an individual test with the *.tstest file extension.
Note!
Each Test Studio test is part of a project and it can't exist on its own. This is why running individual tests requires a project Settings.aiis file. So, you can use thetestoption by itself only when the test file is in the project root folder where the Settings.aiis file.
Otherwise, when the test is nested under any folder in the project use thetestoption in combination with therootone to specify the associated project root folder where the Settings.aiis file is.
Example Commands for test Option
- Run individual test stored in the project root folder:
> "C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe"
test="D:\Test Studio Projects\July2024\demoTest-AOTRunner.tstest"
- Run individual test stored in a sub-folder under the project root one:
>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe"
test="D:\Test Studio Projects\July2024\inProjectFolder\nestedDemoTest.tstest"
root="D:\Test Studio Projects\July2024"
list Option
The list option takes the full path to a test list file with the *.aiilist file extension.
Note!
Each Test Studio test list is part of a project and it can't exist on its own. Test list files are stored in the TestLists sub-folder under the project root one.
The *.aiilist file contains the test list settings applied for it in the Test Studio project, so you can use thelistoption by itself.
Example Commands for list Option
- Run test list:
>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe"
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist"
root Option
The root option takes the full path to the project root folder and is used in combination with either test, or list options.
Options to Customize the Results File
By default the result files generated from the CLI runner execution are output in the Results sub-folder under the project’s root directory.The ArtOfTest.Runner.exe supports outputting the results from a test/test list run in different formats and saving the file in custom location.
Tip
All result related options can be used in combination or on their own.
out Option
The out option defines an alternative folder to store the results to and takes the full path to the specified folder.
Example Commands for out Option
- Run test list outputting the results in specified folder:
>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe"
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist" out="D:\SharedFolder"
result Option
The result sets specific name for the default result file and takes a file name in double quotes.
Example Commands for result Option
- Run test list outputting the results in specified folder and setting custom name for the result file:
>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe"
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist" out="D:\SharedFolder"
result="customName"
xml or html Options
Use either the xml, or the html option to output the result into the corresponding file type - *.xml or *.html file. The different format result file is output in addition to the default result file with file extension *.aiiresult.
Note
xmlorhtmloption is valid only when executing a test list.
Example Commands for xml or html Options
- Run test list outputting the results in xml format:
>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe"
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist"
xml
- Run test list outputting the results in html format in specified folder and setting custom name for the result files:
>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe"
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist" out="D:\SharedFolder"
result="customName"
html
Note
Result files are saved into the project Results sub-folder if theoutoption is not used, and with default name if theresultoption is not used.
junit or junitstep Options
Use the junit or junitstep options to output the result into a junit(step).xml result file. The difference between the two is whether to convert a test or a test step to a junit test in the output result.
Example Commands for junit or junitstep Options
- Run test list outputting the results in junitstep format in specified folder and setting custom name for the result files:
>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe"
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist"
out="D:\SharedFolder"
result="customName"
junitstep
- Run single test outputting the results in junit format:
>>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe"
test="D:\elle\Test Studio Projects\July2024\demoTest-AOTRunner.tstest"
junit
Note
Result files are saved into the project Results sub-folder if theoutoption is not used, and with default name if theresultoption is not used.
persistOnEachStep Option
By default the PersistOnEachStep option is false. If set to true it saves the results after each executed step. It is used when debugging specific cases when the final result is not generated as expected.
Option to Use Custom Settings for the Execution
By default the CLI runner execution uses the settings applied in the Settings.aiis file for a single test run, or the settings specified in the executed test list. The ArtOfTest.Runner.exe supports customizing the execution by using custom settings file.
settings Option
The settings option takes the full path to a JSON file containing custom settings for the run.
Below is an example of a complete JSON setting file that contains all of Telerik's test/test list run configuration settings.
{
"Settings": {
"__type": "ArtOfTest.WebAii.Core.Settings",
"__value": {
"ResponsiveWeb": {
"__type": "ArtOfTest.WebAii.Core.Settings+ResponsiveWebSettings",
"__value": {
"Width": 0,
"Height": 0,
"UserAgent": null
}
},
"Web": {
"__type": "ArtOfTest.WebAii.Core.Settings+WebSettings",
"__value": {
"IsProfilingExecution": false,
"ExecutingBrowsers": [],
"UseMultiBrowserExecution": false,
"RecycleBrowser": false,
"AspNetDevServerPort": -1,
"LocalWebServer": 0,
"EnableUILessRequestViewing": false,
"WebAppPhysicalPath": "",
"DefaultBrowser": 10,
"EnableScriptLogging": false,
"BaseUrl": "",
"KillBrowserProcessOnClose": false,
"KillBrowsersBeforeStart": true,
"AutoCalibrateBrowsers": false,
"UseHttpProxy": false,
"EnableSilverlight": false,
"VerboseHttpProxy": false,
"SilverlightConnectTimeout": 60000,
"SilverlightApplicationPath": null,
"WebComponents": true,
"UseBrowserExtension": false,
"ChromiumFirstInteractionDelay": 500,
"FirefoxMinimumJSClickDelay": 100
}
},
"Wpf": {
"__type": "ArtOfTest.WebAii.Core.Settings+WpfSettings",
"__value": {
"DefaultApplicationPath": null,
"DefaultApplicationArgs": null,
"DefaultApplicationWorkingFolder": null
}
},
"Desktop": {
"__type": "ArtOfTest.WebAii.Core.Settings+DesktopSettings",
"__value": {
"DefaultApplicationPath": null,
"DefaultApplicationArgs": null,
"DefaultApplicationWorkingFolder": null
}
},
"CreateLogFile": true,
"LogLocation": "C:\\WebAiiLog\\",
"QueryEventLogErrorsOnExit": false,
"LogAnnotations": false,
"SimulatedMouseMoveSpeed": 0.3,
"ExecuteInDevelopmentTests": false,
"WaitCheckInterval": 500,
"ElementWaitTimeout": 15000,
"SearchByImageFirst": false,
"ExecuteCommandTimeout": 20000,
"ExecutionDelay": 0,
"RerunFailedTests": false,
"UnexpectedDialogAction": 2,
"AnnotateExecution": false,
"AnnotationMode": 0,
"RecordingMode": 0,
"RecordingCodec": 0,
"RecordingScale": 100,
"RecordingFPS": 5,
"RecordingOutputLocation": "",
"RecordingSizeLimit": 0,
"XMultiMgr": true,
"ClientReadyTimeout": 60000,
"DisableDialogMonitoring": false,
"RunnerResponseTimeout": 0.0,
"EnableImageSearch": true,
"ElementImageSearchTimeout": 15000,
"ElementImageSearchDelay": 300,
"ScrollOnImageSearch": true,
"TelerikComponentsVersion": 0
}
},
"WebSettings": null,
"PropertyBag": null
}
Tip
The settings listed in the settings file correspond to these available in the test list settings.
Publish Results To TFS
Note: All options listed in the help screen in relation to results publishing to TFS are mandatory. Below are some additional notes to each of the options.
The server option takes the TFS server name with its full path in double quotes like this http://myTFS.myDomain.com:8080/tfs.
The build option takes the respective build which the results could be associated to.
The project option takes the name of the current team project which the build belongs to.
The platform option takes the respective platform which is usually Windows and thus the option takes value "win".
The flavor defines if the build is Debug or Release (defaults to "Debug").
Below is a sample command to publish results to TFS from a test list execution.
Exit Codes
ArtOfTest.Runner.exe returns an exit code so the Build Server can check for it on process exit in case of an exception:
| Code | Title | Summary |
|---|---|---|
| 0 | RUN_TESTS_SUCCESS | Run is processed and all tests passed. |
| 1 | RUN_TESTS_ERROR | Run is processed and some tests failed. |
| 2 | NOT_RUN_UNEXPECTED_ERROR | Run is not processed due to unexpected error. |
| 3 | NOT_RUN_COMPILATION_ERROR | Run is not processed due to compilation errors. |
| 10 | NOT_RUN_USAGE_INFO | No run to process, requested usage info. |
| 11 | NOT_RUN_INVALID_COMMAND_ARGS | No run to process due to general invalid arguments error. |
| 12 | NOT_RUN_TEST_NOT_FOUND | Run is not processed due to invalid test path argument. |
| 13 | NOT_RUN_TESTLIST_NOT_FOUND | Run is not processed due to invalid test list path argument. |