New to Telerik Test Studio? Download free 30-day trial


The ArtOfTest.Runner.exe is installed in your Test Studio installation directory. The default location is:

  • C:\Program Files (x86)\Progress\Test Studio\Bin

When working with the execution engine in the command prompt always make sure you have started it as Administrator and have the root directory in context as shown below so that you can call the runner directly.


Help Screen

Here is the Help Screen for ArtOfTest.Runner.exe:


Execution Options

Execution options modify command line test execution behavior. The help screen (displayed by inputting ArtOfTest.Runner.exe -help from the command line) provides a brief explanation of each option. Here are some further notes about specific options.

Several execution options use file paths as values. Keep in mind the following notes when using file paths in the command prompt:

  • Enclose the file path in double quotes.
  • To prevent typing mistakes, use the clipboard paste option within the command prompt window. To find the paste option after copying the path to the clipboard, right mouse click on the top left corner of the command prompt and choose Edit > Paste.


File to Execute Options

  • The test option takes the full path to an individual test file with the .tstest file extension. This file will be located in your main project directory and under any folder structure you have created to organize your tests. You can use the test option by itself. By default results will be stored in the Results folder under the project’s root directory.

Test Execution Syntax
ArtOfTest.Runner Test Execution syntax
Test Execution Result
ArtOfTest.Runner Test Execution result
  • The list option takes the full path to a test list file with the .aiilist file extension. This option will execute a test list which has multiple tests included. All test lists for your project can be found in the test list folder under your project root folder. You can use the test option by itself. By default results will be stored in the Results folder under the project’s root directory. For more information about test lists, see Test Lists.

Test List Execution Syntax
ArtOfTest.Runner Test List Execution syntax
Test List Execution Result
ArtOfTest.Runner Test List Execution result
  • The root option takes full path to the project root folder.
  • The out option allows choice of an alternative folder to store the results to and takes a full path value to the respective folder.

  • The result option allows change of the default result file name and takes file name (including the file extension and in double quotes).

  • If any of the xml or html options are used the respective alternative result file will be stored to the default location if an output folder is not specified.

  • If any of the junit or junitstep options is used a JUnit xml result file will be generated. The difference will be respectively whether to convert a test or a test step to a junit test.

  • The PersistOnEachStep option could be set to true in case explicitly the results are required. This option will save the results after each executed step.

Settings Option

  • The settings option takes the full path to a JSON file containing custom settings for the run.

Below is an example of a complete JSON setting file that contains all of Telerik's test/test list run configuration settings. These are corresponding to the available test list settings.

  "Settings": {
    "__type": "ArtOfTest.WebAii.Core.Settings",
    "__value": {
      "Web": {
        "__type": "ArtOfTest.WebAii.Core.Settings+WebSettings",
        "__value": {
          "IsProfilingExecution": false,
          "ExecutingBrowsers": [],
          "UseMultiBrowserExecution": false,
          "RecycleBrowser": true,
          "AspNetDevServerPort": -1,
          "LocalWebServer": 0,
          "EnableUILessRequestViewing": false,
          "WebAppPhysicalPath": "",
          "DefaultBrowser": 2,
          "EnableScriptLogging": false,
          "BaseUrl": "",
          "KillBrowserProcessOnClose": false,
          "AutoCalibrateBrowsers": false,
          "UseHttpProxy": false,
          "EnableSilverlight": false,
          "VerboseHttpProxy": false,
          "SilverlightConnectTimeout": 60000,
          "SilverlightApplicationPath": null
      "Wpf": {
        "__type": "ArtOfTest.WebAii.Core.Settings+WpfSettings",
        "__value": {
          "DefaultApplicationPath": null
       "ResponsiveWeb": {
        "__type": "ArtOfTest.WebAii.Core.Settings+ResponsiveWeb",
        "__value": {
          "Width": 414,
          "Height": 896,
          "UserAgent": "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1. 38 (KHTML, like Gecko) Version/11.0 Mobile/15A356 Safari/604.1"         
      "CreateLogFile": true,
      "LogLocation": "C:\\\\WebAiiLog\\\\",
      "QueryEventLogErrorsOnExit": false,
      "LogAnnotations": true,
      "SimulatedMouseMoveSpeed": 0.3,
      "ExecuteInDevelopmentTests": false,
      "WaitCheckInterval": 500,
      "ElementWaitTimeout": 15000,
      "ExecuteCommandTimeout": 1000,
      "ExecutionDelay": 0,
      "UnexpectedDialogAction": 2,
      "AnnotateExecution": false,
      "AnnotationMode": 0,
      "XMultiMgr": true,
      "ClientReadyTimeout": 15000,
      "DisableDialogMonitoring": false,
      "RunnerResponseTimeout": 0.0
  "WebSettings": null,
  "PropertyBag": null

Publish Results To TFS

Note: All options listed in the help screen in relation to results publishing to TFS are mandatory. Below are some additional notes to each of the options.

  • The server option takes the TFS server name with its full path in double quotes like this "".

  • The build option takes the respective build which the results could be associated to.

  • The project option takes the name of the current team project which the build belongs to.

  • The platform option takes the respective platform which is usually Windows and thus the option takes value "win".

  • The flavor defines if the build is Debug or Release (defaults to "Debug").

Below is a sample command to publish results to TFS from a test list execution.

Publish Results to TFS

Exit Codes

ArtOfTest.Runner returns an exit code so the Build Server can check for it on process exit in case of an exception:

Code Title Summary
0 RUN_TESTS_SUCCESS Run is processed and all tests passed.
1 RUN_TESTS_ERROR Run is processed and some tests failed.
2 NOT_RUN_UNEXPECTED_ERROR Run is not processed due to unexpected error.
3 NOT_RUN_COMPILATION_ERROR Run is not processed due to compilation errors.
10 NOT_RUN_USAGE_INFO No run to process, requested usage info.
11 NOT_RUN_INVALID_COMMAND_ARGS No run to process due to general invalid arguments error.
12 NOT_RUN_TEST_NOT_FOUND Run is not processed due to invalid test path argument.
13 NOT_RUN_TESTLIST_NOT_FOUND Run is not processed due to invalid test list path argument.
In this article