KB Article T844058
Visible to All Users

EasyTest - How to integrate XAF functional testing with Continuous Integration systems like Azure DevOps

Our User Authentication and Group Authorization API for .NET Apps Powered by the XPO ORM repository includes an XAF application configured for Azure Pipelines - XafSolution. This article describes how to integrate EasyTest into an Azure Pipelines project's build process.
Refer to the azure-pipelines.yml file to get ready pipeline settings from the steps below.

Prerequisites

-  Your solution is published on GitHub.
-  The solution is ready for cloud build. Its NuGet.config references a DevExpress package source, as described in the Integrate NuGet to Popular Continuous Integration Systems topic.
-  The solution uses v19.2.5+.

Step 1 - Create a Pipeline

1.1. Create a new pipeline task in Azure DevOps Pipelines.
Clipboard-File-22.png

1.2. Connect to a GitHub repository where your XAF project is published.
Clipboard-File-3.png

1.3. Select the repository.
Clipboard-File-4.png

1.4. Choose the .NET Desktop template.
Clipboard-File-5.png

1.5. Click Save to commit the generated YAML configuration file to your repository.
Clipboard-File-14.png

Step 2 - Configure Pipeline TasksOpen the generated YAML file and follow the steps below.

2.1. Set the buildConfiguration variable to 'Debug' or 'EasyTest'.
Clipboard-File-18.png

2.2. If the NuGet.config file references your private DevExpress NuGet Feed API key, secure this key as described in the DevExpress NuGet - How to hide unique NuGet feed in Azure DevOps Build pipeline ticket.

2.3. Modify the YAML file's NuGetCommand task to use your repository's NuGet.config file to restore NuGet packages.
Clipboard-File-17.png

NOTE: In the text editor, click the Settings link to open the right panel with task settings.
Clipboard-File-21.png

If you have NuGet.config only for Azure pipelines, we recommend placing this file in the Functional Tests folder. In this case, this file will not affect your solution.

2.4. Add the following PowerShell task after the VSBuild task:

.Net Framework application:

Code
- task: PowerShell@2 displayName: EasyTest inputs: targetType: 'inline' script: | $easyTestPath="EasyTests" $targetFrameworkFilter="*\net452\*" sqllocaldb start MSSQLLocalDB Nuget install DevExpress.EasyTest.TestExecutor -OutputDirectory EasyTest -configfile NuGet.config -Framework net452 New-Item -ItemType directory -Path ".\EasyTest\Bin\" Get-ChildItem -Path "EasyTest\*" -Include *.dll,*.exe -Recurse | Where {$_.FullName -like $targetFrameworkFilter -or $_.FullName -like "*\any\any*"} | Copy-Item -Destination ".\EasyTest\Bin\" Copy-Item "C:\Program Files (x86)\Microsoft.NET\Primary Interop Assemblies\Microsoft.mshtml.dll" -Destination ".\EasyTest\Bin\" EasyTest\Bin\TestExecutor.v19.2.exe $easyTestPath Get-Content -Path $easyTestPath\TestsLog.xml if(Select-String -Pattern 'Result="Failed"', 'Result="Warning"' -Path $easyTestPath\TestsLog.xml) { exit 1 }

.Net Core application (starting with v22.2.7):

Code
- task: PowerShell@2 displayName: EasyTest inputs: targetType: 'inline' script: | $easyTestPath="EasyTests" $targetFrameworkFilter="*\net6.0\*" sqllocaldb start MSSQLLocalDB Nuget install DevExpress.EasyTest.TestExecutor -OutputDirectory EasyTest -configfile NuGet.config -Framework net6.0 New-Item -ItemType directory -Path ".\EasyTest\Bin\" Get-ChildItem -Path "EasyTest\*" -Include *.dll,*.exe,*.json,*.config -Recurse | Where {$_.FullName -like $targetFrameworkFilter -or $_.FullName -like "*\netstandard2.0\*" -or $_.FullName -like "\any\any*"} | Copy-Item -Destination ".\EasyTest\Bin\" Copy-Item "C:\Program Files (x86)\Microsoft.NET\Primary Interop Assemblies\Microsoft.mshtml.dll" -Destination ".\EasyTest\Bin\" EasyTest\Bin\TestExecutor.v22.2.exe $easyTestPath Get-Content -Path $easyTestPath\TestsLog.xml if(Select-String -Pattern 'Result="Failed"', 'Result="Warning"' -Path $easyTestPath\TestsLog.xml) { exit 1 }

In the copied script, specify the actual functional test folder from your GitHub repository and the actual filter expression in the 'easyTestPath' and 'targetFrameworkFilter' variables… This script does the following:
- Installs the EasyTest package.
- Starts Microsoft SQL Server (LocalDB).
- Copies the 'Microsoft.mshtml.dll' file to run functional tests in a Web application.
- Runs functional tests from EasyTest scripts.
- Displays the TestsLog.xml file in the Azure console.

NOTE: If you change the default NuGet.config file location, change the relative path to this file in the NuGetCommand task and PowerShell script.

2.5. Add a PublishPipelineArtifact task after the powerShell script:

Code
- task: PublishPipelineArtifact@1 condition: succeededOrFailed() inputs: targetPath: '.\EasyTests\' artifact: 'TestsResultFiles' publishLocation: 'pipeline'

This task publishes the functional test folder with test result files in the pipeline.
NOTE: Change the relative path from the 'targetPath' parameter according to the location of functional tests in your GitHub repository.

Step 3 - Run EasyTest Scripts and View ResultsThe

YAML template configuration file contains a trigger to run the pipeline when committing to the master branch. Specify your branch name instead of 'master' to run EasyTest when committing to this branch. For more information, see the Pipeline triggers article.
To run EasyTest functional tests manually, open your pipeline and click the Run pipeline button. In the pop-up window, select your branch and click the Run button.
Clipboard-File-16.png

Step 4 - View Test Results

4.1. Select the build and click the commit whose results you want to view.
Clipboard-File-8.png

4.2. Click the EasyTest error to view general information.
Clipboard-File-9.png
Clipboard-File-10.png

4.3. Click Job and the artifact link to download XML output logs and information about failed tests.
Clipboard-File-11.png
Clipboard-File-12.png

NOTE: Currently, you cannot use the Publish Test Result task with the test results file. The format of the test result file is not compatible with this task.

Step 5 - Learn More About Functional & Unit Testing in XAF

Functional Testing | EasyTest Basics | Script Reference
EasyTest - Syntax highlighting, collapsible regions, and code snippets in functional test files (*.etc, *.inc) opened with Notepad++ and Visual Studio Code
How to write unit tests for XAF Actions, Controllers and other custom UI logic
Unit and Functional Testing with Blazor UI

Show previous comments (9)

    I had a Could not load file or assembly 'System.Runtime, Version=5.0.0.0 error in the EasyTest task. The problem was caused by the PS line where the Get-ChildItem gets the exes and dlls from the downloaded DX nuget packages. Now there are other sub folders for the other target frameworks. So I swapped the -notlike with a -like since now there are net5.0-windows and netcoreapp3.0 dlls, if we want to avoid overwriting the net452 ones required for the TestExecutor.

    From:

    Code
    Get-ChildItem -Path "EasyTest\*" -Include *.dll,*.exe -Recurse | Where {$_.FullName -notlike "*\netstandard*\*"} | Copy-Item -Destination ".\EasyTest\Bin\"

    To:

    Code
    Get-ChildItem -Path "EasyTest\*" -Include *.dll,*.exe -Recurse | Where {$_.FullName -like "*\net452\*" -Or $_.FullName -like "*\any\*"} | Copy-Item -Destination ".\EasyTest\Bin\"

    It now works correctly.

    HTH,

    Alex

      Hello DevOps enthusiasts,

      This morning my CI pipeline failed for 2 reasons:

      1. DevExpress released 21.2.3 on the DevExpress feed and the PowerShell script to restore the TestExecutor package didn’t specify an explicit version. So, the latest TestExecutor package was installed, and it wasn’t compatible with my 21.1.6 solution.
      2. The 21.2.3 TestExecutor package now has a reference to the Selenium.WebDriver package which is not available in the DevExpress nuget feed.

      To solve #1 we could easily add the -version switch to nuget install, but this would mean maintenance to the pipeline task itself (the package version and the executable name). To avoid this, I decided to read the DevExpress version from the EasyTest config.xml and use it to restore and execute the appropriate TestExecutor package and executable. When we run the DX project converter, it updates the config.xml so this problem should be definitively solved.

      To solve #2, sadly, right now reusing the nuget authentication like we can with msbuild is not as straightforward (perhaps DX wants to take a look?). We can't re-use the nuget.config, because the script will timeout waiting for interactive authentication credentials (unless the credentials are stored in the nuget.config which we don't want). So the trick I used, is store the full DX nuget feed url in an Azure Key Vault secret and pass it to the script as an environment variable. And pass two -source parameters to the nuget install : one for the full DX feed uri and one for nuget.org so it can restore the selenium package.

      The yaml task added after the build:

      Code
      steps: - task: PowerShell@2 displayName: 'PowerShell Script' inputs: targetType: filePath filePath: ./src/GetAndRunTestExecutor.ps1 env: DevExpressNugetFeed: $(DevExpressNugetFeedFromVault) EasyTestPath: src\XafApp\XafApp.Module\FunctionalTests

      And the PowerShell script:

      Code
      Write-Host "Verifying script environment variables..." $devExpressNugetFeed = $Env:DevExpressNugetFeed if ([String]::IsNullOrWhiteSpace($devExpressNugetFeed)) { throw "Environment variable 'DevExpressNugetFeed' is empty. This should be the full uri including the api key (ex. https://nuget.devexpress.com/**********/api)." } $easyTestPath = $Env:EasyTestPath if ([String]::IsNullOrWhiteSpace($easyTestPath)) { throw "Environment variable 'EasyTestPath' is empty. This should be the relative path to the easy test folder containing script files (ex. src\Project\FunctionalTests)." } $easyTestConfigFile = $easyTestPath + "\config.xml" Write-Host "Reading DevExpress version from the EasyTest config.xml..." [XML]$EasyTestConfig = Get-Content $easyTestConfigFile $assemblyInfoNode = $EasyTestConfig.SelectNodes("//Options/Aliases/Alias[@Name='WinAdapterAssemblyName']") | select Value $versionProperty = $assemblyInfoNode.Value.Split(',')[1].Trim() if ($versionProperty -notlike "Version=*") { throw "Could not get the DevExpress version from the EasyTest config file. Make sure the EasyTest config file contains a WinAdapterAssemblyName or WebAdapterAssemblyName alias." } $devExpressVersion = [System.Version]::Parse($versionProperty.SubString("Version=".Length)) Write-Host "Found DevExpress version '$devExpressVersion' in EasyTest config file" Write-Host "Restoring TestExecutor Packages..." nuget.exe install DevExpress.EasyTest.TestExecutor -version $devExpressVersion -OutputDirectory EasyTest -source $devExpressNugetFeed -source "https://api.nuget.org/v3/index.json" $toolsPath = ".\EasyTest\Bin\" New-Item -ItemType directory -Path $toolsPath Get-ChildItem -Path "EasyTest\*" -Include *.dll,*.exe -Recurse | Where {$_.FullName -like "*\net452\*" -Or $_.FullName -like "*\any\*"} | Copy-Item -Destination $toolsPath Copy-Item "C:\Program Files (x86)\Microsoft.NET\Primary Interop Assemblies\Microsoft.mshtml.dll" -Destination $toolsPath $testExecutorFileName = "TestExecutor.v"+$devExpressVersion.ToString(2)+".exe" $testExecutorFullPath = $toolsPath+$testExecutorFileName if (!(Test-Path $testExecutorFullPath)) { throw "Executable '$testExecutorFullPath' doesn't exists." } sqllocaldb start MSSQLLocalDB Write-Host "Found and launching EasyTest TestExecutor '$testExecutorFullPath'" & $testExecutorFullPath $easyTestPath # Get the result and return exit code Get-Content -Path $easyTestPath\TestsLog.xml if(Select-String -Pattern 'Result="Failed"', 'Result="Warning"' -Path $easyTestPath\TestsLog.xml) { exit 1 }

      You might need to change WinAdapterAssemblyName for WebAdapterAssemblyName when getting the $assemblyInfoNode depending on which adapter you use.

      HTH someone else.

      Alex

      Andrey K (DevExpress Support) 3 years ago

        Hello Alex,

        Thank you for sharing this info with us. We will discuss how to update this article.

        Thanks,
        Andrey

        Disclaimer: The information provided on DevExpress.com and affiliated web properties (including the DevExpress Support Center) is provided "as is" without warranty of any kind. Developer Express Inc disclaims all warranties, either express or implied, including the warranties of merchantability and fitness for a particular purpose. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.

        Confidential Information: Developer Express Inc does not wish to receive, will not act to procure, nor will it solicit, confidential or proprietary materials and information from you through the DevExpress Support Center or its web properties. Any and all materials or information divulged during chats, email communications, online discussions, Support Center tickets, or made available to Developer Express Inc in any manner will be deemed NOT to be confidential by Developer Express Inc. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.