Our User Authentication and Group Authorization API for .NET Apps Powered by the XPO ORM repository includes an XAF application configured for Azure Pipelines - XafSolution. This article describes how to integrate EasyTest into an Azure Pipelines project's build process.
Refer to the azure-pipelines.yml file to get ready pipeline settings from the steps below.
Prerequisites
- Your solution is published on GitHub.
- The solution is ready for cloud build. Its NuGet.config references a DevExpress package source, as described in the Integrate NuGet to Popular Continuous Integration Systems topic.
- The solution uses v19.2.5+.
Step 1 - Create a Pipeline
1.1. Create a new pipeline task in Azure DevOps Pipelines.
1.2. Connect to a GitHub repository where your XAF project is published.
1.3. Select the repository.
1.4. Choose the .NET Desktop template.
1.5. Click Save to commit the generated YAML configuration file to your repository.
Step 2 - Configure Pipeline TasksOpen the generated YAML file and follow the steps below.
2.1. Set the buildConfiguration variable to 'Debug' or 'EasyTest'.
2.2. If the NuGet.config file references your private DevExpress NuGet Feed API key, secure this key as described in the DevExpress NuGet - How to hide unique NuGet feed in Azure DevOps Build pipeline ticket.
2.3. Modify the YAML file's NuGetCommand task to use your repository's NuGet.config file to restore NuGet packages.
NOTE: In the text editor, click the Settings link to open the right panel with task settings.
If you have NuGet.config only for Azure pipelines, we recommend placing this file in the Functional Tests folder. In this case, this file will not affect your solution.
2.4. Add the following PowerShell task after the VSBuild task:
.Net Framework application:
Code- task: PowerShell@2 displayName: EasyTest inputs: targetType: 'inline' script: | $easyTestPath="EasyTests" $targetFrameworkFilter="*\net452\*" sqllocaldb start MSSQLLocalDB Nuget install DevExpress.EasyTest.TestExecutor -OutputDirectory EasyTest -configfile NuGet.config -Framework net452 New-Item -ItemType directory -Path ".\EasyTest\Bin\" Get-ChildItem -Path "EasyTest\*" -Include *.dll,*.exe -Recurse | Where {$_.FullName -like $targetFrameworkFilter -or $_.FullName -like "*\any\any*"} | Copy-Item -Destination ".\EasyTest\Bin\" Copy-Item "C:\Program Files (x86)\Microsoft.NET\Primary Interop Assemblies\Microsoft.mshtml.dll" -Destination ".\EasyTest\Bin\" EasyTest\Bin\TestExecutor.v19.2.exe $easyTestPath Get-Content -Path $easyTestPath\TestsLog.xml if(Select-String -Pattern 'Result="Failed"', 'Result="Warning"' -Path $easyTestPath\TestsLog.xml) { exit 1 }
.Net Core application (starting with v22.2.7):
Code- task: PowerShell@2 displayName: EasyTest inputs: targetType: 'inline' script: | $easyTestPath="EasyTests" $targetFrameworkFilter="*\net6.0\*" sqllocaldb start MSSQLLocalDB Nuget install DevExpress.EasyTest.TestExecutor -OutputDirectory EasyTest -configfile NuGet.config -Framework net6.0 New-Item -ItemType directory -Path ".\EasyTest\Bin\" Get-ChildItem -Path "EasyTest\*" -Include *.dll,*.exe,*.json,*.config -Recurse | Where {$_.FullName -like $targetFrameworkFilter -or $_.FullName -like "*\netstandard2.0\*" -or $_.FullName -like "\any\any*"} | Copy-Item -Destination ".\EasyTest\Bin\" Copy-Item "C:\Program Files (x86)\Microsoft.NET\Primary Interop Assemblies\Microsoft.mshtml.dll" -Destination ".\EasyTest\Bin\" EasyTest\Bin\TestExecutor.v22.2.exe $easyTestPath Get-Content -Path $easyTestPath\TestsLog.xml if(Select-String -Pattern 'Result="Failed"', 'Result="Warning"' -Path $easyTestPath\TestsLog.xml) { exit 1 }
In the copied script, specify the actual functional test folder from your GitHub repository and the actual filter expression in the 'easyTestPath' and 'targetFrameworkFilter' variables… This script does the following:
- Installs the EasyTest package.
- Starts Microsoft SQL Server (LocalDB).
- Copies the 'Microsoft.mshtml.dll' file to run functional tests in a Web application.
- Runs functional tests from EasyTest scripts.
- Displays the TestsLog.xml file in the Azure console.
NOTE: If you change the default NuGet.config file location, change the relative path to this file in the NuGetCommand task and PowerShell script.
2.5. Add a PublishPipelineArtifact task after the powerShell script:
Code- task: PublishPipelineArtifact@1
condition: succeededOrFailed()
inputs:
targetPath: '.\EasyTests\'
artifact: 'TestsResultFiles'
publishLocation: 'pipeline'
This task publishes the functional test folder with test result files in the pipeline.
NOTE: Change the relative path from the 'targetPath' parameter according to the location of functional tests in your GitHub repository.
Step 3 - Run EasyTest Scripts and View ResultsThe
YAML template configuration file contains a trigger to run the pipeline when committing to the master branch. Specify your branch name instead of 'master' to run EasyTest when committing to this branch. For more information, see the Pipeline triggers article.
To run EasyTest functional tests manually, open your pipeline and click the Run pipeline button. In the pop-up window, select your branch and click the Run button.
Step 4 - View Test Results
4.1. Select the build and click the commit whose results you want to view.
4.2. Click the EasyTest error to view general information.
4.3. Click Job and the artifact link to download XML output logs and information about failed tests.
NOTE: Currently, you cannot use the Publish Test Result task with the test results file. The format of the test result file is not compatible with this task.
Step 5 - Learn More About Functional & Unit Testing in XAF
Functional Testing | EasyTest Basics | Script Reference
EasyTest - Syntax highlighting, collapsible regions, and code snippets in functional test files (*.etc, *.inc) opened with Notepad++ and Visual Studio Code
How to write unit tests for XAF Actions, Controllers and other custom UI logic
Unit and Functional Testing with Blazor UI
Absolutely the documentation I've been looking for :) Thx a lot.
One minor Request: I cannot access some of the tickets mentioned in Step 5. Are the links correct?
Regards,
Oliver
@Oliver: Thank you for your feedback. We would greatly appreciate it if you share your experience with EasyTest and Azure DevOps once you have had an opportunity to implement these instructions.
The "How to write unit tests for XAF Actions, Controllers and other custom UI logic" link is indeed not ready yet (we overlooked it). Please stay tuned as 10 articles on unit testing are coming soon.
Since I'm currently in the middle of performing EasyTest functional testing in an Azure DevOps pipeline I wanted to chime in. If you're in the same situation as me and are running your own build agent on your own VM you need to configure it to not run as a windows service (which is the default configuration).
Instead you need to run it as "auto-logon" during startup of the machine because EasyTest requires UI access (and a service does not have that capability or at least not completely). If you say "No" to the "run as service" option you will be offered the one to run it at logon instead. For that you also need to run it as a windows user, so I suggest you create a separate user account for the agent. Personally, I've added that user to the Administrator group, since that is the way Microsoft's own build agents are configured as well.
For completeness sake: If you run it as a service you might be greeted with something like this message which for me at least happened immediately at the login screen and prevents the successful execution of the EasyTest.
Type: InvalidOperationException Message: Showing a modal dialog box or form when the application is not running in UserInteractive mode is not a valid operation. Specify the ServiceNotification or DefaultDesktopOnly style to display a notification from a service application. Data: 0 entries Stack trace: at System.Windows.Forms.Form.ShowDialog(IWin32Window owner) at System.Windows.Forms.Form.ShowDialog() at DevExpress.ExpressApp.Win.Core.Messaging.ShowSystemException(String caption, String exceptionMessage) at DevExpress.ExpressApp.Win.Core.Messaging.ShowSystemException(String caption, Exception exception) at DevExpress.ExpressApp.Win.Core.Messaging.Show(String caption, Exception exception) at DevExpress.ExpressApp.Win.WinApplication.HandleExceptionCore(Exception e) at DevExpress.ExpressApp.Win.WinApplication.HandleException(Exception e) at DevExpress.ExpressApp.Win.WinApplication.Start()
Happy testing :)
Hello Christoph,
Thank you for sharing your experience with us and the XAF community. Just to confirm: did you solve the last error with run it as "auto-logon", correct? For its operation, EasyTest requires you to enable UI visibility for Windows agents.
This KB article describes how to deploy tests using Microsoft-hosted agents. Microsoft-hosted agents are already pre-configured for UI testing. If you use self-hosted Windows agents, you need to configure it to run as an interactive process with auto-logon enabled.
For more information, see the UI testing considerations article.
Please let me know if you have any issues with unit and functional testing in XAF.
Hi Pavel,
yes the last error was solved by running the self-hosted agent with "auto logon" instead of "as a service".
Hello DX,
Can this example (mostly the Powershell portion) be updated to support the new DX Nuget Feed Authorization?
Also the link to the pipeline yml file at the top is not working.
Thanks,
Alex
Hello Alex,
Thank you for this message. We have fixed the links.
To support DX Nuget Feed Authorization, you can use the approach from the answer to DevExpress NuGet - How to hide unique NuGet feed in Azure DevOps Build pipeline. (From the "Alternative approach with NuGet feed URL" section).
Please let me know if this approach meets your requirements.
Hi Alexander,
Yes. I have been using the environment variable to specify the DX nuget feed credentials for a long while, but I recently updated to the nuget feed authorization (as a service connection) as explained in this ticket.
Unfortunately that new approach doesn't work with the PowerShell task above, it outputs this:
2021-10-07T09:42:40.2584177Z ##[section]Starting: PowerShell 2021-10-07T09:42:40.2864004Z ============================================================================== 2021-10-07T09:42:40.2864388Z Task : PowerShell 2021-10-07T09:42:40.2864685Z Description : Run a PowerShell script on Linux, macOS, or Windows 2021-10-07T09:42:40.2865177Z Version : 2.190.0 2021-10-07T09:42:40.2865419Z Author : Microsoft Corporation 2021-10-07T09:42:40.2865767Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell 2021-10-07T09:42:40.2866173Z ============================================================================== 2021-10-07T09:42:42.1082653Z Generating script. 2021-10-07T09:42:42.7614649Z ========================== Starting Command Output =========================== 2021-10-07T09:42:42.7863678Z ##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\52ebfe7b-6ba1-490d-aa3d-3427b4f63810.ps1'" 2021-10-07T09:43:46.2633652Z LocalDB instance "MSSQLLocalDB" started. 2021-10-07T09:43:46.2638932Z 2021-10-07T09:43:47.2343971Z Feeds used: 2021-10-07T09:43:47.2346327Z https://nuget.devexpress.com/api/ 2021-10-07T09:43:47.2346502Z 2021-10-07T09:43:47.2380808Z Installing package 'DevExpress.EasyTest.TestExecutor' to 'D:\a\1\s\EasyTest'. 2021-10-07T09:43:47.5344501Z GET https://nuget.devexpress.com/api/FindPackagesById()?id='DevExpress.EasyTest.TestExecutor'&semVerLevel=2.0.0 2021-10-07T09:43:47.7745065Z MSBuild auto-detection: using msbuild version '16.11.0.36601' from 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\bin'. 2021-10-07T09:43:52.1711966Z Please provide credentials for: https://nuget.devexpress.com/api/ 2021-10-07T09:43:52.1916749Z Object reference not set to an instance of an object. 2021-10-07T09:43:52.1917810Z UserName: Password: Unauthorized https://nuget.devexpress.com/api/FindPackagesById()?id='DevExpress.EasyTest.TestExecutor'&semVerLevel=2.0.0 26ms 2021-10-07T09:48:48.2357028Z 2021-10-07T09:48:48.2358175Z 2021-10-07T09:48:48.2361406Z Directory: D:\a\1\s\EasyTest 2021-10-07T09:48:48.2362006Z 2021-10-07T09:48:48.2362176Z 2021-10-07T09:48:48.2542996Z Mode LastWriteTime Length Name 2021-10-07T09:48:48.2545057Z ---- ------------- ------ ---- 2021-10-07T09:48:48.2546907Z d----- 10/7/2021 9:48 AM Bin 2021-10-07T09:48:49.9005667Z EasyTest\Bin\TestExecutor.v21.1.exe : The module 'EasyTest' could not be loaded. For more information, run 2021-10-07T09:48:49.9006907Z 'Import-Module EasyTest'. 2021-10-07T09:48:49.9007632Z At D:\a\_temp\52ebfe7b-6ba1-490d-aa3d-3427b4f63810.ps1:9 char:1 2021-10-07T09:48:49.9027200Z + EasyTest\Bin\TestExecutor.v21.1.exe $easyTestPath 2021-10-07T09:48:49.9027822Z + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2021-10-07T09:48:49.9028425Z + CategoryInfo : ObjectNotFound: (EasyTest\Bin\TestExecutor.v21.1.exe:String) [], ParentContainsErrorReco 2021-10-07T09:48:49.9029079Z rdException 2021-10-07T09:48:49.9029549Z + FullyQualifiedErrorId : CouldNotAutoLoadModule 2021-10-07T09:48:49.9029934Z 2021-10-07T09:48:50.0210158Z ##[error]PowerShell exited with code '1'. 2021-10-07T09:48:50.1597337Z ##[section]Finishing: PowerShell
I will use the environment variable approach, but it would be nice if it also worked with the credentials in the service connection.
Thanks,
Alex
Hello Alex,
We have not tested this example with NuGet feed authorization. We do have plans to do this in the future. However, I cannot share the exact date.
I had a
Could not load file or assembly 'System.Runtime, Version=5.0.0.0
error in the EasyTest task. The problem was caused by the PS line where the Get-ChildItem gets the exes and dlls from the downloaded DX nuget packages. Now there are other sub folders for the other target frameworks. So I swapped the-notlike
with a-like
since now there arenet5.0-windows
andnetcoreapp3.0
dlls, if we want to avoid overwriting the net452 ones required for the TestExecutor.From:
Get-ChildItem -Path "EasyTest\*" -Include *.dll,*.exe -Recurse | Where {$_.FullName -notlike "*\netstandard*\*"} | Copy-Item -Destination ".\EasyTest\Bin\"
To:
Get-ChildItem -Path "EasyTest\*" -Include *.dll,*.exe -Recurse | Where {$_.FullName -like "*\net452\*" -Or $_.FullName -like "*\any\*"} | Copy-Item -Destination ".\EasyTest\Bin\"
It now works correctly.
HTH,
Alex
Hello DevOps enthusiasts,
This morning my CI pipeline failed for 2 reasons:
Selenium.WebDriver
package which is not available in the DevExpress nuget feed.To solve #1 we could easily add the
-version
switch tonuget install
, but this would mean maintenance to the pipeline task itself (the package version and the executable name). To avoid this, I decided to read the DevExpress version from the EasyTest config.xml and use it to restore and execute the appropriate TestExecutor package and executable. When we run the DX project converter, it updates the config.xml so this problem should be definitively solved.To solve #2, sadly, right now reusing the nuget authentication like we can with msbuild is not as straightforward (perhaps DX wants to take a look?). We can't re-use the nuget.config, because the script will timeout waiting for interactive authentication credentials (unless the credentials are stored in the nuget.config which we don't want). So the trick I used, is store the full DX nuget feed url in an Azure Key Vault secret and pass it to the script as an environment variable. And pass two
-source
parameters to thenuget install
: one for the full DX feed uri and one for nuget.org so it can restore the selenium package.The yaml task added after the build:
steps: - task: PowerShell@2 displayName: 'PowerShell Script' inputs: targetType: filePath filePath: ./src/GetAndRunTestExecutor.ps1 env: DevExpressNugetFeed: $(DevExpressNugetFeedFromVault) EasyTestPath: src\XafApp\XafApp.Module\FunctionalTests
And the PowerShell script:
Write-Host "Verifying script environment variables..." $devExpressNugetFeed = $Env:DevExpressNugetFeed if ([String]::IsNullOrWhiteSpace($devExpressNugetFeed)) { throw "Environment variable 'DevExpressNugetFeed' is empty. This should be the full uri including the api key (ex. https://nuget.devexpress.com/**********/api)." } $easyTestPath = $Env:EasyTestPath if ([String]::IsNullOrWhiteSpace($easyTestPath)) { throw "Environment variable 'EasyTestPath' is empty. This should be the relative path to the easy test folder containing script files (ex. src\Project\FunctionalTests)." } $easyTestConfigFile = $easyTestPath + "\config.xml" Write-Host "Reading DevExpress version from the EasyTest config.xml..." [XML]$EasyTestConfig = Get-Content $easyTestConfigFile $assemblyInfoNode = $EasyTestConfig.SelectNodes("//Options/Aliases/Alias[@Name='WinAdapterAssemblyName']") | select Value $versionProperty = $assemblyInfoNode.Value.Split(',')[1].Trim() if ($versionProperty -notlike "Version=*") { throw "Could not get the DevExpress version from the EasyTest config file. Make sure the EasyTest config file contains a WinAdapterAssemblyName or WebAdapterAssemblyName alias." } $devExpressVersion = [System.Version]::Parse($versionProperty.SubString("Version=".Length)) Write-Host "Found DevExpress version '$devExpressVersion' in EasyTest config file" Write-Host "Restoring TestExecutor Packages..." nuget.exe install DevExpress.EasyTest.TestExecutor -version $devExpressVersion -OutputDirectory EasyTest -source $devExpressNugetFeed -source "https://api.nuget.org/v3/index.json" $toolsPath = ".\EasyTest\Bin\" New-Item -ItemType directory -Path $toolsPath Get-ChildItem -Path "EasyTest\*" -Include *.dll,*.exe -Recurse | Where {$_.FullName -like "*\net452\*" -Or $_.FullName -like "*\any\*"} | Copy-Item -Destination $toolsPath Copy-Item "C:\Program Files (x86)\Microsoft.NET\Primary Interop Assemblies\Microsoft.mshtml.dll" -Destination $toolsPath $testExecutorFileName = "TestExecutor.v"+$devExpressVersion.ToString(2)+".exe" $testExecutorFullPath = $toolsPath+$testExecutorFileName if (!(Test-Path $testExecutorFullPath)) { throw "Executable '$testExecutorFullPath' doesn't exists." } sqllocaldb start MSSQLLocalDB Write-Host "Found and launching EasyTest TestExecutor '$testExecutorFullPath'" & $testExecutorFullPath $easyTestPath # Get the result and return exit code Get-Content -Path $easyTestPath\TestsLog.xml if(Select-String -Pattern 'Result="Failed"', 'Result="Warning"' -Path $easyTestPath\TestsLog.xml) { exit 1 }
You might need to change
WinAdapterAssemblyName
forWebAdapterAssemblyName
when getting the$assemblyInfoNode
depending on which adapter you use.HTH someone else.
Alex
Hello Alex,
Thank you for sharing this info with us. We will discuss how to update this article.
Thanks,
Andrey