-
Notifications
You must be signed in to change notification settings - Fork 495
Add Device Tests #3277
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
ne0rrmatrix
wants to merge
24
commits into
CommunityToolkit:main
Choose a base branch
from
ne0rrmatrix:DeviceTests
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Add Device Tests #3277
Changes from 12 commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
8695ed8
Add device tests for CommunityToolkit.Maui
ne0rrmatrix 1e5b5d7
Add unit tests for MediaElement and Views components
ne0rrmatrix e293d0a
Add build mappings for Debug and Release configurations in solution f…
ne0rrmatrix d3d1950
Potential fix for pull request finding
ne0rrmatrix 58f4a28
Merge branch 'main' into DeviceTests
ne0rrmatrix c96a70e
Updated to Use Maui XHarness
ne0rrmatrix 830f254
Updated to match Maui Xharness behavior
ne0rrmatrix eda34bf
Merge branch 'DeviceTests' of https://github.com/ne0rrmatrix/MauiOld …
ne0rrmatrix 4657c7a
Fix bad merge
ne0rrmatrix 2c4af21
Remove unused configuration sections from solution files
ne0rrmatrix dc33c44
Suppress warning for unused PropertyChanged event in TestPopupViewModel
ne0rrmatrix 8323f87
Enhance DeviceRunner and VisualRunnerPage with timeout handling and d…
ne0rrmatrix 89f409a
Refactor device tests to use DeviceRunners infrastructure
ne0rrmatrix 906b24f
Update .gitignore for benchmarks, test results, CLAUDE.md
ne0rrmatrix 845c5a9
Update test infra: remove test runner, clarify docs, add tools
ne0rrmatrix b718bd7
Merge branch 'main' into DeviceTests
ne0rrmatrix cd79712
Remove dotnet-tools.json
ne0rrmatrix a486ffd
Enhance device test configurations and update Snackbar tests for expe…
ne0rrmatrix 2d6d7e3
Remove unnecessary dependencies for device test jobs in CI workflow
ne0rrmatrix c3ed00b
Add Xcode version setup step and mark DeviceIdiom test as expected fa…
ne0rrmatrix 203e6df
Add Xcode version environment variable for macOS Catalyst device tests
ne0rrmatrix 13a5515
Refactor tests to eliminate reflection usage for accessing internal m…
ne0rrmatrix f3adc2b
Remove "ExpectedFailure" trait from various test classes to streamlin…
ne0rrmatrix 3d2b3d8
Add "ExpectedFailure" trait to DeviceIdiom test for improved categori…
ne0rrmatrix File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| --- | ||
| description: 'Building and running device tests for the .NET MAUI Community Toolkit: the XHarness/XunitFrontController test runner, handler creation without page navigation, and avoiding infinite re-run loops' | ||
| applyTo: 'src/CommunityToolkit.Maui.DeviceTests/**/*.cs' | ||
| --- | ||
|
|
||
| ## Device Tests | ||
|
|
||
| Device tests live in `src/CommunityToolkit.Maui.DeviceTests` and run inside a real MAUI application on a target platform (Windows, Android, iOS, MacCatalyst). They verify platform-specific behavior that unit tests cannot. | ||
|
|
||
| ### Test runner architecture | ||
|
|
||
| The runner mirrors the [dotnet/maui](https://github.com/dotnet/maui) team's approach: | ||
|
|
||
| - **`DeviceRunner`** discovers and executes tests directly through xunit's `XunitFrontController` (from the `xunit.runner.utility` package). Do **not** load the XHarness runner types via reflection (`Activator.CreateInstance` for `XUnitTestRunner`, or `XmlResultJargon` for results) — that fails silently and tests never run. | ||
| - Results are counted per-test from the execution sink messages (`ITestPassed`, `ITestFailed`, `ITestSkipped`). `ITestAssemblyFinished` does not expose a summary in the xunit version in use. | ||
| - The execution sink implements `IExecutionSink` directly (including `OnMessageWithTypes`) and must be marked `partial` to satisfy the CsWinRT analyzer on Windows (CsWinRT1028). The outer `DeviceRunner` must also be `partial` for the same reason. | ||
|
|
||
| ### Creating handlers in tests | ||
|
|
||
| When a test needs a handler/platform view, create it directly with `element.ToHandler(context)` using the application's `MauiContext` — do **not** replace `window.Page` to force handler creation: | ||
|
|
||
| ```csharp | ||
| var context = Application.Current?.Handler?.MauiContext; | ||
| var handler = element.ToHandler(context); | ||
| ``` | ||
|
|
||
| Replacing `window.Page` hides the visual test runner page (leaving stray content such as a "Click Me" button) and forces the runner to restore the page afterward. Handler creation must run on the main thread (`MainThread.InvokeOnMainThreadAsync`). | ||
|
|
||
| ### Run tests exactly once | ||
|
|
||
| `VisualRunnerPage` runs tests from `OnAppearing`. Anything that reassigns `window.Page` (or otherwise re-shows the page) re-fires `OnAppearing` and can cause an infinite re-run loop. Guard with a `hasRun` flag so the suite executes a single time per app launch. | ||
|
|
||
| ### Packages and feeds | ||
|
|
||
| - xunit **v2** (`2.9.3`) is required — XHarness is not compatible with xunit v3. | ||
| - XHarness packages (`Microsoft.DotNet.XHarness.TestRunners.Xunit`) come from the `dotnet-eng` Azure DevOps NuGet feed configured in `NuGet.config`, not nuget.org. | ||
|
|
||
| ### Running | ||
|
|
||
| ```bash | ||
| dotnet build src/CommunityToolkit.Maui.DeviceTests/CommunityToolkit.Maui.DeviceTests.csproj -f net10.0-windows10.0.19041.0 -t:Run | ||
| ``` | ||
|
|
||
| The visual runner displays results in the app UI and reports pass/fail counts to the trace log. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| using CommunityToolkit.Maui.DeviceTests.Runners; | ||
|
|
||
| namespace CommunityToolkit.Maui.DeviceTests; | ||
|
|
||
| public partial class App : Application | ||
| { | ||
| readonly VisualRunnerPage visualRunnerPage; | ||
|
|
||
| public App(VisualRunnerPage visualRunnerPage) | ||
| { | ||
| this.visualRunnerPage = visualRunnerPage; | ||
| } | ||
|
|
||
| protected override Window CreateWindow(IActivationState? activationState) | ||
| { | ||
| return new Window(visualRunnerPage); | ||
| } | ||
| } |
69 changes: 69 additions & 0 deletions
69
src/CommunityToolkit.Maui.DeviceTests/CommunityToolkit.Maui.DeviceTests.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <TargetFrameworks>$(NetVersion)-android;$(NetVersion)-ios;$(NetVersion)-maccatalyst</TargetFrameworks> | ||
| <TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('windows'))">$(TargetFrameworks);$(NetVersion)-windows10.0.19041.0</TargetFrameworks> | ||
| <OutputType>Exe</OutputType> | ||
| <UseMaui>true</UseMaui> | ||
| <SingleProject>true</SingleProject> | ||
| <IsTestProject>true</IsTestProject> | ||
| <RootNamespace>CommunityToolkit.Maui.DeviceTests</RootNamespace> | ||
|
|
||
| <!-- Display name --> | ||
| <ApplicationTitle>CommunityToolkit.Maui Device Tests</ApplicationTitle> | ||
|
|
||
| <!-- App Identifier --> | ||
| <ApplicationId>com.microsoft.CommunityToolkit.Maui.DeviceTests</ApplicationId> | ||
| <ApplicationIdGuid>A1B2C3D4-E5F6-7890-ABCD-EF1234567890</ApplicationIdGuid> | ||
|
|
||
| <!-- Versions --> | ||
| <ApplicationDisplayVersion>1.0</ApplicationDisplayVersion> | ||
| <ApplicationVersion>1</ApplicationVersion> | ||
|
|
||
| <!-- Windows unpackaged for easier local debugging --> | ||
| <WindowsPackageType>None</WindowsPackageType> | ||
| <AllowUnsafeBlocks>true</AllowUnsafeBlocks> | ||
|
|
||
| <!-- Suppress xunit analyzer rules for pre-existing test patterns --> | ||
| <!-- IL3000: Assembly.Location is valid for MAUI device test apps (not single-file published) --> | ||
| <NoWarn>$(NoWarn);xUnit2032;xUnit1004;IL3000</NoWarn> | ||
|
|
||
| <SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios'">15.0</SupportedOSPlatformVersion> | ||
| <SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'maccatalyst'">15.0</SupportedOSPlatformVersion> | ||
| <SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'android'">21.0</SupportedOSPlatformVersion> | ||
| <SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'windows'">10.0.17763.0</SupportedOSPlatformVersion> | ||
| <TargetPlatformMinVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'windows'">10.0.17763.0</TargetPlatformMinVersion> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="Microsoft.Maui.Controls" Version="$(MauiPackageVersion)" /> | ||
| <PackageReference Include="Microsoft.DotNet.XHarness.TestRunners.Xunit" Version="11.0.0-prerelease.26230.4" /> | ||
| <PackageReference Include="xunit" Version="2.9.3" /> | ||
| <PackageReference Include="xunit.runner.utility" Version="2.9.3" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\CommunityToolkit.Maui\CommunityToolkit.Maui.csproj" /> | ||
| <ProjectReference Include="..\CommunityToolkit.Maui.Core\CommunityToolkit.Maui.Core.csproj" /> | ||
| <ProjectReference Include="..\CommunityToolkit.Maui.Camera\CommunityToolkit.Maui.Camera.csproj" /> | ||
| <ProjectReference Include="..\CommunityToolkit.Maui.MediaElement\CommunityToolkit.Maui.MediaElement.csproj" /> | ||
| <ProjectReference Include="..\CommunityToolkit.Maui.Maps\CommunityToolkit.Maui.Maps.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <MauiIcon Include="Resources\AppIcon\appicon.svg" ForegroundFile="Resources\AppIcon\appiconfg.svg" Color="#512BD4" /> | ||
| <MauiSplashScreen Include="Resources\Splash\splash.svg" Color="#512BD4" BaseSize="128,128" /> | ||
| </ItemGroup> | ||
|
|
||
| <!-- Include/exclude platform-specific files --> | ||
| <ItemGroup Condition="!$(TargetFramework.Contains('-android'))"> | ||
| <Compile Remove="**\*.android.cs" /> | ||
| </ItemGroup> | ||
| <ItemGroup Condition="!$(TargetFramework.Contains('-ios')) and !$(TargetFramework.Contains('-maccatalyst'))"> | ||
| <Compile Remove="**\*.macios.cs" /> | ||
| </ItemGroup> | ||
| <ItemGroup Condition="!$(TargetFramework.Contains('-windows'))"> | ||
| <Compile Remove="**\*.windows.cs" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| global using Xunit; | ||
|
|
||
| [assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly, DisableTestParallelization = true, MaxParallelThreads = 1)] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you should use the https://github.com/mattleibow/DeviceRunners project from @mattleibow, that is what .net maui uses and can run on CI
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will migrate to @matt-bartholomew project design
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you meant @mattleibow
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have refactored it as suggested