Added screenshot comparison mode to TestRunner (pause=-4) - #3861
Draft
chambm wants to merge 33 commits into
Draft
Conversation
* Added ScreenshotComparer to compare tutorial screenshots against existing files * Added ScreenshotComparisonResults to accumulate results across multiple tests * Extended ScreenshotDiff with percentage calculations excluding title bar * Screenshot failures now cause TestRunner to return non-zero exit code (when comparison mode is enabled) * Report shows diff percentage per screenshot grouped by test name * Failed comparison screenshots are written to TestResults/ScreenshotDiffs/<Tutorial Name> (the diff, the original, and new), for easy access as artifacts in TeamCity or local testing
* added exception for ScreenshotDiffs in test cleanup code
…mappings * added per-channel color tolerance (3) and allowed system color mappings for known Windows theme differences * added dominant color pair reporting for screenshot comparisons to ignore systematic color shifts * changed diff image saving to write for any non-zero diff, not just failures above threshold * removed title bar exclusion logic in favor of color tolerance and system color mapping approach * restricted screenshot comparison to tutorials with CoverShotName and fixed small molecule tutorial tests to unset CoverShotName * activated window and brought to front before taking screenshots to avoid capturing wrong content (easy to do when you're using the computer while running a comparison test) * stopped deleting TestResults between runs
* split Report into ReportCurrent (per-test) and ReportSummary (overall with failed screenshot list including description and failure reason) * added screen= parameter to TestRunner for choosing screenshot capture screen, and move SkylineWindow to chosen screen at test start * improved tc-perftests.bat with abort detection, TEAMCITY_VERSION guard, and cleanup subroutine
* added AdjustToBitmapCoords call in ClipRegionAndEraseBackground for non-primary screen support * moved SkylineWindow to chosen screenshot screen at test start when IsRecordingScreenShots * clear previous diff images for test before starting new comparison * updated CodeInspectionTest for new Shell32Test DllImport count (2)
* added GetResultsToSave() to only save failed comparisons on TeamCity, all diffs locally * fixed code inspection
…nd new auto-record mode using that location * copied 2038 tutorial screenshots from Documentation/Tutorials to TestTutorial/TutorialScreenshots with per-language directories (en, ja, zh-CHS) for comparison testing * added pause=-5 "Auto-Screenshots for development" mode to record screenshots to TutorialScreenshotPath instead of TutorialPath, and added "Screenshot Comparisons" mode to SkylineTester UI * made MoveToScreenshotScreen and GetScreenshotScreen static, and moved window positioning outside IsRecordingScreenShots so it works for any test mode * wrapped IsPauseForScreenShots and IsAutoScreenShotMode in !IsPass0 to avoid screenshot comparison in pass0 (which uses different data), and likewise fixed DiaTutorialTest.IsFullImportMode * improved ScreenshotComparisonResults report format with dashes and screenshot descriptions
* changed screen= default from 1 to -1 (auto-select largest screen) * added GetLargestScreen() using OrderByDescending pixel area, preferring primary display for ties * updated help text and error message to document -1 as valid value Co-Authored-By: Claude <noreply@anthropic.com>
…anup * forced offscreen=false when pause < 0 (screenshot modes) so view restores and layout work correctly on TeamCity * cleaned up temp CSV files left behind by RunTool in QuasarTutorialTest and MSstatsTutorialTest (pause=-4 failure) * set MainWindowLocation in InitializeSkylineSettings to start Skyline on the target screenshot screen from initialization * always call MoveToScreenshotScreen regardless of ScreenshotScreenIndex value Co-Authored-By: Claude <noreply@anthropic.com>
* fixed SystemSleep displayRequired condition to match negative pause values (<= -3, not == 3) * cleaned up em dashes in comments Co-Authored-By: Claude <noreply@anthropic.com>
…23_screenshot_comparison_mode # Conflicts: # pwiz_tools/Skyline/TestUtil/ScreenshotInfo.cs # pwiz_tools/Skyline/TestUtil/TestFunctional.cs
…t for screenshot debugging - Recorded DIASearchStellar reference screenshots for en, ja, zh-CHS (pause=-5) - Added per-monitor DPI scale logging in TestRunner for screenshot modes (ShcoreTest.GetScaleFactor) - Temporarily limited tc-perftests.bat to 6 tests with screenshot failures for faster CI iteration Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…uild log Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… focus retry Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- s-19: Changed to PauseForScreenShot<AssociateProteinsDlg> to capture just the dialog instead of full Skyline window (was 31% diff on TC) - s-18: Added ProgressLock to freeze search progress at fixed line count for deterministic screenshots - Cover shot: Added ActivateScreenshotForm and FilterTimeMessageLock - Updated screenshots for en/ja/zh-CHS Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Replaced fixed 1500ms Thread.Sleep with WaitForScreenshotToSettle that compares against reference screenshots and exits early when settled (<1% diff) * Added QuickDiffPercent to ScreenshotDiff using same color tolerance and allowed mappings as full comparison * Reduced ActivateScreenshotForm sleep from 500ms to 100ms * Added Win11 accent color bleed mappings for title bar tinting (purple/blue-tinted gray -> neutral gray) * Fixed AdjustToBitmapCoords to always subtract screen origin for multi-monitor support * Fixed DPI-independent text measurement in AxisLabelScaler * Synced ~1900 tutorial screenshots from Documentation/Tutorials Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
* Fixed ShcoreTest DllImport name casing ("Shcore.dll" -> "shcore.dll")
* Added ShcoreTest to expectedPInvokeApi dictionary in CodeInspectionTest
* Updated User32Test expected DllImport count from 9 to 10 for MonitorFromPoint
Co-Authored-By: Claude <noreply@anthropic.com>
…23_screenshot_comparison_mode # Conflicts: # pwiz_tools/Skyline/TestRunnerLib/RunTests.cs
…_screenshots mode to ImageComparer * Recaptured all 19 PeakBoundaryImputation-DIA tutorial screenshots with fixed window dimensions * Added developer_screenshots ImageSource for comparing TutorialScreenshots against Documentation/Tutorials * Added GetAlternatePath() to ScreenshotFile to swap between the two screenshot locations * Updated ImageComparer to auto-detect TutorialScreenshots folders and enumerate all PNGs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ng to match original dimensions * Reverted 19 recaptured screenshots back to original size * Set fixed window dimensions (1680x1032) for screenshot recording and comparison modes * Height of 1032 matches the original screenshots captured on 1920x1080 with 48px taskbar Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…n full suite Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nt not collected) When NavBarButtonViewsOnDropDownOpening creates a new ContextMenuStrip, the old one remained subscribed to the static SystemEvents.UserPreferenceChanged event, keeping the entire chain alive: SystemEvents → ContextMenuStrip → ToolStripMenuItems → NavBar → BindingNavigator → SkylineWindow. Additionally, BindingListSource.Dispose did not clear its internal BindingListView or base.DataSource, leaving CurrencyManager cache entries that chained through ViewInfo → DataSchema → SkylineWindow. The fixes form an interdependent disposal chain: - NavBar.cs: Dispose old ContextMenuStrip before creating new one (breaks static event root) - BindingListSource.cs: Clear ViewContext, BindingListView, and base.DataSource on dispose - BoundDataGridView.cs: Add Dispose to unsubscribe events and clear _itemProperties - NavBar.Designer.cs: Dispose dynamic dropdown and null _bindingListSource - DataboundGridControl.designer.cs: Dispose BindingListSource before components container - DocumentGridForm.cs: Null-safe BindingListSource?.ViewInfo for disposal cascade events - AuditLogForm.cs: Null _skylineWindow in OnHandleDestroyed - BindingListView.cs: Clear _reportResults in Dispose - DataGridViewPasteHandler.cs: Add IDisposable to unsubscribe KeyDown - DataboundGridControl.cs: Guard MainGridPopulated against null BindingListSource Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mentGridForm) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… investigation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Re-activate the target window on each retry iteration so that if the first activation doesn't bring the window to the foreground (e.g. on TeamCity agents), subsequent retries will re-activate and capture the correct content instead of a blank or wrong window. Co-Authored-By: Claude <noreply@anthropic.com>
In recording mode (pause=-5), after capturing each screenshot, temporarily disables window drop shadows via SystemParametersInfo(SPI_SETDROPSHADOW) and captures a second version. If the two differ by more than 1%, saves the no-shadow version as s-XX-noshadow.png alongside the standard reference. In comparison mode (pause=-4), compares against both the standard and noshadow references (if it exists), using whichever gives the smaller diff. This handles the case where Windows Server 2022 (smaller shadows) is compared against references captured on Windows 11 (larger shadows). Also skips overwriting screenshots when pixel content is unchanged (< 0.01% diff) to avoid unnecessary git churn from PNG re-encoding differences. Updates reference screenshots for SmallMoleculeMethodDevCEOpt, DIA-QE-DIANN, and DIASearchStellar tutorials. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mostly-Authored-By: Claude noreply@anthropic.com ;)