Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for automatically taking screenshots at the end of each UI test (passed or failed). #130

Merged
merged 25 commits into from
Nov 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
e8559fe
Added more logging (Solution_Name_Is_Added_To_Chat_Input very rarely …
PiotrKarczmarz Oct 24, 2024
4b28c10
Fixed logging.
PiotrKarczmarz Oct 25, 2024
4baa676
Turned on taking screenshots for failed UI tests:
PiotrKarczmarz Oct 25, 2024
7a6ffc4
Fixed wildcard usage for screenshots.
PiotrKarczmarz Oct 25, 2024
4231be3
Switched to upload-artifact@v4 and uploading entire Screenshots direc…
PiotrKarczmarz Oct 25, 2024
7c93145
Testing file upload.
PiotrKarczmarz Oct 25, 2024
73651a1
Testing file upload.
PiotrKarczmarz Oct 25, 2024
1f88291
Testing file upload.
PiotrKarczmarz Oct 25, 2024
230f8f2
Testing file upload.
PiotrKarczmarz Oct 25, 2024
0d35f16
Testing file upload.
PiotrKarczmarz Oct 25, 2024
f21c7cd
Trying to change VM screen resolution.
PiotrKarczmarz Oct 25, 2024
3b54c27
Trying to change VM screen resolution.
PiotrKarczmarz Oct 25, 2024
61c49fc
Trying to change VM screen resolution.
PiotrKarczmarz Oct 25, 2024
702dca7
Trying to change VM screen resolution.
PiotrKarczmarz Oct 25, 2024
a0f0e94
Taking screenshots during UI tests (ChatLoggedBasicTests)
PiotrKarczmarz Oct 25, 2024
b7f27fd
More logging for screenshots.
PiotrKarczmarz Oct 25, 2024
c988ecc
Switched to Assembly.GetExecutingAssembly().Location for screenshots.
PiotrKarczmarz Oct 25, 2024
eff3e1c
Removing file name from the path.
PiotrKarczmarz Oct 25, 2024
73e72c0
Taking screenshots for ChatNotLoggedStateTests.
PiotrKarczmarz Oct 28, 2024
63a032c
Fixed data for Complex_JSON_Data_Should_Be_Serializable() test.
PiotrKarczmarz Oct 28, 2024
9d55d56
Updated nightly script.
PiotrKarczmarz Oct 28, 2024
e6fe475
Turned on Entered_Prompt_Show_Up_In_Today_History() UI test.
PiotrKarczmarz Oct 30, 2024
94cff68
Turned off Task Entered_Prompt_Show_Up_In_Today_History() (waiting fo…
PiotrKarczmarz Oct 31, 2024
88164f4
Switched to main for CI.
PiotrKarczmarz Oct 31, 2024
5800023
Fixed formating.
PiotrKarczmarz Oct 31, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/cake-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ jobs:
- name: Add msbuild to PATH
uses: microsoft/setup-msbuild@v1.3

- name: Change Screen Resolution
shell: pwsh
run: Set-DisplayResolution -Width 1920 -Height 1080 -Force

- name: ⚙️ Prepare Visual Studio
run: '&"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\devenv.exe" /RootSuffix Exp /ResetSettings General.vssettings'

Expand Down Expand Up @@ -69,6 +73,14 @@ jobs:
cd src
dotnet test .\Cody.Core.Tests\bin\Debug\Cody.Core.Tests.dll .\Cody.VisualStudio.Tests\bin\Debug\Cody.VisualStudio.Tests.dll -v detailed -l:trx

- name: Upload screenshots for UI tests
uses: actions/upload-artifact@v4
if: always()
with:
name: UI Tests Screenshots
path: src/Cody.VisualStudio.Tests/bin/Debug/Screenshots
retention-days: 20

- name: Publish Test Results
uses: EnricoMi/publish-unit-test-result-action/windows@v2
if: always()
Expand Down
12 changes: 12 additions & 0 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ jobs:
- name: Add msbuild to PATH
uses: microsoft/setup-msbuild@v1.3

- name: Change Screen Resolution
shell: pwsh
run: Set-DisplayResolution -Width 1920 -Height 1080 -Force

- name: ⚙️ Prepare Visual Studio
run: '&"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\devenv.exe" /RootSuffix Exp /ResetSettings General.vssettings'

Expand Down Expand Up @@ -53,6 +57,14 @@ jobs:
cd src
dotnet test .\Cody.VisualStudio.Tests\bin\Debug\Cody.VisualStudio.Tests.dll -v detailed -l:trx

- name: Upload screenshots for UI tests
uses: actions/upload-artifact@v4
if: always()
with:
name: UI Tests Screenshots
path: src/Cody.VisualStudio.Tests/bin/Debug/Screenshots
retention-days: 5

- name: Publish Test Results
uses: EnricoMi/publish-unit-test-result-action/windows@v2
if: always()
Expand Down
2 changes: 1 addition & 1 deletion src/Cody.Core.Tests/CustomConfigurationTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ public void Complex_JSON_Data_Should_Be_Serializable()
// given
var key1 = "cody.autocomplete.enabled";
var key2 = "cody.customHeaders";
var key3 = "cody.customHeaders";
var key3 = "cody.excludeFiles";
var configurationJson = $@"{{
""{key1}"": true,
""{key2}"": {{
Expand Down
10 changes: 7 additions & 3 deletions src/Cody.VisualStudio.Tests/ChatLoggedBasicTests.cs
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using EnvDTE;
using Xunit;
using Xunit.Abstractions;

namespace Cody.VisualStudio.Tests
{
public class ChatLoggedBasicTests : PlaywrightTestsBase
public class ChatLoggedBasicTests : PlaywrightTestsBase, IDisposable
{
public ChatLoggedBasicTests(ITestOutputHelper output) : base(output)
{
Expand Down Expand Up @@ -80,5 +78,11 @@ public async Task Entered_Prompt_Show_Up_In_Today_History()
Assert.Contains(chatHistoryEntries, x => x.Contains(prompt));

}

public void Dispose()
{
var testName = GetTestName();
TakeScreenshot(testName);
}
}
}
8 changes: 7 additions & 1 deletion src/Cody.VisualStudio.Tests/ChatNotLoggedStateTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

namespace Cody.VisualStudio.Tests
{
public class ChatNotLoggedStateTests : PlaywrightTestsBase
public class ChatNotLoggedStateTests : PlaywrightTestsBase, IDisposable
{
public ChatNotLoggedStateTests(ITestOutputHelper output) : base(output)
{
Expand Down Expand Up @@ -43,5 +43,11 @@ public async Task Loads_Properly_InNotLoggedState()
// then
Assert.Equal(text, textContents.First());
}

public void Dispose()
{
var testName = GetTestName();
TakeScreenshot(testName);
}
}
}
2 changes: 2 additions & 0 deletions src/Cody.VisualStudio.Tests/Cody.VisualStudio.Tests.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
<Reference Include="System" />
<Reference Include="System.Configuration" />
<Reference Include="System.Core" />
<Reference Include="System.Drawing" />
<Reference Include="System.Runtime" />
<Reference Include="System.Runtime.Serialization" />
<Reference Include="System.Xml.Linq" />
Expand All @@ -52,6 +53,7 @@
</Compile>
<Compile Include="ChatNotLoggedStateTests.cs" />
<Compile Include="ChatLoggedBasicTests.cs" />
<Compile Include="ScreenshotUtil.cs" />
<Compile Include="PlaywrightTestsBase.cs" />
<Compile Include="PlaywrightInitializationTests.cs" />
<Compile Include="CodyPackageTests.cs" />
Expand Down
14 changes: 12 additions & 2 deletions src/Cody.VisualStudio.Tests/PlaywrightTestsBase.cs
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,11 @@ protected async Task SetAccessToken(string accessToken)

protected async Task ShowChatTab() => await Page.GetByTestId("tab-chat").ClickAsync();

protected async Task ShowHistoryTab() => await Page.GetByTestId("tab-history").ClickAsync();
protected async Task ShowHistoryTab()
{
await Page.GetByTestId("tab-history").ClickAsync();
await Task.Delay(500);
}

protected async Task ShowPromptsTab() => await Page.GetByTestId("tab-prompts").ClickAsync();

Expand Down Expand Up @@ -162,8 +166,14 @@ protected async Task<IReadOnlyCollection<ContextTag>> GetChatContextTags()
{
var tagsList = new List<ContextTag>();

WriteLog("Searching for Chat ...");
var chatBox = await Page.QuerySelectorAsync("[aria-label='Chat message']");
if (chatBox == null) throw new Exception("ChatBox is null. Probably not authenticated.");
if (chatBox == null)
{
WriteLog("Chat NOT found.");
throw new Exception("ChatBox is null. Probably not authenticated.");
}
WriteLog("Chat found.");

var list = await chatBox.QuerySelectorAllAsync("span[data-lexical-decorator='true']");
foreach (var item in list)
Expand Down
1 change: 1 addition & 0 deletions src/Cody.VisualStudio.Tests/Properties/AssemblyInfo.cs
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,4 @@
// [assembly: AssemblyVersion("1.0.*")]
[assembly: TestFramework("Xunit.VsTestFramework", "VsixTesting.Xunit")]
[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly, DisableTestParallelization = true, MaxParallelThreads = 1)]
[assembly: VsTestSettings(TakeScreenshotOnFailure = true)]
52 changes: 52 additions & 0 deletions src/Cody.VisualStudio.Tests/ScreenshotUtil.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
using System;
using System.Runtime.InteropServices;
using System.Drawing;
using System.Drawing.Imaging;

namespace Cody.VisualStudio.Tests
{
public class ScreenshotUtil
{
public static void CaptureWindow(IntPtr hwnd, string path)
{
var rect = default(RECT);
GetWindowRect(hwnd, ref rect);
CaptureScreenArea(
path,
left: rect.Left,
top: rect.Top,
width: rect.Right - rect.Left,
height: rect.Bottom - rect.Top);
}

public static void CaptureScreenArea(string path, int left, int top, int width, int height)
{
using (var bitmap = new Bitmap(width, height))
using (var image = Graphics.FromImage(bitmap))
{
image.CopyFromScreen(
sourceX: left,
sourceY: top,
blockRegionSize: new Size(width, height),
copyPixelOperation: CopyPixelOperation.SourceCopy,
destinationX: 0,
destinationY: 0);

bitmap.Save(path, ImageFormat.Png);
}
}

[DllImport("user32.dll")]
private static extern bool GetWindowRect(IntPtr hWnd, ref RECT rect);

[StructLayout(LayoutKind.Sequential)]
private struct RECT
{
public int Left;
public int Top;
public int Right;
public int Bottom;
}
}

}
36 changes: 34 additions & 2 deletions src/Cody.VisualStudio.Tests/TestsBase.cs
Original file line number Diff line number Diff line change
@@ -1,14 +1,17 @@
using System;
using System.Drawing.Imaging;
using System.IO;
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Threading.Tasks;
using System.Windows;
using Cody.Core.Logging;
using EnvDTE;
using EnvDTE80;
using Microsoft.VisualStudio.Shell;
using Microsoft.VisualStudio.Shell.Interop;
using Microsoft.VisualStudio.TextManager.Interop;
using Xunit.Abstractions;
using System.Diagnostics;
using Thread = System.Threading.Thread;

namespace Cody.VisualStudio.Tests
Expand Down Expand Up @@ -38,17 +41,46 @@ public void WriteLog(string message, string type = "", [CallerMemberName] string
}
}

protected string GetTestName()
{
var test = (ITest)_logger.GetType()
.GetField("test", BindingFlags.Instance | BindingFlags.NonPublic)
?.GetValue(_logger);

var testName = test?.DisplayName;
return testName;
}

protected void TakeScreenshot(string name)
{
var safeName = string.Join("_", name.Split(Path.GetInvalidFileNameChars()));
var date = DateTime.Now.ToString("yyyy-MM-dd hh.mm.ss");

var directory = "Screenshots";
var directoryPath = Path.GetFullPath(Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), directory));
var path = Path.Combine(directoryPath, $"{date} {safeName}.png");

WriteLog($"Saving screenshots to:{directoryPath}");
Directory.CreateDirectory(directoryPath);

ScreenshotUtil.CaptureWindow(Process.GetCurrentProcess().MainWindowHandle, path);
WriteLog($"Screenshot saved to:{path}");
}

private IVsUIShell _uiShell;
protected IVsUIShell UIShell => _uiShell ?? (_uiShell = (IVsUIShell)Package.GetGlobalService(typeof(SVsUIShell)));

private DTE2 _dte;
protected DTE2 Dte => _dte ?? (_dte = (DTE2)Package.GetGlobalService(typeof(DTE)));
protected DTE2 Dte => _dte ?? (_dte = (DTE2)Package.GetGlobalService(typeof(EnvDTE.DTE)));

protected async Task OpenSolution(string path)
{

WriteLog($"Opening solution '{path}' ...");
Dte.Solution.Open(path);

await Task.Delay(TimeSpan.FromSeconds(5));
WriteLog("Delay after solution open stopped.");
}

protected void CloseSolution() => Dte.Solution.Close();
Expand Down
1 change: 1 addition & 0 deletions src/Cody.sln
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
build.cake = build.cake
..\.github\workflows\cake-build.yml = ..\.github\workflows\cake-build.yml
..\.github\workflows\code-style.yml = ..\.github\workflows\code-style.yml
..\.github\workflows\nightly.yml = ..\.github\workflows\nightly.yml
..\.github\workflows\publish.yml = ..\.github\workflows\publish.yml
..\.github\workflows\release-preview.yml = ..\.github\workflows\release-preview.yml
EndProjectSection
Expand Down
Loading