Port to .NET Core 3.0 and revamp engineering/build/test #201

mjcheetham · 2019-10-29T13:18:22Z

Non-goals

Signing

New commands
dotnet build: Build all managed code
dotnet test: Run all managed unit tests
dotnet publish -r [osx-x64|win-x64]: Build all code including Scalar installer packages and scripts

Scripts
BuildScalarForMac.sh/BuildScalarForWindows.bat: Build all code and generate installers and install scripts
RunUnitTests.(sh|bat): Run the managed unit tests
RunFunctionalTests.(sh|bat): Run the functional tests
RestorePackages.bat: No-op

Directories

.
├── out                          (build output)
│   ├── <Project1>
│   ├──    ...
│   ├── <ProjectN>
│   │
│   ├── Scalar.Distribution.Mac  (distribution: packages, scripts)
│   │  └── dist
│   │      └── <Configuration>
│   │          ├── InstallScalar.sh
│   │          │
│   │          ├── Scalar
│   │          │   └── Scalar.<version>.pkg
│   │          ├── GCM
│   │          │   └── gcmcore-osx-<version>.pkg
│   │          └── Git
│   │              └── git-<version>.pkg
│   │
│   └── Scalar.Distribution.Windows
│      └── dist
│          └── <Configuration>
│              ├── InstallScalar.bat
│              │
│              ├── Scalar
│              │   └── Scalar.<version>.exe
│              └── Git
│                  └── Scalar.<version>.exe
│
└── src                          (source)
    ├── <Project1>
    ├──    ...
    ├── <ProjectN>
    │
    ├── Dependencies.props       (Unified NuGet package versions)
    └── Directory.Build.props    (Common build props, including minimum Git version)
    └── Directory.Build.targets  (Common build target)

Changes

Functional tests drops in CI do not include copies of the product; the installer drop must be consumed and installed.
The metapackage on macOS has been dropped.. we only build the Scalar package and leave installation of the correct version of Git to the outside.
The Platform projects have been folded into the Common project, under the Platforms/ directory. There is now one platform loader that instantiates the correct type based on the current OS at runtime.
The Scalar.Build project has been slimmed down massively; we only need:
- Generate Scalar constants (version)
- Compile templated file (x-plat replacement for sed to replace placeholder tokens)
- Generate Windows application manifest (set supported Windows versions)
The Scalar.Mac and Scalar.Windows projects have been folded into one, with a runtime fork to create the correct platform-specific types. The same applies for the other .Mac/.Windows projects.
Scalar.Tests has been renamed to Scalar.TestInfratructure to better reflect its purpose.
The Scalar.Notifications/Scalar.Mac directory has been flattened, and an MSBuild project been added that wraps the xcodeproj project. This allows us to chain in an xcodebuild into the MSBuild build with the dotnet CLI (see dotnet build/publish).
Introduce shims for some missing APIs in .NET Core from .NET Framework, mainly things around ACLs which have been implemented in the new type DirectoryEx in the Common project.
Include a copy of internal NamedPipeServerStream methods from the CoreFX to create pipes with the appropriate ACLs on Windows.

mjcheetham · 2019-10-30T12:54:46Z

Currently builds fail intermittently on Windows with NuGet/Home#8692

mjcheetham · 2019-10-30T14:58:01Z

Currently builds fail intermittently on Windows with NuGet/Home#8692

Looks like a NuGet bug that was fixed in 5.1.
NuGet/Home#7854

Need to update the version of NuGet we're using in Pipelines.

derrickstolee · 2019-10-30T18:08:50Z

I checked out your code on my Windows machine. Building with Scripts/BuildScalarForWindows.bat worked like a charm. Visual Studio opened the solution just fine. Test Explorer sees the unit tests AND the functional tests. I only tried running the unit tests, which worked (except for tests that use [TestCaseSource(...)] instead of [TestCase]).

I've re-cloned the Office repo using this branch and am currently hydrating files using the gvfs-helper. I'll double-check the maintenance steps will start running from the service.

derrickstolee · 2019-10-30T20:04:25Z

@mjcheetham: I had @wilbaker complete his PR, but didn't realize the conflicts it would cause for you. The plan is to hold off on further major changes until this is ready, unless we think it's going to take a long time. For instance, William isn't going to delete the Scalar.Mount project until this is done, but he will be working on some logic in the service.

cc: @jrbriggs

wilbaker · 2019-10-30T20:45:37Z

@mjcheetham: I had @wilbaker complete his PR, but didn't realize the conflicts it would cause for you. The plan is to hold off on further major changes until this is ready, unless we think it's going to take a long time. For instance, William isn't going to delete the Scalar.Mount project until this is done, but he will be working on some logic in the service.

Yep apologies for the conflicts! Fortunately they are easy to resolve.

The change I made to Scalar/Scalar.Windows.csproj was simply removing two files, and so the resolution there is to delete Scalar/Scalar.Windows.csproj.
The changes I made to Setup.iss had only one conflict, and the resolved code looks like this:

procedure CurStepChanged(CurStep: TSetupStep);
begin
  case CurStep of
    ssInstall:
      begin
        UninstallService('Scalar.Service', True);
      end;
    ssPostInstall:
      begin
        InstallScalarService();
        RegisterUserWithService();
      end;
    end;
end;

If you're interested I pushed a branch here that has a merge of master into refactor3 with the above conflicts resolved.

cc: @jrbriggs, @derrickstolee

mjcheetham · 2019-10-31T14:54:36Z

/azp run microsoft.scalar

.azure-pipelines/templates/macos-build-and-unit-test.yml

wilbaker

Left a few questions after trying out the new code for a bit on the Mac.

Scripts/Mac/RunUnitTests.sh

wilbaker · 2019-10-31T18:56:09Z

Scripts/Mac/BuildScalarForMac.sh

-
-echo 'Running Scalar unit tests...'
-$Scalar_PUBLISHDIR/Scalar.UnitTests || exit 1
+dotnet publish $SCALAR_SRCDIR/Scalar.sln --runtime osx-x64 --configuration $CONFIGURATION || exit 1


With the move to .NET Core 3.0, should we still be able to build using VS for Mac? Or will we have to use the script?

I'm seeing this error when I try in the IDE:

Related, what is the best way to debug Scalar on Mac now? I used to be able to do the following:

Build Scalar in VS for Mac

Right-click on the project I want to run, and select Run With -> Custom Configuration ...

Point VS at the Published binary I want to test

Start debugging

But now that VS for Mac is unable to build, it does not appear to let me start debugging this way.

What do you use to debug .NET Core 3 applications on Mac?

What do you use to debug .NET Core 3 applications on Mac?

Related, how can we run the unit tests from an IDE on Mac?

Related, how can we run the unit tests from an IDE on Mac?

To clarify, all I'm looking for is the best way to debug the unit tests on Mac. With the switch to using dotnet test I'm not sure what the best process is for doing so.

I'm seeing this error when I try in the IDE:

Ah.. this is great (and fixable).. So VS for Mac still uses Mono MSBuild to do it's 'in IDE' builds, not the dotnet CLI. The custom build tasks I have currently target the .NET Core RoslynCodeTaskFactory. I have an update that I'll push to fix this, and to select the correct factory depending on the MSBuild runtime.

Related, what is the best way to debug Scalar on Mac now? I used to be able to do the following:

Build Scalar in VS for Mac

Right-click on the project I want to run, and select Run With -> Custom Configuration ...

Point VS at the Published binary I want to test

Start debugging

You do largely the same thing. If you just want to debug the project, right-click it in the Solution Pad and click ''Start Debugging Project":

To specify command-line arguments, use 'Run With' > 'Custom Configuration' again and pick 'Start Project' and 'Run Action: Debug' (and add your arguments or environment variables):

Related, how can we run the unit tests from an IDE on Mac?

To clarify, all I'm looking for is the best way to debug the unit tests on Mac. With the switch to using dotnet test I'm not sure what the best process is for doing so.

Open the Unit Tests Pad under View > Pads > Unit Tests, select a test to run or debug!

You can also hit 'run all' and see all the lovely green ticks ✅ (hopefully)!
You will also see that the ignored tests are the Windows ones on Mac.

@mjcheetham I tried grabbing the latest code from your branch (e3b2d34), and I'm now seeing a different build error in VS for Mac which prevents me from running the unit tests:

Build FAILED.

/Users/wilbaker/Repos/Scalar/src/Scalar.Common/Scalar.Common.csproj(24,29): error MSB4064: The "PackagePath" parameter is not supported by the "GetGitInstallerVersion" task. Verify the parameter exists on the task, and it is a settable public instance property.

/Users/wilbaker/Repos/Scalar/src/Scalar.Common/Scalar.Common.csproj(24,5): error MSB4063: The "GetGitInstallerVersion" task could not be initialized with its input parameters.
0 Warning(s)
2 Error(s)

Any ideas? I have tried nuking and building again but that did not seem to help.

I confirmed the following worked as expected on Mac:

Build Scalar from Terminal using BuildScalarForMac.sh

Run unit tests with RunUnitTests.sh (no warnings! 🎉 )

Run the install script that is placed in Scalar.Distribution.Mac

In case it matters, this is what dotnet --version reports:

~/Repos/Scalar/src/Scripts/Mac>dotnet --version 3.0.100

@mjcheetham as discussed offline, no need to block on VS for Mac support.

I was able to debug the unit tests using VSCode and the following approach:

Open Terminal Window in VSCode

export VSTEST_HOST_DEBUG=1

dotnet test

Attach to the PID reported in terminal:

Host debugging is enabled. Please attach debugger to testhost process to continue. Process Id: 23870, Name: dotnet

If you know a better approach, please let me know 😄

Scalar.UnitTests/Program.cs

wilbaker · 2019-11-01T23:30:24Z

Scripts/Mac/CreateScalarDistribution.sh


-set -e
+. "$(dirname ${BASH_SOURCE[0]})/../InitializeEnvironment.sh"


I think this may need to be updated to:

"$(dirname ${BASH_SOURCE[0]})/InitializeEnvironment.sh"

When I ran this as it is now there was an error that InitializeEnvironment.sh could not be found:

~/Repos/Scalar/src/Scripts/Mac>./CreateScalarDistribution.sh ./CreateScalarDistribution.sh: line 3: ./../InitializeEnvironment.sh: No such file or directory No need to run this script anymore! Run 'BuildScalarForMac.sh' and look in: /Scalar.Distribution.Mac/dist

I could also just remove this since it's just here to give a nice "error" to people with muscle memory. 😉

derrickstolee

Sorry for merging #213 too early. I should have waited.

Here is a potential merge for you: 9827bdb

Scalar.Distribution.Windows/InstallScalar.template.bat

derrickstolee · 2019-11-06T18:04:55Z

Scalar.Distribution.Windows/InstallScalar.template.bat

+ECHO.
+ECHO ==============================
+ECHO Installing Git for Mac for Scalar
+%SCALAR_DISTRIBUTION_ROOT%\Git\%GIT_INSTALLER_EXE% /DIR="C:\Program Files\Git" /NOICONS /COMPONENTS="ext,ext\shellhere,ext\guihere,assoc,assoc_sh" /GROUP="Git" /VERYSILENT /SUPPRESSMSGBOXES /NORESTART || EXIT /B 1


This step is failing in the Windows functional tests. Should we change the "very silent" option until we know more about what's going on?

derrickstolee

Just making it clear that you can merge as soon as you have passing builds. Let's get this in and then see what small problems we have afterwards.

We get rate limited querying the GitHub API for the release assets to download GCM Core for Mac. Replace this with a simple hardcoding of the GCM Core macOS installer package URL.

Workaround a bug in the NuGet restore logic shipped with the dotnet CLI. NuGet/Home#8692

derrickstolee · 2019-11-07T17:55:33Z

Thanks for your hard work here, @mjcheetham! I just merged to unblock the other PRs.

mjcheetham marked this pull request as ready for review October 29, 2019 13:19

mjcheetham changed the title ~~[WIP] Port to .NET Core 3.0 and revamp engineering/build/test~~ Port to .NET Core 3.0 and revamp engineering/build/test Oct 29, 2019

derrickstolee added this to the MVP milestone Oct 29, 2019

derrickstolee added the engineering label Oct 29, 2019

wilbaker reviewed Oct 31, 2019

View reviewed changes

.azure-pipelines/templates/macos-build-and-unit-test.yml Show resolved Hide resolved

wilbaker suggested changes Oct 31, 2019

View reviewed changes

wilbaker mentioned this pull request Oct 31, 2019

Remove 'scalar status' verb #210

Merged

wilbaker reviewed Nov 1, 2019

View reviewed changes

wilbaker approved these changes Nov 4, 2019

View reviewed changes

derrickstolee approved these changes Nov 4, 2019

View reviewed changes

derrickstolee reviewed Nov 6, 2019

View reviewed changes

Scalar.Distribution.Windows/InstallScalar.template.bat Outdated Show resolved Hide resolved

derrickstolee reviewed Nov 6, 2019

View reviewed changes

derrickstolee mentioned this pull request Nov 6, 2019

Update Git to 2.24.0 #192

Merged

derrickstolee approved these changes Nov 6, 2019

View reviewed changes

wilbaker mentioned this pull request Nov 6, 2019

LooseObjectsStep: simplify code for getting directory paths #218

Merged

mjcheetham added 9 commits November 7, 2019 16:05

Upgrade common/pre build items and targets

0a57c88

Upgrade Scalar project

3ab834a

Upgrade Scalar.Common project

4f19ba5

Upgrade Scalar.Platform.* projects

15d4427

Upgrade Scalar.Mount project

6c3def3

Upgrade Scalar.Service project

7127617

Upgrade Scalar.Upgrader project

60b19f2

Upgrade Scalar.Tests project (rename: test infra)

a7ea6a3

Upgrade Scalar.UnitTests project

2b6c330

mjcheetham added 15 commits November 7, 2019 16:05

Integrate Scalar.Notification Xcode project

23c8dfb

Upgrade Scalar.Service.UI (Windows) project

013b7f7

Upgrade Scalar.FunctionalTests project

08c1d87

Upgrade Scalar.Installer.Windows project

5adb59b

Upgrade Scalar.Installer.Mac project

6dd6fce

Upgrade solution file

d9d768f

Add ruleset file

6209515

Fix .NET Core/Framework API differences

7515fdc

Add workaround for missing pipe security APIs

82351d9

Workaround JSON.NET breaking change with .NET Core 3

ae79aae

Update gitignore for JetBrains IDEs

cf50b5d

Add distribution projects for Mac/Windows

25cf471

Update build scripts and CI

31d4dab

Replace smart-download of GCM Core with dumb-download

9325c72

We get rate limited querying the GitHub API for the release assets to download GCM Core for Mac. Replace this with a simple hardcoding of the GCM Core macOS installer package URL.

Workaround dotnet CLI/NuGet restore issue

ad94ae4

Workaround a bug in the NuGet restore logic shipped with the dotnet CLI. NuGet/Home#8692

derrickstolee merged commit 7b6f1cb into microsoft:master Nov 7, 2019

mjcheetham deleted the refactor3 branch November 7, 2019 18:19

wilbaker mentioned this pull request Nov 7, 2019

Remove Scalar.UnitTests.Windows #152

Closed

derrickstolee mentioned this pull request Nov 13, 2019

Finish conversion to .NET Core #4

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port to .NET Core 3.0 and revamp engineering/build/test #201

Port to .NET Core 3.0 and revamp engineering/build/test #201

mjcheetham commented Oct 29, 2019

mjcheetham commented Oct 30, 2019

mjcheetham commented Oct 30, 2019

derrickstolee commented Oct 30, 2019

derrickstolee commented Oct 30, 2019

wilbaker commented Oct 30, 2019 •

edited

Loading

mjcheetham commented Oct 31, 2019 •

edited

Loading

wilbaker left a comment

wilbaker Oct 31, 2019

wilbaker Oct 31, 2019

wilbaker Oct 31, 2019

mjcheetham Nov 1, 2019 •

edited

Loading

mjcheetham Nov 1, 2019

mjcheetham Nov 1, 2019

wilbaker Nov 1, 2019

wilbaker Nov 1, 2019

wilbaker Nov 4, 2019

wilbaker Nov 1, 2019

mjcheetham Nov 4, 2019

derrickstolee left a comment

derrickstolee Nov 6, 2019

derrickstolee left a comment

derrickstolee commented Nov 7, 2019


		set -e
		. "$(dirname ${BASH_SOURCE[0]})/../InitializeEnvironment.sh"

Port to .NET Core 3.0 and revamp engineering/build/test #201

Port to .NET Core 3.0 and revamp engineering/build/test #201

Conversation

mjcheetham commented Oct 29, 2019

mjcheetham commented Oct 30, 2019

mjcheetham commented Oct 30, 2019

derrickstolee commented Oct 30, 2019

derrickstolee commented Oct 30, 2019

wilbaker commented Oct 30, 2019 • edited Loading

mjcheetham commented Oct 31, 2019 • edited Loading

wilbaker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjcheetham Nov 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derrickstolee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derrickstolee left a comment

Choose a reason for hiding this comment

derrickstolee commented Nov 7, 2019

wilbaker commented Oct 30, 2019 •

edited

Loading

mjcheetham commented Oct 31, 2019 •

edited

Loading

mjcheetham Nov 1, 2019 •

edited

Loading