Skip to content
This repository has been archived by the owner on Jan 23, 2023. It is now read-only.
/ corefx Public archive

Update required software and developer workflow #30919

Merged
merged 5 commits into from
Aug 23, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Documentation/building/advanced-inner-loop-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

This tutorial describes how to build and run application code that targets self-compiled .NET Core binaries without using Visual Studio, the .NET Core SDK Host (`dotnet.exe`) or a project file (e.g. `csproj`). Follow these steps to quickly validate changes you've made in the product e.g. by running benchmarks or tests on it.

If you are on Windows you currently need to use the `Developer Command Prompt for VS 2017` to build corefx/coreclr! For the sake of completeness, we have placed our repositories under `d:\git\`.
For the sake of completeness, we have placed our repositories under `d:\git\`.

## Compile corefx with self-compiled coreclr binaries
If you've made changes to coreclr make sure to also build it and pass its binaries to corefx.
Expand Down
12 changes: 6 additions & 6 deletions Documentation/building/code-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,16 +56,16 @@ This will do the build and testing as with the normal ```build```, but it will r

You can also build and test with code coverage for a particular test project rather than for the whole repo. Normally to build and test a particular test suite, from the same directory as that test suite's .csproj, you'd run:

msbuild /t:BuildAndTest
dotnet msbuild /t:BuildAndTest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan is to keep both msbuild and dotnet msbuild working so I wonder if we should simply make a note of that somewhere and just reference msbuild in the instructions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question, why should we continue advertising the Desktop msbuild if using the dotnet msbuild is (finally) possible in corefx (from a feature perspective). I don't see any compelling reason to stick with the Desktop flavor. We now already require the .NET Core SDK to be installed which means in 99% of the cases dotnet will be in the path.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a fair question. Last I checked even if you called dotnet msbuild on a windows machine with full msbuild it would use the full version so I think we are going to need to keep our builds working for both and in such a case we should allow folks to use either. In the past I know the performance was much better with full msbuild over core msbuild but I'm not sure if that is still the case or not, but another potential reason.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the past I know the performance was much better with full msbuild over core msbuild but I'm not sure if that is still the case or not, but another potential reason.

I discussed that with @eerhardt offline and when he switched building from the project root with dotnet msbuild he measured performance and it was equal: #30675 (comment). I agree that the dotnet msbuild was slower in the past but I believe it's now on par with the Desktop flavor.

Last I checked even if you called dotnet msbuild on a windows machine with full msbuild it would use the full version

That's interesting. Would be great if we could find out if that's still the case. I just did a quick invocation of dotnet msbuild:

C:\git\corefx\src\System.Text.RegularExpressions>dotnet msbuild /t:Rebuild
Microsoft (R) Build Engine version 15.7.179.6572 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

and the Task Manager shows .NET Core Host. I believe it's not using the Desktop msbuild anymore underneath but we should verify.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I thought dotnet msbuild would use the full msbuild your correct that doesn't seem to be the case from some quick testing. However the full msbuild is still used when building from VS, so we will need to maintain our build infrastructure to work with both. So it is really just a matter of which do we advertise in our docs. I'm OK advertising the dotnet msbuild and just adding a note that says you can also call msbuild on our projects.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I'd still be against mandatory pinning for local builds, but my objection wouldn't be as visceral.

I think official builds should be fully repeatable, but freedom to bring my own tools for unofficial, local work is valuable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but freedom to bring my own tools

This is also SUPER important for source-build and platform boot-strapping purposes. It is great that official builds mandate that this specific set of tools MUST be used. But we also need a way where I can tell this build "this is the toolset to use".

That way we can bootstrap source-build by building once using a pre-built toolset. Then discarding the pre-built toolset, and turning around and re-building using the toolset I just built from source. That's a requirement in most distros that I've seen.

And for platform boot-strapping, it is obvious that I can't use some already built version of the SDK to build on "PlatformX", when there is no existing version that can run on "PlatformX".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow that escalated quickly. I recommend creating an issue to discuss updating the SDK via maestro PRs more frequently further and keep the discussion here streamlined on how to invoke the same SDK for both root and individual project builds. Ideally with dotnet build to be consistent as @stephentoub also noted.

I guess the right approach for now is to modify the path to /Tools/dotnetcli/. I'll update the docs and try it myself locally to check if I run into any errors.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nguerrera if you have better ideas now would be a good time to voice them in the arcade repo

I will do so.

but there will always be tension there as we cannot dogfood the latest sdk build due to other requirements such as source-build.

This actually makes my point much more concrete. We have external reasons to ensure that we can build with some particular toolset, but those shouldn't prevent local dogfooding with newer bits. CI can validate that I haven't taken a dependency on something too new without cutting off my ability to dogfood the new stuff.

But we also need a way where I can tell this build "this is the toolset to use"

Yes, and some one-off gesture shouldn't be needed to dogfood toolsets locally. Source-build will do what it needs to do, but nobody will bother opting in to dogfooding if it doesn't happen naturally/automatically IMHO.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


To do so with code coverage, append the ```/p:Coverage=true``` argument:

msbuild /t:BuildAndTest /p:Coverage=true
dotnet msbuild /t:BuildAndTest /p:Coverage=true

The results for this one library will then show up in the aforementioned index.htm file. For example, to build, test, and get code coverage results for the System.Diagnostics.Debug library, from the root of the repo one can do:

cd src\System.Diagnostics.Debug\tests\
msbuild /t:BuildAndTest /p:Coverage=true
dotnet msbuild /t:BuildAndTest /p:Coverage=true

And then once the run completes:

Expand All @@ -76,11 +76,11 @@ And then once the run completes:
Some of the libraries for which contracts and tests live in the corefx repo are actually fully or partially implemented in the core runtime library in another repo, e.g. the implementation that backs the System.Runtime contract is in System.Private.CoreLib.dll in either the coreclr or corert repo. To run coverage reports for these projects, you need to build System.Private.CoreLib locally from the coreclr repo. To get coverage of System.Private.CoreLib while running the tests for a particular library:

1. Follow the steps outlined at [Testing with Private CoreClr Bits](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md#testing-with-private-coreclr-bits). Make sure to include the optional steps listed as being required for code coverage.
2. Add /p:CodeCoverageAssemblies="System.Private.CoreLib" to the previously discussed msbuild command, e.g. msbuild /t:BuildAndTest /p:Coverage=true /p:CodeCoverageAssemblies="System.Private.CoreLib"
2. Add /p:CodeCoverageAssemblies="System.Private.CoreLib" to the previously discussed dotnet msbuild command, e.g. dotnet msbuild /t:BuildAndTest /p:Coverage=true /p:CodeCoverageAssemblies="System.Private.CoreLib"

The build and test projects take care of copying assemblies and PDBs as needed for coverage runs. The resulting code coverage report should now also include details for System.Private.CoreLib.

Note: as of 10/2017 OpenCover, the default coverage tool, requires PDBs to be Windows PDBs - the needed conversions are automatically performed by coverage runs. You can determine if it is a Windows PDB by doing 'more System.Private.CoreLib.pdb', if it begins with 'Microsoft C/C++ MSF 7.00' it is a Windows PDB. If you need a Windows PDB the Pdb2Pdb tool will convert (or you can do a msbuild /t:rebuild /p:DebugType=full).
Note: as of 10/2017 OpenCover, the default coverage tool, requires PDBs to be Windows PDBs - the needed conversions are automatically performed by coverage runs. You can determine if it is a Windows PDB by doing 'more System.Private.CoreLib.pdb', if it begins with 'Microsoft C/C++ MSF 7.00' it is a Windows PDB. If you need a Windows PDB the Pdb2Pdb tool will convert (or you can do a dotnet msbuild /t:rebuild /p:DebugType=full).

## Cross-platform Coverage
As of 07/2018 CoreFx is only able to get coverage information on Windows. To correct this we are experimenting with [coverlet](https://github.com/tonerdo/coverlet).
Expand All @@ -103,7 +103,7 @@ On Unix just specifying `/p:Coverage=True` triggers the usage of coverlet. Howev
1. Install the [dotnet/cli global tool reportgenerator](https://www.nuget.org/packages/dotnet-reportgenerator-globaltool), add it to the PATH if not automatically added by dotnet/cli (it can be only for the current session)
2. On Linux install the Arial font required for report generation: `sudo apt-get install ttf-mscorefonts-installer`

After that you request a coverage run using either `dotnet msbuild` or the `msbuild.sh` from a test folder, e.g.:
After that you request a coverage run from a test folder, e.g.:

```
dotnet msbuild /t:RebuildAndTest /p:Coverage=True
Expand Down
50 changes: 17 additions & 33 deletions Documentation/building/windows-instructions.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,11 @@
Building CoreFX on Windows
==========================

You can build .NET Core either via the command line or by using Visual Studio.

## Required Software

1. **Visual Studio** must be installed. Supported versions:
* [Visual Studio 2015](https://www.visualstudio.com/vs/older-downloads/) (Community, Professional, Enterprise). The Community version is completely free.
* [Visual Studio 2017](https://www.visualstudio.com/downloads/) (Community, Professional, Enterprise). The Community version is completely free.
2. **[CMake](https://cmake.org/)** must be installed from [the CMake download page](https://cmake.org/download/#latest) and added to your path.

### Visual Studio 2015

* [Visual Studio 2015 Update 1](https://www.visualstudio.com/en-us/news/vs2015-update1-vs.aspx) is required.
* You must select **Programming Languages | Visual C++ | Common Tools for Visual C++ 2015** while installing VS 2015 (or modify your install to include it).
1. **[Visual Studio 2017](https://www.visualstudio.com/downloads/)** (Community, Professional, Enterprise) with the latest update must be installed. The Community version is completely free.
2. **[.NET Core SDK](https://www.microsoft.com/net/download/windows)** >= v2.1.401 must be installed which will add the `dotnet` CLI to your path.
3. **[CMake](https://cmake.org/)** must be installed from [the CMake download page](https://cmake.org/download/#latest) and added to your path.

### Visual Studio 2017

Expand Down Expand Up @@ -42,7 +34,6 @@ The following are the minimum requirements:
* Windows 10 SDK or Windows 8.1 SDK
* Visual Studio C++ Core Features
* VC++ 2017 v141 Toolset (x86, x64)
* MSBuild
* .NET Framework 4.6 Targeting Pack
* Windows Universal CRT SDK
* VC++ 2015.3 v140 Toolset (x86, x64)
Expand All @@ -57,11 +48,11 @@ This will install all the components needed.

Note that you will need to adjust the install path to reflect your version, "Community", "Professional", "Enterprise" or "Preview"

## Building From the Command Line
For the best possible experience make sure to have the latest version of Visual Studio 2017 installed.

You must use the Developer Command Prompt. It will have a name like "Developer Command Prompt for VS 2017" or similar in your start menu.
## Building From the Command Line

From a (non-admin) Developer Command Prompt window:
From a (non-admin) Command Prompt window:

- `build.cmd` - Will cause basic tool initialization and build the default configuration for refs, libs, and packages.
- `build-tests.cmd` - Will build and run tests for the default configuration.
Expand All @@ -70,8 +61,7 @@ For information on different configurations see [project-guidelines](../coding-g

**Note**: Before working on individual projects or test projects you **must** run `build.cmd` from the root once before beginning that work. It is also a good idea to run `build.cmd` whenever you pull a large set of unknown changes into your branch.

Visual Studio Solution (.sln) files exist for related groups of libraries. These can be loaded to build, debug and test inside
the Visual Studio IDE.
Visual Studio Solution (.sln) files exist for related groups of libraries. These can be loaded to build, debug and test inside the Visual Studio IDE.

Note that when calling the script `build.cmd` attempts to build both the native and managed code.
Only use it when the parameters that you are passing to the script apply for both components. Otherwise, use the scripts `build-native.cmd` and `build-managed.cmd` respectively.
Expand All @@ -91,17 +81,16 @@ For more details, or to test an individual project, see the [developer guide top

### Debugging tests in Visual Studio

1. Install VS 2015 Preview or later including Web Developer Tools
2. Open solution of interest in VS 2015
3. Right click test project and select 'Set as startup project'
4. Set breakpoint appropriately
5. F5 (Debug)
1. Open solution of interest
2. Right click test project and select 'Set as startup project'
3. Set breakpoint appropriately
4. F5 (Debug)

### Debugging NETFX tests in Visual Studio

Once you've built the source code for netfx from the root (`build.cmd -framework:netfx`) follow these steps:

1. Build test project with the following parameters `msbuild /t:buildandtest /p:targetgroup=netfx /p:testdebugger=devenv.exe`. This will open Visual Studio with the runner as startup project and its corresponding arguments.
1. Build test project with the following parameters `dotnet msbuild /t:buildandtest /p:targetgroup=netfx /p:testdebugger=devenv.exe`. This will open Visual Studio with the runner as startup project and its corresponding arguments.
2. Open project properties and fill in the next information:
* Debugger Type -> Managed (v4.6, v4.5, v4.0)
* Environment -> you need to add an environment variable as follows:
Expand All @@ -112,16 +101,11 @@ Once you've built the source code for netfx from the root (`build.cmd -framework
For advanced debugging using WinDBG see [Debugging CoreFX on Windows](https://github.com/dotnet/corefx/blob/master/Documentation/debugging/windows-instructions.md)

### Notes
* At any given time, the corefx repo might be configured to use a more recent compiler than
the one used by the most recent Visual Studio IDE release. This means the corefx codebase might
* At any given time, the corefx repo might be configured to use a [more recent compiler](../../../DotnetCLIVersion.txt) than
the one used by the installed .NET Core SDK. This means the corefx codebase might
be using language features that are not understood by the IDE, which might result in errors that
show up as red squiggles while writing code. Such errors should, however, not affect the actual compilation.

* Running tests from using the VS test explorer does not currently work after we switched to running on CoreCLR. [We will be working on enabling full VS test integration](https://github.com/dotnet/corefx/issues/1318) but we don't have an ETA yet. In the meantime, use the steps above to launch/debug the tests using the console runner.

* VS 2015 is required to debug tests running on CoreCLR as the CoreCLR
debug engine is a VS 2015 component.
show up as red squiggles while writing code. Such errors should, however, not affect the actual compilation.

* If the Xamarin PCL profiles are installed, the build will fail due to [issue #449](https://github.com/dotnet/corefx/issues/449). A possible workaround is listed [in the issue](https://github.com/dotnet/corefx/issues/449#issuecomment-95117040) itself.
* Running tests from using the VS test explorer does not currently work after we switched to running on CoreCLR. [We will be working on enabling full VS test integration](https://github.com/dotnet/corefx/issues/20627) but we don't have an ETA yet. In the meantime, use the steps above to launch/debug the tests using the console runner.

* If your build fails with "[...].dll - Access is denied" errors, it might be because Visual Studio/MSBuild is locking these files. Try shutting down `VBCSCompiler.exe`, `devenv.exe` and `MSBuild.exe` from the task manager before building again.
* If your build fails with "[...].dll - Access is denied" errors, it might be because Visual Studio/MSBuild is locking these files. Try shutting down `VBCSCompiler.exe` and `dotnet.exe` from the task manager before building again.
4 changes: 2 additions & 2 deletions Documentation/coding-guidelines/adding-api-guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ the implementation without compat concerns in future releases.
- Update the ProjectReferences in the respective pkgproj's to align with the build configurations in the
`<Library>\src\<Library>.builds` file. This may entail adding new references or removing references according to whatever changes were made to the .builds file.
- If assembly or package version is updated the package index needs to be updated by running
`msbuild <Library>/pkg/<Library>.pkgproj /t:UpdatePackageIndex`
`dotnet msbuild <Library>/pkg/<Library>.pkgproj /t:UpdatePackageIndex`

**Update tests**
- If changing target framework
Expand All @@ -93,7 +93,7 @@ the implementation without compat concerns in future releases.
- Set `TestTFMs` metadata to a list of target frameworks to run on, will generally match the `SupportedFramework` metadata in the pkgproj (ex: [tests\System.Runtime.builds](https://github.com/dotnet/corefx/blob/master/src/System.Runtime/tests/System.Runtime.Tests.builds#L8))
- If `TestTFMs` is empty it defaults to [netcoreapp1.0](https://github.com/dotnet/corefx/commit/57af41ef1439ad2e443e42d03d55d41613e4c02e#diff-cd0fc5e0bad8102e1a45aa7575bdd102R155)
- Add new test code following [conventions](https://github.com/dotnet/corefx/blob/master/Documentation/coding-guidelines/project-guidelines.md#code-file-naming-conventions) for new files to that are specific to the new target framework.
- To run just the new test configuration run `msbuild <Library>.csproj /t:RebuildAndTest /p:TargetGroup=<TargetGroup>`
- To run just the new test configuration run `dotnet msbuild <Library>.csproj /t:RebuildAndTest /p:TargetGroup=<TargetGroup>`

## FAQ
_**<a name="isnetstandard">Is your API part of netstandard?</a>**_
Expand Down
2 changes: 1 addition & 1 deletion Documentation/coding-guidelines/interop-guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ TargetsUnix is true for both OSX and Linux builds and can be used to include cod
You should not test the value of the OSGroup property directly, instead use one of the values above.

#### Project Files
Whenever possible, a single .csproj should be used per assembly, spanning all target platforms, e.g. System.Console.csproj includes conditional entries for when targeting Windows vs when targeting Linux. A property can be passed to msbuild to control which flavor is built, e.g. msbuild /p:OSGroup=OSX System.Console.csproj.
Whenever possible, a single .csproj should be used per assembly, spanning all target platforms, e.g. System.Console.csproj includes conditional entries for when targeting Windows vs when targeting Linux. A property can be passed to dotnet msbuild to control which flavor is built, e.g. `dotnet msbuild /p:OSGroup=OSX System.Console.csproj`.

### Constants
- Wherever possible, constants should be defined as "const". Only if the data type doesn't support this (e.g. IntPtr) should they instead be static readonly fields.
Expand Down
26 changes: 7 additions & 19 deletions Documentation/coding-guidelines/project-guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,41 +131,29 @@ Temporary versions are at https://github.com/dotnet/corefx/blob/dev/eng/src/Tool
- UAP F5 -> `uap-Windows_NT`

## Project configurations for VS
For each unique configuration needed for a given library project a configuration property group should be added to the project so it can be selected and built in VS and also clearly identify the various configurations.<BR/>

`<PropertyGroup Condition="'$(Configuration)|$(Platform)' == '$(OSGroup)-$(TargetGroup)-$(ConfigurationGroup)|$(Platform)'">`
For each unique configuration needed for a given library project a configuration entry separated by a ';' should be added to the project so it can be selected and built in VS and also clearly identify the various configurations.<BR/>

`$(TargetGroup)-$(OSGroup)-$(ConfigurationGroup)|$(Platform`
- Note that the majority of managed projects, currently all in corefx, $(Platform) is overridden to be AnyCPU.

`<Configurations>netcoreapp-Unix-Debug;netcoreapp-Unix-Release;netcoreapp-Windows_NT-Debug;netcoreapp-Windows_NT-Release;uap-Windows_NT-Debug;uap-Windows_NT-Release;uapaot-Windows_NT-Debug;uapaot-Windows_NT-Release</Configurations>`

####*Examples*
Project configurations for a pure IL library project which targets the defaults.
```xml
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|AnyCPU'" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Release|AnyCPU'" />
```
Project configurations with a unique implementation on Unix and Windows
```xml
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Unix-Debug|AnyCPU'" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Unix-Release|AnyCPU'" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Windows_NT-Debug|AnyCPU'" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Windows_NT-Release|AnyCPU'" />
<Configurations>netcoreapp-Unix-Debug;netcoreapp-Unix-Release;netcoreapp-Windows_NT-Debug;netcoreapp-Windows_NT-Release</Configurations>
```
Project configurations that are unique for a few different target frameworks and runtimes
```xml
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|AnyCPU'" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Release|AnyCPU'" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'uap101aot-Debug|AnyCPU'" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'uap101aot-Release|AnyCPU'" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'uap101-Debug|AnyCPU'" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'uap101-Release|AnyCPU'" />
<Configurations>netcoreapp-Windows_NT-Debug;netcoreapp-Windows_NT-Release;uap-Windows_NT-Debug;uap-Windows_NT-Release;uapaot-Windows_NT-Debug;uapaot-Windows_NT-Release</Configurations>
```

## Updating Configurations

We have a build task that you can run to automatically update all the projects with the above boilerplate as well as updating all the solution files for the libraries. Whenever you change the list of configurations for a project you can regenerate all these for the entire repo by running:

```
msbuild build.proj /t:UpdateVSConfigurations
dotnet msbuild build.proj /t:UpdateVSConfigurations
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eerhardt is the UpdateVSConfigurations buildtools task updated to respect SDK project configurations?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, he updated it in: dotnet/buildtools@94f18eb

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

```

If you want to scope the geneneration you can either undo changes that you don't need or you can temporally limit the set of projects or directories by updating the item set in the UpdateVSConfigurations target in https://github.com/dotnet/corefx/blob/master/build.proj
Expand Down
Loading