Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross-platform build-time code generation #3445

Merged
merged 3 commits into from
Oct 3, 2017

Conversation

ReubenBond
Copy link
Member

History: #3430 reverted #3424, this mostly reverts that revert.

The previous PR for this included a commit which moved the bootstrap codegen from net461 to netcoreapp2.0 and apparently that was a bad move (not entirely sure why, but I imagine it's because VS is using full framework MSBuild - haven't checked).

I added one extra commit: so that VS doesn't ever show the files from the bootstrap projects, we exclude all of those files during design time build. The major effect of this was duplicates when navigating around project files (eg, using ReSharper) - they're gone now.

Copy link
Contributor

@attilah attilah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ReubenBond yes, the previous VS problem was because of the usage of full .Net MSBuild by VS.

Now I tested this PR rebased on master and it worked from build.cmd and VS too!

LGTM

@attilah
Copy link
Contributor

attilah commented Sep 26, 2017

@ReubenBond Putting the changes from #3426 still make codegen fail with -1 error code.

If Orleans.Core.Abstractions is an indirect reference like in this case only OrleansRuntime and Orleans.Transactions are reference assemblies, then we're failing.

I think we need to check the flattened full list of references not just the direct references when checking Orleans reference in a codegen candidate assembly.

@sergeybykov sergeybykov added this to the 2.0.0-beta milestone Sep 26, 2017
@ReubenBond ReubenBond force-pushed the xplat-codegen branch 2 times, most recently from 50b2a07 to 985be22 Compare September 27, 2017 04:17
}

var config = new ClusterConfiguration();
var loggerFactory = new LoggerFactory();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using() so that it get's disposed after done codegening


var config = new ClusterConfiguration();
var loggerFactory = new LoggerFactory();
loggerFactory.AddProvider(new FileLoggerProvider("ClientGenerator.log"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, this is putting a ClientGenerator.log alonside the code. It should be in the intermediate folder, correct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I ported this over from ClientGenerator in master, but I didn't like it (because of that file being generated) so I'm using ConsoleLoggerProvider now, which has required pulling in a new package.


var config = new ClusterConfiguration();
var loggerFactory = new LoggerFactory();
loggerFactory.AddProvider(new FileLoggerProvider("ClientGenerator.log"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comments as above

#if NET461
var generatedCode = AppDomainCodeGenerator.GenerateCode(this.options);
#else
var generatedCode = this.GenerateCodeInternal();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a little odd how the conditional compilation was made, but it's correct. Ideally I would have the branching (but similar) code all in 1 file (or 2, but of the same shape). No big deal for this PR though

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right - it's definitely a bit odd. I'll address it, thanks

@@ -0,0 +1,58 @@
<Project TreatAsLocalProperty="CodeGenDirectory;IsCore;TaskAssembly;IsCore;OutputFileName">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what it does, but IsCore is duplicated here

@ReubenBond ReubenBond force-pushed the xplat-codegen branch 2 times, most recently from c6d9e9a to 5ae4e80 Compare September 28, 2017 14:49
@mmitche
Copy link
Member

mmitche commented Sep 28, 2017

@dotnet-bot test functional

@mmitche
Copy link
Member

mmitche commented Sep 28, 2017

@dotnet-bot test this please

@ReubenBond
Copy link
Member Author

There seems to be an issue with CI informing GitHub of build completion:

09:57:17 Notifying endpoint with url 'helix-int-notification-url'
09:57:17 ERROR: Failed to notify endpoint with url 'helix-int-notification-url'
09:57:17 java.net.MalformedURLException: no protocol: helix-int-notification-url

@ReubenBond
Copy link
Member Author

@dotnet-bot test functional

@ReubenBond
Copy link
Member Author

🚂choo choo

@@ -26,6 +26,7 @@
<GenerateDocumentationFile Condition="'$(Configuration)'=='Release'">true</GenerateDocumentationFile>
<NoWarn>$(NoWarn);1591</NoWarn>
<AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
<GenerateBindingRedirectsOutputType>true</GenerateBindingRedirectsOutputType>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this really needed at this very general level?

@jdom
Copy link
Member

jdom commented Sep 29, 2017

Please don't merge. Our load tests are failing with this change:

2017-09-29T18:12:30.9303492Z   Orleans-CodeGen - Generating file D:\build\agent\_work\7\s\LoadTests\LoadTestGrainInterfaces\obj\Release\net462\LoadTestGrainInterfaces.orleans.g.cs
2017-09-29T18:12:31.2033299Z   -- Code Generation FAILED -- 
2017-09-29T18:12:31.2043295Z   
2017-09-29T18:12:31.2043295Z   Exc level 0: System.BadImageFormatException: Could not load file or assembly 'System.Runtime.InteropServices.RuntimeInformation, Version=0.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)
2017-09-29T18:12:31.2043295Z      at Microsoft.Extensions.Logging.Console.ConsoleLogger..ctor(String name, Func`3 filter, Boolean includeScopes, ConsoleLoggerProcessor loggerProcessor)
2017-09-29T18:12:31.2043295Z      at Microsoft.Extensions.Logging.Console.ConsoleLoggerProvider.CreateLoggerImplementation(String name)
2017-09-29T18:12:31.2043295Z      at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
2017-09-29T18:12:31.2043295Z      at Microsoft.Extensions.Logging.LoggerFactory.CreateLoggers(String categoryName)
2017-09-29T18:12:31.2043295Z      at Microsoft.Extensions.Logging.LoggerFactory.CreateLogger(String categoryName)
2017-09-29T18:12:31.2043295Z      at Orleans.Runtime.LoggerWrapper..ctor(String name, ILoggerFactory loggerFactory)
2017-09-29T18:12:31.2043295Z      at Orleans.Runtime.LoggerWrapper`1..ctor(ILoggerFactory loggerFactory)
2017-09-29T18:12:31.2043295Z      at Orleans.Serialization.SerializationManager..ctor(IServiceProvider serviceProvider, IOptions`1 serializatonProviderOptions, ITraceConfiguration traceConfig, ILoggerFactory loggerFactory)
2017-09-29T18:12:31.2053292Z      at Orleans.CodeGeneration.CodeGenerator.GenerateSourceForAssembly(Assembly grainAssembly)
2017-09-29T18:12:31.2053292Z      at Orleans.CodeGeneration.CodeGenerator.GenerateCodeInAppDomain(CodeGenOptions options)
2017-09-29T18:12:31.2053292Z      at Orleans.CodeGeneration.CodeGenerator.GenerateCodeInAppDomain(CodeGenOptions options)
2017-09-29T18:12:31.2053292Z      at Orleans.CodeGeneration.CodeGenerator.GenerateCodeInternal(CodeGenOptions options)
2017-09-29T18:12:31.2053292Z      at Orleans.CodeGeneration.CodeGenerator.GenerateCode(CodeGenOptions options)
2017-09-29T18:12:31.2053292Z      at Orleans.CodeGeneration.Program.Main(String[] args)
2017-09-29T18:12:31.2053292Z   Exc level 1: System.BadImageFormatException: Could not load file or assembly 'file:///C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net462\ref\System.Runtime.InteropServices.RuntimeInformation.dll' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)
2017-09-29T18:12:31.2053292Z      at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
2017-09-29T18:12:31.2053292Z      at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
2017-09-29T18:12:31.2053292Z      at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark)
2017-09-29T18:12:31.2053292Z      at System.Reflection.Assembly.LoadFrom(String assemblyFile)
2017-09-29T18:12:31.2065404Z      at Orleans.CodeGeneration.CodeGenerator.<>c__DisplayClass5_0.<GenerateCodeInAppDomain>g__ResolveAssembly0(Object sender, ResolveEventArgs args)
2017-09-29T18:12:31.2065404Z      at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String assemblyFullName)
2017-09-29T18:12:31.2065404Z   Exc level 2: System.BadImageFormatException: Cannot load a reference assembly for execution.

@ReubenBond ReubenBond force-pushed the xplat-codegen branch 2 times, most recently from 226ab50 to dfbfe8d Compare September 30, 2017 04:53
@jdom
Copy link
Member

jdom commented Sep 30, 2017

@dotnet-bot test functional

1 similar comment
@jdom
Copy link
Member

jdom commented Oct 1, 2017

@dotnet-bot test functional

@jdom
Copy link
Member

jdom commented Oct 2, 2017

Could try load be a breaking change? For example if in full .net we were generating a serializer for some system type we were using, but with the new approach we end up trying to load a ref assembly and hence skipping it

@ReubenBond
Copy link
Member Author

When we fail to load (because it's a ref assembly), it will return null and use the default assembly resolution logic. It's not perfect because I don't know how we should handle conflicts between assemblies the CodeGen needs and assemblies the target needs - we will try the targets first and then fallback to default behavior.

@ReubenBond
Copy link
Member Author

Do you have any ideas for how to make this work, by the way? It's driving me insane.

@jdom
Copy link
Member

jdom commented Oct 2, 2017

No, I don't. ReflectionOnly was the safe approach, I don't know now. In the future we might want to use the Roslyn approach instead of loading compiled binaries, but not sure what's best for today's approach.

@ReubenBond
Copy link
Member Author

Were we ever performing reflection-only loading in the code generator? I think that aspect hasn't changed

@jdom
Copy link
Member

jdom commented Oct 2, 2017

ah, I don't know, I just assumed we were, and maybe it was why it wasn't failing there, but TBH I don't know... In that case then we might not be breaking existing users...

@ReubenBond ReubenBond force-pushed the xplat-codegen branch 3 times, most recently from 007a6f0 to d351216 Compare October 2, 2017 23:02
@jdom
Copy link
Member

jdom commented Oct 3, 2017

Not to do in this PR, as it looks like it's ready to merge, but since these .Bootstrap.csproj projects are largely duplicates of the others, which just minor differences: could we just import the other project and then modify what is changed? This way we'll avoid staleness

@jdom jdom merged commit 01eada9 into dotnet:master Oct 3, 2017
@ReubenBond
Copy link
Member Author

@jdom is it as simple as defining some overriding properties and importing the original project? It's worth experimenting with

@github-actions github-actions bot locked and limited conversation to collaborators Dec 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants