-
Notifications
You must be signed in to change notification settings - Fork 790
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for Generative Type Providers when targeting .NET Core 2.0 #2406
Comments
Also would like to add this comment here from @KevinRansom on #2400:
|
and from @cartermp
|
Do we know why What alternatives we have if I think that I have a TP that should be generative - SwaggerProvider. |
The issue is tracked here |
Ok, found issue created by Kevin - https://github.com/dotnet/coreclr/issues/1709 |
Duplicating my comments from the other issue this is linked from: One of the big things is that generative type providers service the wider ecosystem not just F#, they are the most important aspect in that respect. Unchecked quotations for generative providers is also something that sorely needed to avoid having to reflectively call quotation builders to build the type provider. |
How about the successor(?) of IKVM.Reflection.Emit from Jeroen Frijters as his name already mentioned in the System.Reflection.Emit.AssemblyBuilder.Save corefx issue [1]? A preview release are available from NuGet. "Managed.Reflection is a fully managed replacement of System.Reflection and System.Reflection.Emit. Unlike System.Reflection, it is not tied to the runtime it is running on. It targets .NET Standard 1.3, which means it runs on .NET Core 1.0 and .NET Framework 4.6 and up." Repository: [1] https://github.com/dotnet/corefx/issues/4491#issuecomment-189756092 |
@zpodlovics considering the comments I think (after discussing with the maintainers of course) such a PR making it work would be welcome... Its just nothing they have time for right now |
Ive partially patched reflection stuff with |
@dsyme / etc Would a PR that used a dotnetcore version of ilvm reflection be accepted here? |
@7sharp9 You mean Managed.Reflection? |
Could Erasing Type Providers already work (in theory)? IDE (e.g. Visual Studio) is run on non-dotnet-core mode and the compiled code should be run on dotnet core / standard? |
@Thorium Yes, in theory Erasing TPs should work on .NET Core 2.0, and thus could target .NET Standard 2.0. That assessment still has to be done. Just last week, support for .NET Core 2.0 has been merged as in-band for the .NET Core SDK, so the assessment and proper work here can now be done. |
@sergey-tihon Yeas I meant Managed.Reflection, with that project hanging in the balance Im in two minds whether to use it as a replacement. |
To get generatives working do we just have to re-write providedtypes.fs to use an alternative @cartermp / @sergey-tihon |
|
Well, can I help here somehow? |
I believe we are blocked on having the compiler target .NET Core 2.0 |
What's current status of the issue? There are too many issues related to TP support in .Net Core/Standard, not easy to pick needed info. Thanks! |
No update on status at the moment. We're looking at this once .NET Core 2.0 is released. #3303 is a workaround for now. |
At some point there was quotation support in Fable but I removed it because it was flaky and I didn't find many practical uses for it. About System.Type and reflection in general, there's some limited support in Fable. However, I don't think this is an issue. Fable 0.7 was compatible with TPs (I'm only talking about erasure TPs here) and FCS already gave you the erasure code in the AST, so I didn't have to worry about quotations, etc (FCS did crash sometimes when accessing the AST generated by TPs but most of the cases were fixed). The main issue with TPs and Fable was that most of the TPs generated code that used BCL APIs not supported by Fable (IO, networking). That's the reason Fable and TPs haven't had a nice story together so far. Example of a type provider that was compatible with Fable at the time. |
Hmmm... that's probably with the F# compiler running as a .NET Core or .NET Framework component. I think the Fable-compiled FSharp.CompilerService was compiled without TP support, cc @ncave |
Yeah, sorry. I was talking about running Fable on .NET. When compiling FCS to JS I'm not sure, maybe it makes sense to have a specific mechanism for Fable that emits JS directly, similar to how the |
I'm more interested if we could fundamentally improve the situation with different APIs. For example does fable even have to know about type providers or could that be an implementation detail? |
@dsyme @7sharp9 @matthid I'd like to discuss the current difficulties in generative type provider support for .NET Core. I am still new in the land of type providers, so I could ask really silly things, but could you please give me the directions? Please correct me if I'm saying anything wrong. We've done a small research with @Dolfik1 on that matter, and have came to the following conclusions.
If these conclusions are correct (and saving the generated assembly is really the biggest problem), then we could really come up with some hacky (hopefully not very intrusive) solution to make it work. Currently I think about using Mono.Reflection to parse the dynamic assembly generated by the type provider (✓ have verified that it works on .NET Core 2.0) and then Mono.Cecil to reconstruct and save the assembly into byte array. I have some questions regarding that.
|
I found the System.Reflection.Metadata library, it seems that we can generate assemblies using this library only. |
While I can't answer your questions (@dsyme is much better suited for that) i doubt a dependency on third party libraries would be accepted, as that would mean that FSharp.Core will get a dependency outside of BCL. Though I'm glad someone is looking into this, it's a much needed requirement ;). |
@abelbraaksma if that's the biggest issue, we could solve that in one way or another. One option is to either port or rewrite in F# the corresponding parts of Mono.Cecil/Reflection and/or System.Reflection.Metadata. But, if I'm following Don correctly, in this comment he suggests us to use the PS: please note that we're mostly talking about Type Provider SDK and F# compiler code, and not the FSharp.Core. Any provider-related changes will reside in either the compiler, the TP SDK, or both. I can't see how they could leak to FSharp.Core. |
@ForNeVeR so far I see the situation we should pick everything existing up. Still though, dependencies are only being added to the F # compiler, not F# Core (how did this come up?). Your conclusion make sense to me. We should focus on fixing type providers... I hope to see a response soon from @dsyme (as soon as he can spend time for our questions :) ) |
@ForNeVeR Yes, your analysis is correct
Yes, I'm wondering if we should just go down that route. There are pros and cons but it seems like a good option to be free of dependencies.
That's not the case: it is the FSharp.TypeProviders.SDK bits we are talking about, not FSharp.Core |
Alright, then. We have the following plan. @Dolfik1 is now implementing a prototype of a I propose we first try to convert any simple type provider with our prototype. If that will work (and we won't encounter any blockers), then we could proceed with reimplementation of the prototype in F# with Is that okay? |
My bad, sorry for the fog. As @dsyme stated:
Which moots my comment ;) (and luckily so). |
@Dolfik1 can you keep me in the loop on the results of your prototype ... it is an interesting proposal. |
@KevinRansom Of course. At the moment we have been able to generate a basic assembly, it is not yet valid, but at least dotPeek shows that all the metadata and the IL code are in place. |
@ForNeVeR I'm not sure it's necessary to do a prototype via System.Reflection.Metadata. I think we should just add the appropriate IL generator and binary writer to ProvidedTypes.fs https://github.com/fsprojects/FSharp.TypeProviders.SDK/tree/master/src |
@KevinRansom asked offline
Yeah it’s a good question. First the usual background… Historically generative type providers were only introduced to allow other code generators (e.g. C# compiler, or svcgen.exe etc) to be hosted as TPs, with a thin erasing wrapper. In that scenario “some other tool” did the codegen. Over time the community (I think it was @v2m acting in community mode initially) added an ability to take this way of specifying erasing type providers (the TP SDK) and repurpose it as a way to author generative TPs, basically the On the positive side, the code was outside our repo, and we didn’t need to know anything about it - if people wanted a new feature or to expand the range of quotations allowed then they just sent a simple PR, and all the QA was on the TP SDK. On the negative side it used Reflection Emit so can’t do cross-targeting. But the approach basically worked. So to your question – yes, in theory it would be possible to abandon this approach and instead have the compiler give each TP a utility method in the TypeProviderConfig, e.g.
which would internally to the compiler be implemented as either
Or
Either way this would not use reflection emit and would reuse the ilwrite in the compiler. However, this still means implementing either Now, for technical reasons However, in reality the code generated by TPs tends to be very simple, and TPs tend to call into runtime library code for anything complex. That’s why @v2m's “do assembly generation in the TP SDK” approach worked well enough. After all, the generated code doesn’t really require full optimization, nor tailcalls, nor any magic done in IlxGen.fs. And with the TP SDK the community could iterate fast. And there are some advantages to disincentivizing people from generating complex code using TPs anyway - e.g. there's no debug story for that code. Anyway, all these are reasons as to why redoing some compilation logic in the TP SDK has historically made sense. I’ll think it over more, but it still feels to me that the most simple path to unblock is to prototype the p.s. One thing that could force us to go down the |
@dsyme thanks so much for your comment. It's both interesting and valuable. Regarding to the "prototype" based on System.Reflection.Metadata: @Dolfik1 just was too enthusiastic about that approach, so he's already done a large part of work, and I feel bad about abandoning it. We could eventually even make this prototype an externally-useful (e.g. not tied to F# type providers) artifact of the research. Also, it seems that we're almost done with the prototype, and it could give us an insight on any further issues we'll encounter while integrating binary writer with the Type Provider SDK. I agree with your rationale, and we'll try to follow that path soon (probably in a week or two). If anybody wants to begin integrating |
@dsyme please also consider fsharp/fslang-suggestions#154 |
@ForNeVeR I'm trying to understand what the role of the AssemblyGenerator would be. It seems it assumes the assembly coming in has code backed by IL, e.g. here. But in that case the Assembly would already normally have corresponding bytes in its existing on-disk image. For generative F# type providers authored using the TPSDK, the situation is that we have an artificial |
I see what you're asking, but the TAST is not a public data structure. Currently, generative type providers authored using the TPSDK specify the code contents of a generated assembly using quotations, e.g. |
@dsyme at the same time, to me it looked like a generative TP would emit the code into the dynamically generated assembly: https://github.com/fsprojects/FSharp.TypeProviders.SDK/blob/36c892674c276265f2c028b8a2e36a99445acb3e/src/ProvidedTypes.fs#L7495-L7496 ...and then get its' bytes using the standard Initially I thought that's the essential part of type provider functioning, but now I'm just a bit confused. |
@ForNeVeR Yes, that's correct - though at the start of the process the code for each method is an F# quotation - see that same file |
@ForNeVeR The TPSDK has now been updated with support for cross-targeting generative type providers by integrating the binary writer. You should now be able to build generative type providers for .NET Core though that has not yet been specifically tested (and might require an updated F# compiler for .NET Core). |
I'm closing this as the TPSDK has now been updated to support generative type providers when targeting .NET Core If targeting .NET Standard or .NET Core and using type providers you still need to apply the workaround specified here: #3303. (That is a blunt workaround for issue #3736 which needs to be resolved (and type providers re-published) before type providers can be created which will load into .NET Core-based tooling) |
From #2400:
The text was updated successfully, but these errors were encountered: