FSharp.Core 4.4.0.0 always gets referenced when using Type Provider

[latkin]

Originally opened by @ctaggart on Codeplex. Seems like a weird/severe enough problem that we should get eyes on it asap.

---

I've been troubleshooting this issue with Visual Studio 2015 Preview the last couple of nights and created a very simple FSharp.Data.SqlClient project to reproduce the issue. The project that uses the type provider ends up referencing both 4.4.0.0 and 4.3.1.0 of FSharp.Core. I was expecting it to only reference 4.3.1.0 that the project is set to. We figured out that we can still get it to run on 4.3.1.0 by doing a binding redirect down.

Please have a look at this issue:
fsprojects/FSharp.Data.SqlClient#79

It is super easy to reproduce:

PS> git clone https://github.com/ctaggart/Paket09x.git -b PeopleSql
PS> .\Paket09x\PeopleSql.sln

cheers,
Cameron

---

KevinRansom wrote Nov 19, 2014 at 9:16 AM [x]

Thanks for this report we will take a look to try to figure out what makes sense to do here.

Kevin

---

latkin wrote Nov 26, 2014 at 4:58 PM [x]

At the project level, everything is working as expected, and 4.3.1.0 is the only FSharp.Core referenced in the compiler command line.

At some point when the provided types are generated during compilation, various F#-specific types are required (unit, async...) and the compiler assumes those come from the same FSharp.Core version that it is using (4.4.0.0 in this case). A reference is somehow injected to 4.4.0.0 due to this, as best as I can tell. Reproing with the OSS compiler results in unintentional reference to 4.4.0.9055, which follows the same pattern.

[KevinRansom]

I have talked this over with Don and we agree that this is an issue with the type provider. the compiler does what it is told and the type provider has told it to use fsharp.core.dll. I considered remapping the references in the compiler, but that sort of magic does not really belong in the compiler.

What is needed is providedtypes.fs needs a new set of helpers which will make types, methodinfos, generic types etc only from the set of assemblies passed to the type provider by the compiler. I notice that there is a bunch of code in the TP for handling assembly resolution failure and so loading stuff randomly from the compiling machine. None of that will work well for portable libraries, and will work even worse if we re-host the compiler on coreclr, also when we are doing multi-targeting, I.e. using FSharp 4.0 to compile an F# 3.1 or F# 3.0 application, the only runtime difference is the set of target assemblies.

@ovatsus as maintainer of the TypeProviders starter pack, would you like to start a thread on how to solve this. I will help make sure the solution will work for the cross platform scenario.

[ovatsus]

This is very tricky business, unfortunately, with assembly mismatches all over.
FSharp.Data is the only TP lib I know of that tries to support portable, and it's really hard. First approach was to build a mechanism to fix up all the types, that breaks in NetCore because there's issues with mscorlib. I started trying another approach but I'm stuck: fsprojects/FSharp.Data#672

[KevinRansom]

I agree it ain't easy. So the type provider must only use types from the reference set for quotations and provided types. If the type can't be loaded from there then it is a missing type error. If you use LoadFrom and then GetType() and MakeGenericType etc. Type Forwarders and binding redirects should not be honored.

Remapping is possible but ultimately a hack, and the compiler doesn't really know about profiles, it just knows about the set of assemblies that it gets from the command line and the source code being compiled. If I had the bandwidth I would look at this, since it is an important issue, and one that is only going to get more important.

[latkin]

What is our guidance to TP authors/consumers, then? If our position is that this is an issue with the TP, how do they fix their issue?

If there is no good way to "fix" this, I guess we just advise folks that it's unsupported to down-target when using type providers (i.e. type providers are not forward-compatible).

[dsyme]

It is definitely possible to fix this in type providers, and most likely it is possible to have some very general machinery in ProvidedTypes.fs that fixes this in most cases for most type providers. It feels like we need to iterate towards some additions to ProvidedTypes.fs (in the Type Provider Starter Pack) which get us what we need....

[latkin]

Can you elaborate on how this can be fixed in TPs, so that @ctaggart and co. can make appropriate adjustments? If the TP was built entirely using 3.1 toolchain, what mechanism do we offer to prevent vFuture toolchain from adding bogus new references?

[dmitry-a-morozov]

I made an attempt to fix the problem.

I realized that I use AssemblyQualifiedName for types from FSharp.Core.
https://github.com/fsprojects/FSharp.Data.SqlClient/blob/104ca6be317dd752817b1272eacdeb69e5d7bb63/src/SqlClient/DesignTime.fs#L165
That's fixed

Also per @KevinRansom suggestion I changed assembly resolution logic from
https://github.com/fsprojects/FSharp.Data.SqlClient/blob/master/src/SqlClient/SqlProgrammabilityProvider.fs#L36
to
https://github.com/fsprojects/FSharp.Data.SqlClient/blob/38ba3b625c424aa69a7075100ad147adc4f60479/src/SqlClient/SqlProgrammabilityProvider.fs#L60

But the problem is not gone.

To experiment I added a console app as test projects that targets .NET 4.0 + FSharp.Core 4.3.0.0.
https://github.com/fsprojects/FSharp.Data.SqlClient/tree/master/src/SqlClient.Tests/SqlClient.Tests.NET40
Surprisingly this console application exe contains references to both FSharp.Core 4.3.0.0 and FSharp.Core 4.3.1.0.
How's that possible? Both type provider library and application compiled with FSharp.Core 4.3.0.0. So FSharp.Core 4.3.1.0 no-where in the picture.
When attempt to run this application fails with same error as mentioned in the original issue #23.
Binding redirects (either to 4.3.0.0 or forward to 4.3.1.0) helps to solve this.
So it is forward-compatible.

Please assist.
//cc @vladima

[KevinRansom]

This is a fundamentally very hard problem. TypeProvider authors need to think like compiler writers, and only bind types from the set of types provided to the compiler. The tendency is to think like an app and library author and bind to loadable types. As far as I can tell no type provider has yet done this correctly, I would dearly like to figure out how to solve this, but I am snowed under with other stuff to do.

What is required is a binder that will return types from a set of reachable assemblies. Those types are the ones that need to be returned to the compiler.

I had suggested and implemented a prototype of a filter between the compiler and the type provider that fixed up the references to only reachable types and it even worked. But Don convinced me that it was an invalid approach that primarily masked bugs in the type provider.

[dmitry-a-morozov]

Hi there.
I would like to mention 3 things:

1. I really need a help because I'm lost. I thought I have some understanding of the issue but not anymore. Guidelines would be nice.
2. Why FSharp.Core is never in the list of https://msdn.microsoft.com/en-us/library/hh323947.aspx ?
3. Here is something I don't understand at all.
   This a project that uses FSharp.Data.SqlClient
   https://github.com/fsprojects/FSharp.Data.SqlClient/tree/2753a097f224015c76f01346311c453a43240649/src/SqlClient.Tests/SqlClient.Tests.NET40
   The project targets .NET 4 and FSharp.Core 4.3.0.0.
   but when compiled in VS 2013 it produces binary that has dependency on both versions of FSharp.Core: 4.3.0.0 and 4.3.2.0
   when compiled in VS 2012 targeting .NET 4 and FSharp.Core 4.3.0.0 it depends as expected only FSharp.Core 4.3.0.0
   

Can you clarify?

Easiest way to repro locally is pull the whole repo
https://github.com/fsprojects/FSharp.Data.SqlClient.git
Open SqlClient.sln from root folder.
Then start project with (or without) debugging.
It opens test solution which includes SqlClient.Tests.NET40 project.

[veikkoeeva]

@dmitry-a-morozov This is a long-shot, but during the last half a year or so I've got the feeling that whenever there's Assembly.LoadFrom, one wants to apply policies as well or things turn up wrong at some point for the end user. More info at Fsharp.Data #821.

[dsyme]

Hi @dmitry-a-morozov - addressing one small part of this puzzle - In #478 I describe how the ReferencedAssemblies property is not populated fully and doesn't include FSharp.Core and mscorlib in the references. This is simply a bug but an annoying one as it would be very useful information to help with multi-targeting.

[dmitry-a-morozov]

Hi @dsyme. Thanks for the update. Keep me posted. Eventually I will need somebody's help or guidelines to fix the issue. Unless F# team will find a way to magically resolve it.