ApiCompatibility.AssemblySymbolLoader design should be revisited

[safern]

The current model for AssemblySymbolLoader is that you create an instance of the object and then that object keeps track of the loaded assemblies and search paths. Also, the IEnumerable<IAssemblySymbol> LoadAssemblies APIs run synchronously and don't do this load lazily. The reason to do that synchronously is because it has a HasLoadWarnings(out IEnumerable<string> warnings) api, and when loading assemblies it populates that array. However that array is stored for the life time of the object and if you call multiple times LoadAssemblies it would return the aggregate of warnings found.

Also, we might be able to do a lazy loading of the assemblies using the enumerator model if that meets the customer expectations.

Some options for the context is having the user pass down a context as a parameter, that way we don't have customers accidentally cross the context boundaries and get the assembly that was previously loaded when they wanted a new assembly (assemblies are cached by name).

Also the way we are loading references might need to be improved: #16817 (comment)

cc: @ericstj @Anipik