Skip to content

Commit

Permalink
[Xamarin.Android.Build.Tasks] Path is too long (dotnet#700)
Browse files Browse the repository at this point in the history 
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=30147

Windows (still) has a max path limit of 260 characters..
This can cause us a problem if the user creates a project in
a directory which already takes up most of that limit.. Like on
their Desktop!

This is because of the intermediate structure that was introduced
to handle embedding resources into the assemblies. The current
intermediate structure is as follows

	$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\library_project_imports\
	$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\native_library_imports\

Now consider that `$(AssemblyName)` can sometimes end up with something lile
"Xamarin.Android.Support.v7.AppCompat.21.0.3.0" you can easily see how we
start getting into trouble.

There was an attempt to fix this up a while ago (722dcc05) by introducing the
`$(UseShortFileNames)` property. However while this did introduce a
directory structure which was shorter is broke backwards compatability
with older versions of xamarin-android.

This is because of the structore of the zip files we that are embedded
into the assemblies. The current system just extracts the zip into
the `$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\`
directory and assumes that it contains a `library_project_imports` directory.

All the build tasks also assumed that as well.

So the fix needs to be done on a number of fronts. Firstly we need to
update the `$(_LibraryProjectImportsDirectoryName)` and
`$(_NativeLibraryImportsDirectoryName)` to be something shorter.
Next up we need to shorten "__library_project_imports__" to something
else verbose as well. This should cut down on the amount of the MAX_PATH
we chew up.

The real key to this is to NOT change the structure of the zip files in the
assemblies! So when we generate a zip from "jlibs" we make sure that the
folder in the zip is called "library_project_imports". And when we extract
the zip file we makle sure that "library_project_imports" is replaced by
the shorter name.
This will ensure that we are backward compatbile with older versions BUT
more importantly we get to use the shorter directory structure.
The files for native librarys are not extracted to disk but are extracted
from memory so as long as the structure remains the same i.e "native_library_imports"
that code does not need to change.

The other thing we need to do is to update ResolveLibraryProjectImports Task to
upgrade the system when it runs. So if we already have a "libraryprojectimports.cache"
in place we just use that as is. But if we re-run the ResolveLibraryProjectImports
task (due to a change or a clean build) we detect if we have the older structure in place
and just remove it.. Since we are going to regenerate the entire cache again anyway
we might as well start from scratch.

With this in place it becomes possible we can now enable `$(UseShortFileNames)`
by default!

So the new structure that is created is as follows

	$(IntermediateOutputPath)\lp\<id>\jl
	$(IntermediateOutputPath)\lp\<id>\nl

The <id> will be a single integer value which can be mapped to the source
assembly via a new map.cache file. This .cache file will contain a list of
assemblys. We use the `Index` of the assembly in the list to figure out
which cache directory to use.

	UnnamedProject
	Library1

In this case `Library1` would have an index of `1` and `UnnamedProject` will
be `0`. The old behaviour can be enabled by setting
`$(UseShortFileNames)` to `False`.

Note in order to keep existing bindings generator behaviour consistent,
the BindingsGenerator task will not use the `$(UseShortFileNames)`
property to control how it generates its .cs files. Instead a new
property

	$(UseShortGeneratorFileNames)

which can be used to control if the generator produces short names
(e.g 1.cs, 2.cs). This will be `False` by default.
  • Loading branch information
@dellis1972 @jonpryor
dellis1972 authored and jonpryor committed Jul 27, 2017
1 parent 7dadd00 commit 0667a2b
Show file tree
Hide file tree
Showing 14 changed files with 273 additions and 136 deletions.
6 changes: 5 additions & 1 deletion src/Xamarin.Android.Build.Tasks/Tasks/Aapt.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ public class Aapt : AsyncTask
public string ImportsDirectory { get; set; }
public string OutputImportDirectory { get; set; }
public bool UseShortFileNames { get; set; }
public string AssemblyIdentityMapFile { get; set; }

public string ResourceNameCaseMap { get; set; }

Expand All @@ -90,6 +91,7 @@ public class Aapt : AsyncTask
public string AndroidSdkPlatform { get; set; }

Dictionary<string,string> resource_name_case_map = new Dictionary<string,string> ();
AssemblyIdentityMap assemblyMap = new AssemblyIdentityMap ();

bool ManifestIsUpToDate (string manifestFile)
{
Expand Down Expand Up @@ -212,6 +214,8 @@ public override bool Execute ()
foreach (var arr in ResourceNameCaseMap.Split (';').Select (l => l.Split ('|')).Where (a => a.Length == 2))
resource_name_case_map [arr [1]] = arr [0]; // lowercase -> original

assemblyMap.Load (AssemblyIdentityMapFile);

ThreadingTasks.Parallel.ForEach (ManifestFiles, () => 0, DoExecute, (obj) => { Complete (); });

base.Execute ();
Expand Down Expand Up @@ -337,7 +341,7 @@ string ExpandString (string s)
return s.Substring (0, st + 1) + ExpandString (s.Substring (st + 1));
int ast = st + "${library.imports:".Length;
string aname = s.Substring (ast, ed - ast);
return s.Substring (0, st) + Path.Combine (OutputImportDirectory, UseShortFileNames ? MonoAndroidHelper.GetLibraryImportDirectoryNameForAssembly (aname) : aname, ImportsDirectory) + Path.DirectorySeparatorChar + ExpandString (s.Substring (ed + 1));
return s.Substring (0, st) + Path.Combine (OutputImportDirectory, UseShortFileNames ? assemblyMap.GetLibraryImportDirectoryNameForAssembly (aname) : aname, ImportsDirectory) + Path.DirectorySeparatorChar + ExpandString (s.Substring (ed + 1));
}
else
return s;
Expand Down
2 changes: 1 addition & 1 deletion src/Xamarin.Android.Build.Tasks/Tasks/CreateLibraryResourceArchive.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ public override bool Execute ()

if (Files.ArchiveZip (outpath, f => {
using (var zip = new ZipArchiveEx (f)) {
zip.AddDirectory (OutputDirectory, outDirInfo.Name);
zip.AddDirectory (OutputDirectory, "library_project_imports");
}
})) {
Log.LogDebugMessage ("Saving contents to " + outpath);
Expand Down
2 changes: 1 addition & 1 deletion src/Xamarin.Android.Build.Tasks/Tasks/CreateManagedLibraryResourceArchive.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ public override bool Execute ()

if (Files.ArchiveZip (outpath, f => {
using (var zip = new ZipArchiveEx (f)) {
zip.AddDirectory (OutputDirectory, outDirInfo.Name);
zip.AddDirectory (OutputDirectory, "library_project_imports");
}
})) {
Log.LogDebugMessage ("Saving contents to " + outpath);
Expand Down
2 changes: 1 addition & 1 deletion src/Xamarin.Android.Build.Tasks/Tasks/CreateNativeLibraryArchive.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ public override bool Execute ()

if (Files.ArchiveZip (outpath, f => {
using (var zip = new ZipArchiveEx (f)) {
zip.AddDirectory (OutputDirectory, outDirInfo.Name);
zip.AddDirectory (OutputDirectory, "native_library_imports");
}
})) {
Log.LogDebugMessage ("Saving contents to " + outpath);
Expand Down
191 changes: 104 additions & 87 deletions src/Xamarin.Android.Build.Tasks/Tasks/ResolveLibraryProjectImports.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ public class ResolveLibraryProjectImports : Task
[Required]
public bool UseShortFileNames { get; set; }

[Required]
public string AssemblyIdentityMapFile { get; set; }

public string CacheFile { get; set;}

[Output]
Expand All @@ -51,13 +54,14 @@ public class ResolveLibraryProjectImports : Task
[Output]
public ITaskItem [] ResolvedResourceDirectoryStamps { get; set; }

string imports_dir = "library_project_imports";
AssemblyIdentityMap assemblyMap = new AssemblyIdentityMap();

public ResolveLibraryProjectImports ()
{
}

// Extracts library project contents under e.g. obj/Debug/[__library_projects__/*.jar | res/*/*]
// Extracts library project contents under e.g. obj/Debug/[lp/*.jar | res/*/*]
public override bool Execute ()
{
Log.LogDebugMessage ("ResolveLibraryProjectImports Task");
Expand All @@ -72,6 +76,8 @@ public override bool Execute ()
var resolvedAssetDirectories = new List<string> ();
var resolvedEnvironmentFiles = new List<string> ();

assemblyMap.Load (AssemblyIdentityMapFile);

using (var resolver = new DirectoryAssemblyResolver (Log.LogWarning, loadDebugSymbols: false)) {
Extract (resolver, jars, resolvedResourceDirectories, resolvedAssetDirectories, resolvedEnvironmentFiles);
}
Expand Down Expand Up @@ -112,6 +118,8 @@ public override bool Execute ()
document.Save (CacheFile);
}

assemblyMap.Save (AssemblyIdentityMapFile);

Log.LogDebugTaskItems (" Jars: ", Jars.Select (s => new TaskItem (s)).ToArray ());
Log.LogDebugTaskItems (" ResolvedResourceDirectories: ", ResolvedResourceDirectories.Select (s => new TaskItem (s)).ToArray ());
Log.LogDebugTaskItems (" ResolvedAssetDirectories: ", ResolvedAssetDirectories.Select (s => new TaskItem (s)).ToArray ());
Expand All @@ -135,13 +143,20 @@ static string GetTargetAssembly (ITaskItem assemblyName)
}

// Extracts library project contents under e.g. obj/Debug/[__library_projects__/*.jar | res/*/*]
// Extracts library project contents under e.g. obj/Debug/[lp/*.jar | res/*/*]
void Extract (
DirectoryAssemblyResolver res,
ICollection<string> jars,
ICollection<string> resolvedResourceDirectories,
ICollection<string> resolvedAssetDirectories,
ICollection<string> resolvedEnvironments)
{
// lets "upgrade" the old directory.
string oldPath = Path.GetFullPath (Path.Combine (OutputImportDirectory, "..", "__library_projects__"));
if (!OutputImportDirectory.Contains ("__library_projects__") && Directory.Exists (oldPath)) {
MonoAndroidHelper.SetDirectoryWriteable (Path.Combine (oldPath, ".."));
Directory.Delete (oldPath, recursive: true);
}
var outdir = new DirectoryInfo (OutputImportDirectory);
if (!outdir.Exists)
outdir.Create ();
Expand All @@ -154,26 +169,95 @@ void Extract (
.Select (a => GetTargetAssembly (a))
.Where (a => a != null)
.Distinct ()) {
foreach (var imp in new string [] {imports_dir, "library_project_imports"}.Distinct ()) {
string assemblyIdentName = Path.GetFileNameWithoutExtension (assemblyPath);
if (UseShortFileNames) {
assemblyIdentName = Xamarin.Android.Tasks.MonoAndroidHelper.GetLibraryImportDirectoryNameForAssembly (assemblyIdentName);
}
string outDirForDll = Path.Combine (OutputImportDirectory, assemblyIdentName);
string importsDir = Path.Combine (outDirForDll, imp);
string assemblyIdentName = Path.GetFileNameWithoutExtension (assemblyPath);
if (UseShortFileNames) {
assemblyIdentName = assemblyMap.GetLibraryImportDirectoryNameForAssembly (assemblyIdentName);
}
string outDirForDll = Path.Combine (OutputImportDirectory, assemblyIdentName);
string importsDir = Path.Combine (outDirForDll, ImportsDirectory);
#if SEPARATE_CRUNCH
// FIXME: review these binResDir thing and enable this. Eclipse does this.
// Enabling these blindly causes build failure on ActionBarSherlock.
//string binResDir = Path.Combine (importsDir, "bin", "res");
//string binAssemblyDir = Path.Combine (importsDir, "bin", "assets");
#endif
string resDir = Path.Combine (importsDir, "res");
string assemblyDir = Path.Combine (importsDir, "assets");

// Skip already-extracted resources.
var stamp = new FileInfo (Path.Combine (outdir.FullName, assemblyIdentName + ".stamp"));
if (stamp.Exists && stamp.LastWriteTime > new FileInfo (assemblyPath).LastWriteTime) {
Log.LogDebugMessage ("Skipped resource lookup for {0}: extracted files are up to date", assemblyPath);
#if SEPARATE_CRUNCH
// FIXME: review these binResDir thing and enable this. Eclipse does this.
// FIXME: review these binResDir/binAssemblyDir thing and enable this. Eclipse does this.
// Enabling these blindly causes build failure on ActionBarSherlock.
//string binResDir = Path.Combine (importsDir, "bin", "res");
//string binAssemblyDir = Path.Combine (importsDir, "bin", "assets");
if (Directory.Exists (binResDir))
resolvedResourceDirectories.Add (binResDir);
if (Directory.Exists (binAssemblyDir))
resolvedAssetDirectories.Add (binAssemblyDir);
#endif
string resDir = Path.Combine (importsDir, "res");
string assemblyDir = Path.Combine (importsDir, "assets");
if (Directory.Exists (resDir))
resolvedResourceDirectories.Add (resDir);
if (Directory.Exists (assemblyDir))
resolvedAssetDirectories.Add (assemblyDir);
continue;
}

if (Directory.Exists (outDirForDll))
Directory.Delete (outDirForDll, true);

var assembly = res.GetAssembly (assemblyPath);

foreach (var mod in assembly.Modules) {
// android environment files
foreach (var envtxt in mod.Resources
.Where (r => r.Name.StartsWith ("__AndroidEnvironment__", StringComparison.OrdinalIgnoreCase))
.Where (r => r is EmbeddedResource)
.Cast<EmbeddedResource> ()) {
if (!Directory.Exists (outDirForDll))
Directory.CreateDirectory (outDirForDll);
var finfo = new FileInfo (Path.Combine (outDirForDll, envtxt.Name));
using (var fs = finfo.Create ()) {
var data = envtxt.GetResourceData ();
fs.Write (data, 0, data.Length);
}
resolvedEnvironments.Add (finfo.FullName);
}

// embedded jars (EmbeddedJar, EmbeddedReferenceJar)
var resjars = mod.Resources
.Where (r => r.Name.EndsWith (".jar", StringComparison.InvariantCultureIgnoreCase))
.Select (r => (EmbeddedResource) r);
foreach (var resjar in resjars) {
var data = resjar.GetResourceData ();
if (!Directory.Exists (importsDir))
Directory.CreateDirectory (importsDir);
using (var outfs = File.Create (Path.Combine (importsDir, resjar.Name)))
outfs.Write (data, 0, data.Length);
}

// Skip already-extracted resources.
var stamp = new FileInfo (Path.Combine (outdir.FullName, assemblyIdentName + ".stamp"));
if (stamp.Exists && stamp.LastWriteTime > new FileInfo (assemblyPath).LastWriteTime) {
Log.LogDebugMessage ("Skipped resource lookup for {0}: extracted files are up to date", assemblyPath);
// embedded AndroidResourceLibrary archive
var reszip = mod.Resources.FirstOrDefault (r => r.Name == "__AndroidLibraryProjects__.zip") as EmbeddedResource;
if (reszip != null) {
if (!Directory.Exists (outDirForDll))
Directory.CreateDirectory (outDirForDll);
var finfo = new FileInfo (Path.Combine (outDirForDll, reszip.Name));
using (var fs = finfo.Create ()) {
var data = reszip.GetResourceData ();
fs.Write (data, 0, data.Length);
}

// temporarily extracted directory will look like:
// __library_projects__/[dllname]/[library_project_imports | jlibs]/bin
using (var zip = MonoAndroidHelper.ReadZipFile (finfo.FullName)) {
Files.ExtractAll (zip, outDirForDll, modifyCallback: (entryFullName) => {
return entryFullName.Replace ("library_project_imports", ImportsDirectory);
});
}

// We used to *copy* the resources to overwrite other resources,
// which resulted in missing resource issue.
// Here we replaced copy with use of '-S' option and made it to work.
#if SEPARATE_CRUNCH
// FIXME: review these binResDir/binAssemblyDir thing and enable this. Eclipse does this.
// Enabling these blindly causes build failure on ActionBarSherlock.
Expand All @@ -186,80 +270,13 @@ void Extract (
resolvedResourceDirectories.Add (resDir);
if (Directory.Exists (assemblyDir))
resolvedAssetDirectories.Add (assemblyDir);
continue;
}

if (Directory.Exists (outDirForDll))
Directory.Delete (outDirForDll, true);

Directory.CreateDirectory (importsDir);

var assembly = res.GetAssembly (assemblyPath);

foreach (var mod in assembly.Modules) {
// android environment files
foreach (var envtxt in mod.Resources
.Where (r => r.Name.StartsWith ("__AndroidEnvironment__", StringComparison.OrdinalIgnoreCase))
.Where (r => r is EmbeddedResource)
.Cast<EmbeddedResource> ()) {
if (!Directory.Exists (outDirForDll))
Directory.CreateDirectory (outDirForDll);
var finfo = new FileInfo (Path.Combine (outDirForDll, envtxt.Name));
using (var fs = finfo.Create ()) {
var data = envtxt.GetResourceData ();
fs.Write (data, 0, data.Length);
}
resolvedEnvironments.Add (finfo.FullName);
}

// embedded jars (EmbeddedJar, EmbeddedReferenceJar)
var resjars = mod.Resources
.Where (r => r.Name.EndsWith (".jar", StringComparison.InvariantCultureIgnoreCase))
.Select (r => (EmbeddedResource) r);
foreach (var resjar in resjars) {
var data = resjar.GetResourceData ();
using (var outfs = File.Create (Path.Combine (importsDir, resjar.Name)))
outfs.Write (data, 0, data.Length);
}

// embedded AndroidResourceLibrary archive
var reszip = mod.Resources.FirstOrDefault (r => r.Name == "__AndroidLibraryProjects__.zip") as EmbeddedResource;
if (reszip != null) {
if (!Directory.Exists (outDirForDll))
Directory.CreateDirectory (outDirForDll);
var finfo = new FileInfo (Path.Combine (outDirForDll, reszip.Name));
using (var fs = finfo.Create ()) {
var data = reszip.GetResourceData ();
fs.Write (data, 0, data.Length);
}

// temporarily extracted directory will look like:
// __library_projects__/[dllname]/[library_project_imports | jlibs]/bin
using (var zip = MonoAndroidHelper.ReadZipFile (finfo.FullName))
Files.ExtractAll (zip, outDirForDll);

// We used to *copy* the resources to overwrite other resources,
// which resulted in missing resource issue.
// Here we replaced copy with use of '-S' option and made it to work.
#if SEPARATE_CRUNCH
// FIXME: review these binResDir/binAssemblyDir thing and enable this. Eclipse does this.
// Enabling these blindly causes build failure on ActionBarSherlock.
if (Directory.Exists (binResDir))
resolvedResourceDirectories.Add (binResDir);
if (Directory.Exists (binAssemblyDir))
resolvedAssetDirectories.Add (binAssemblyDir);
#endif
if (Directory.Exists (resDir))
resolvedResourceDirectories.Add (resDir);
if (Directory.Exists (assemblyDir))
resolvedAssetDirectories.Add (assemblyDir);

finfo.Delete ();
}
finfo.Delete ();
}
}

if (Directory.Exists (importsDir))
stamp.Create ().Close ();
}
}

foreach (var f in outdir.GetFiles ("*.jar")
Expand Down
23 changes: 20 additions & 3 deletions src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BindingBuildTest.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
using NUnit.Framework;
using System.IO;
using System.Linq;
using System.Collections.Generic;

namespace Xamarin.Android.Build.Tests
{
Expand Down Expand Up @@ -263,7 +264,7 @@ public void BindngFilterUnsupportedNativeAbiLibraries ()
}

[Test]
public void BindingCheckHiddenFiles ()
public void BindingCheckHiddenFiles ([Values (true, false)] bool useShortFileNames)
{
var binding = new XamarinAndroidBindingProject () {
UseLatestPlatformSdk = true,
Expand All @@ -273,15 +274,31 @@ public void BindingCheckHiddenFiles ()
binding.Jars.Add (new AndroidItem.LibraryProjectZip ("Jars\\mylibrary.aar") {
WebContent = "https://www.dropbox.com/s/astiqp8jo97x91h/mylibrary.aar?dl=1"
});
binding.SetProperty (binding.ActiveConfigurationProperties, "UseShortFileNames", useShortFileNames);
using (var bindingBuilder = CreateDllBuilder (Path.Combine ("temp", "BindingCheckHiddenFiles", "Binding"))) {
bindingBuilder.Verbosity = Microsoft.Build.Framework.LoggerVerbosity.Diagnostic;
Assert.IsTrue (bindingBuilder.Build (binding), "binding build should have succeeded");
var proj = new XamarinAndroidApplicationProject ();
proj.OtherBuildItems.Add (new BuildItem ("ProjectReference", "..\\Binding\\UnnamedProject.csproj"));
proj.SetProperty (proj.ActiveConfigurationProperties, "UseShortFileNames", useShortFileNames);
using (var b = CreateApkBuilder (Path.Combine ("temp", "BindingCheckHiddenFiles", "App"))) {
Assert.IsTrue (b.Build (proj), "Build should have succeeded.");
var dsStorePath = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, "__library_projects__",
"UnnamedProject", "library_project_imports");
var assemblyMap = b.Output.GetIntermediaryPath (Path.Combine ("lp", "map.cache"));
if (useShortFileNames)
Assert.IsTrue (File.Exists (assemblyMap), $"{assemblyMap} should exist.");
else
Assert.IsFalse (File.Exists (assemblyMap), $"{assemblyMap} should not exist.");
var assemblyIdentityMap = new List<string> ();
if (useShortFileNames) {
foreach (var s in File.ReadLines (assemblyMap)) {
assemblyIdentityMap.Add (s);
}
}
var assmeblyIdentity = useShortFileNames ? assemblyIdentityMap.IndexOf ("UnnamedProject").ToString () : "UnnamedProject";
var libaryImportsFolder = useShortFileNames ? "lp" : "__library_projects__";
var jlibs = useShortFileNames ? "jl" : "library_project_imports";
var dsStorePath = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, libaryImportsFolder,
assmeblyIdentity, jlibs);
Assert.IsTrue (Directory.Exists (dsStorePath), "{0} should exist.", dsStorePath);
Assert.IsFalse (File.Exists (Path.Combine (dsStorePath, ".DS_Store")), "{0} should NOT exist.",
Path.Combine (dsStorePath, ".DS_Store"));
Expand Down
Loading

0 comments on commit 0667a2b

Please sign in to comment.