Archiver follows symlinks on Windows 7

[petr-ujezdsky]

We have many symlinks pointing to huge directories within our project. Because this archiver follows symlinks on Windows machines, the build stalls when eg. copying standard resources.

I have found the bug at AbstractArchiver.java:

    private boolean isSymlinkSupported()
    {
        return Os.isFamily( Os.FAMILY_UNIX ) && Java7Reflector.isAtLeastJava7();
    }

This should be changed to something like

return (Os.isFamily( Os.FAMILY_UNIX ) || Os.isFamily( Os.FAMILY_WINDOWS )) && Java7Reflector.isAtLeastJava7();

And the next strange thing is the usage of this method:

collection.setFollowingSymLinks( !isSymlinkSupported() );

When the OS does not support symlinks then the collection should follow them?

[krosenvold]

Unfortunately it is not that simple. Traditionally java has not had the capability to create, modify or delete windows symlinks. I though this was still the case, at least it is not the case for Os.FAMILY_WINDOWS in general, maybe a certain subset of these. If you can make unit tests that demonstrate symlink capabiltity on your given Windows version we can open for that (or find other technical docs describing when/what the limitations of the windows symlink implementation are).

The default behaviour for years on end has been to follow symlinks like normal directories, since conventional wisdom has been that symlinks act somewhat like directories, so the code you are looking at is not odd from that perspective.

[hellpf]

Hi, from my point of view, the behaviour of working with symlinks should be the same across all OS (osx/linux/windows). If you are not sure, just add option to the pom.xml.

[krosenvold]

There's always room for surprises ! Please clone plexus-io and run on your local os (mvn clean install). I just added some "capability" tests that may reveal the level of windows support. I'll see if I can find some windows boxes of my own

[plamentotev]

Hi,
Both Java 7 and Java 8 could create symlinks on Windows 7 and Windows 10. But keep in mind that by default you need admin privileges to create symlinks on Windows.

[plamentotev]

I could confirm that on Java 8 + Windows 10 all tests are passing.

[krosenvold]

This seems like a pretty cool thing to fix. Historically our symlink support has been weak/nonexistant and some of our current approaches are not good enough. I'm preparing a small post fot the maven developers list where we can discuss how the different issues should be handled. Please join the discussion there.

[petr-ujezdsky]

The default behaviour for years on end has been to follow symlinks like normal directories [...]

This is actually not true when you look at the code

collection.setFollowingSymLinks( !isSymlinkSupported() );

When you have machine that supports symlinks (eg. linux for your current implementation), the symlinks are NOT followed.

I do not know if this is intentional

Historically our symlink support has been weak/nonexistant

but from what you said this could be bug as well.

[krosenvold]

When you have machine that supports symlinks (eg. linux for your current implementation), the symlinks are NOT followed.

Yes :) That's because historic behaviour was changed about a year ago. On platforms that support symlinks we NEVER follow them, because we are able to represent them correctly on the roundtrip.

So we never "follow" symlnks (resolve them) when symlinks are supported, simply because we handle the link properly. I am still uncertain if there is any decent use case for resolving/following symlinks on platforms that support it. I'm still also working on the writeup I promised yesterday :)

[petr-ujezdsky]

Hi, any news?

[michael-o]

Locally applied:

diff --git a/src/main/java/org/codehaus/plexus/archiver/AbstractArchiver.java b/src/main/java/org/codehaus/plexus/archiver/AbstractArchiver.java
index dbcf07e..709701a 100755
--- a/src/main/java/org/codehaus/plexus/archiver/AbstractArchiver.java
+++ b/src/main/java/org/codehaus/plexus/archiver/AbstractArchiver.java
@@ -335,7 +335,7 @@ public void addFileSet( @Nonnull final FileSet fileSet )
         // The PlexusIoFileResourceCollection contains platform-specific File.separatorChar which
         // is an interesting cause of grief, see PLXCOMP-192
         final PlexusIoFileResourceCollection collection = new PlexusIoFileResourceCollection();
-        collection.setFollowingSymLinks( !isSymlinkSupported() );
+        collection.setFollowingSymLinks( isSymlinkSupported() );

         collection.setIncludes( fileSet.getIncludes() );
         collection.setExcludes( fileSet.getExcludes() );

and I get this

[ERROR] testSymlinkDirArchiver(org.codehaus.plexus.archiver.SymlinkTest)  Time elapsed: 0.476 s  <<< ERROR!
org.codehaus.plexus.archiver.ArchiverException: Unable to create directory or parent directory of /usr/home/mosipov/Projekte/plexus-archiver/target/output/dirarchiver-symlink/symLinkToDirOnTheOutside/FileInDirOnTheOutside.txt
        at org.codehaus.plexus.archiver.SymlinkTest.testSymlinkDirArchiver(SymlinkTest.java:103)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   SymlinkTest.testSymlinkDirArchiver:103 » Archiver Unable to create directory o...
[INFO]
[ERROR] Tests run: 143, Failures: 0, Errors: 1, Skipped: 1

The test looks broken to me because the target directory and file do not exist:

target/output/
├── dirarchiver-symlink
│   ├── aDirWithALink
│   │   └── backOutsideToFileX -> ../fileX.txt
│   ├── aRegularDir
│   │   └── aRegularFile.txt
│   ├── fileR.txt
│   ├── fileW.txt
│   ├── fileX.txt
│   ├── symDir -> targetDir
│   ├── symLinkToDirOnTheOutside -> ../dirOnTheOutside
│   └── targetDir
│       └── targetFile.txt
├── symlinks.tar
├── symlinks.zip
├── untaredSymlinks
│   ├── aDirWithALink
│   │   └── backOutsideToFileX -> ../fileX.txt
│   ├── aRegularDir
│   │   └── aRegularFile.txt
│   ├── fileR.txt
│   ├── fileW.txt
│   ├── fileX.txt
│   ├── symDir -> targetDir
│   ├── symLinkToDirOnTheOutside -> ../dirOnTheOutside
│   ├── symLinkToFileOnTheOutside -> ../onTheOutside.txt
│   ├── symR -> fileR.txt
│   ├── symW -> fileW.txt
│   ├── symX -> fileX.txt
│   └── targetDir
│       └── targetFile.txt
└── unzippedSymlinks
    ├── aDirWithALink
    │   └── backOutsideToFileX -> ../fileX.txt
    ├── aRegularDir
    │   └── aRegularFile.txt
    ├── fileR.txt
    ├── fileW.txt
    ├── fileX.txt
    ├── symDir -> targetDir
    ├── symLinkToDirOnTheOutside -> ../dirOnTheOutside
    ├── symLinkToFileOnTheOutside -> ../onTheOutside.txt
    ├── symR -> fileR.txt
    ├── symW -> fileW.txt
    ├── symX -> fileX.txt
    └── targetDir
        └── targetFile.txt

[michael-o]

I need to revert my previous statement. The offending piece of code collection.setFollowingSymLinks( !isSymlinkSupported() ); is correct.

Origin:

$ tree /usr/home/mosipov/Projekte/plexus-archiver/src/testesources/symlinks/src
/usr/home/mosipov/Projekte/plexus-archiver/src/test/resources/symlinks/src
├── aDirWithALink
│   └── backOutsideToFileX -> ../fileX.txt
├── aRegularDir
│   └── aRegularFile.txt
├── fileR.txt
├── fileW.txt
├── fileX.txt
├── symDir -> targetDir/
├── symLinkToDirOnTheOutside -> ../dirOnTheOutside/
├── symLinkToFileOnTheOutside -> ../onTheOutside.txt
├── symR -> fileR.txt
├── symW -> fileW.txt
├── symX -> fileX.txt
└── targetDir
    └── targetFile.txt

5 directories, 10 files

Target:

$ tree target/output/dirarchiver-symlink
target/output/dirarchiver-symlink
├── aDirWithALink
│   └── backOutsideToFileX -> ../fileX.txt
├── aRegularDir
│   └── aRegularFile.txt
├── fileR.txt
├── fileW.txt
├── fileX.txt
├── symDir -> targetDir
├── symLinkToDirOnTheOutside -> ../dirOnTheOutside
├── symLinkToFileOnTheOutside -> ../onTheOutside.txt
├── symR -> fileR.txt
├── symW -> fileW.txt
├── symX -> fileX.txt
└── targetDir
    └── targetFile.txt

4 directories, 11 files

If you have symlinks followed, it will resolve them as if there are no symlinks.

I am inclined to throw that away altogether and pass false by default.

[plamentotev]

@michael-o I don't quite follow you. Why do you want to replace

collection.setFollowingSymLinks( !isSymlinkSupported() );

with

collection.setFollowingSymLinks( false );

[michael-o]

Simple reason, people don't understand what this parameter is for. The purpose is for the PlexusIoCollection: it completely resolves links, thus omitted them from the archive. Since this is disabled on Unix-like and cannot be configured, I cannot see a reason keeping the boolean condition.

[petr-ujezdsky]

Am I getting things right that with

collection.setFollowingSymLinks( false );

the archiver will simply copy symlinks as they are (treating the symlink as a 'file'), and with

collection.setFollowingSymLinks( true );

the archiver will traverse the symlink and copy all files within it (treating the symlink as a directory)?

What I want to ultimately achieve is to make plexus skip traversing into symlinks on my Windows 7+ system. As I have stated in the issue on top, we have symlinks (created at build) to huge directory trees (made from another symlinks, resolved tree could be like 100 000+ files) that we do not want to be processed by plexus archiver.

As of now the isSymlinkSupported() returns false for Windows so the case collection.setFollowingSymLinks( true ); is used, hence the problem with symlinks traversing.
Your hardcoded collection.setFollowingSymLinks( false ); is actually very similar to what I have originally proposed by making the isSymlinkSupported() return true for Windows.

The only big issue I can see is that symlinks on Windows 7+ can be created only under admin privileges as @plamentotev mentioned above. Even if you add the right for creating symlinks to the standard user he still can not create them (Microsoft, why u do this??).