[JENKINS-47324] - Reduce usage of File.mkdirs() #3173

KrishanBhasin · 2017-12-04T21:51:12Z

See JENKINS-47324.

Proposed changelog entries

Reduce usage of renameTo()

Submitter checklist

JIRA issue is well described
Changelog entry appropriate for the audience affected by the change (users or developer, depending on the change). Examples
* Use the Internal: prefix if the change has no user-visible impact (API, test frameworks, etc.)
Appropriate autotests or explanation to why this change has no tests
For dependency updates: links to external changelogs and, if possible, full diffs

Desired reviewers

@dwnusbaum

KrishanBhasin · 2017-12-04T21:53:22Z

I have opened this PR early despite the code clearly not being ready yet as I am looking for feedback as I go - this is quite a challenging change for me so any help is appreciated!

KrishanBhasin · 2017-12-04T21:57:15Z

core/src/main/java/hudson/FilePath.java

@@ -1962,7 +1963,7 @@ public void renameTo(final FilePath target) throws IOException, InterruptedExcep
        act(new SecureFileCallable<Void>() {
            private static final long serialVersionUID = 1L;
            public Void invoke(File f, VirtualChannel channel) throws IOException {
-            	reading(f).renameTo(creating(new File(target.remote)));
+                Files.move(reading(f).toPath(), creating(new File(target.remote)).toPath(), StandardCopyOption.COPY_ATTRIBUTES, LinkOption.NOFOLLOW_LINKS);


I don't try/catch here as Files.move() throws an IOException, which is what we would like to propagate out from this class.

That's fine, but you do want to merge with master and use Util#fileToPath anywhere you use File#toPath so that the runtime exception InvalidPathException is wrapped as an IOException.

Also, the documentation for Files#move doesn't include StandardCopyOption.COPY_ATTRIBUTES, so I'd remove it here. (I think most implementations of Files#move attempt to copy attributes by default.)

KrishanBhasin · 2017-12-04T22:01:52Z

core/src/main/java/hudson/FilePath.java

@@ -1121,7 +1122,7 @@ public URI toURI() throws IOException, InterruptedException {
        return act(new SecureFileCallable<URI>() {
            private static final long serialVersionUID = 1L;
            public URI invoke(File f, VirtualChannel channel) {
-                return f.toURI();
+                return f.toPath().toUri();


I don't see a tangible benefit of this change - can anyone explain? It is listed as the 'new' equivalent of the old method here

There is no strong benefit. However, in your case you are at risk of a new InvalidPathException, which may be thrown by toPath()

Replace mkdir() with Files.createTempDirectory()

oleg-nenashev · 2017-12-04T22:43:57Z

core/src/main/java/hudson/FilePath.java

@@ -1121,7 +1122,7 @@ public URI toURI() throws IOException, InterruptedException {
        return act(new SecureFileCallable<URI>() {
            private static final long serialVersionUID = 1L;
            public URI invoke(File f, VirtualChannel channel) {
-                return f.toURI();
+                return f.toPath().toUri();


There is no strong benefit. However, in your case you are at risk of a new InvalidPathException, which may be thrown by toPath()

oleg-nenashev · 2017-12-04T22:44:53Z

core/src/main/java/hudson/FilePath.java

@@ -587,11 +588,11 @@ private void unzip(File dir, File zipFile) throws IOException {
                ZipEntry e = entries.nextElement();
                File f = new File(dir, e.getName());
                if (e.isDirectory()) {
-                    mkdirs(f);
+                    Files.createDirectories(f.toPath());


Use Util#toPath() created by @dwnusbaum a while ago. Otherwise there is a risk of unhandled runtime InvalidPathException

Also FilePath#mkdirs(File) calls filterNonNull().mkdirs(dir);, which checks permissions when doing remote operations. This removes that check. I would update the mkdirs method itself to use Files#createDirectories, and then you won't need to change all of its callers.

oleg-nenashev · 2017-12-04T22:45:43Z

Marked the PR as "work-in-progress" according to the comment

dwnusbaum · 2017-12-05T22:05:07Z

core/src/main/java/hudson/FilePath.java

@@ -1409,7 +1410,7 @@ public FilePath createTempDir(final String prefix, final String suffix) throws I
                public String invoke(File dir, VirtualChannel channel) throws IOException {
                    File f = File.createTempFile(prefix, suffix, dir);
                    f.delete();
-                    f.mkdir();
+                    Files.createTempDirectory(f.toPath(), null);


This function is being updated by #3161, so I'd remove this change as it will cause a merge conflict.

dwnusbaum · 2017-12-05T22:07:03Z

core/src/main/java/hudson/FilePath.java

@@ -1962,7 +1963,7 @@ public void renameTo(final FilePath target) throws IOException, InterruptedExcep
        act(new SecureFileCallable<Void>() {
            private static final long serialVersionUID = 1L;
            public Void invoke(File f, VirtualChannel channel) throws IOException {
-            	reading(f).renameTo(creating(new File(target.remote)));
+                Files.move(reading(f).toPath(), creating(new File(target.remote)).toPath(), StandardCopyOption.COPY_ATTRIBUTES, LinkOption.NOFOLLOW_LINKS);


That's fine, but you do want to merge with master and use Util#fileToPath anywhere you use File#toPath so that the runtime exception InvalidPathException is wrapped as an IOException.

dwnusbaum · 2017-12-05T22:10:23Z

core/src/main/java/hudson/FilePath.java

@@ -587,11 +588,11 @@ private void unzip(File dir, File zipFile) throws IOException {
                ZipEntry e = entries.nextElement();
                File f = new File(dir, e.getName());
                if (e.isDirectory()) {
-                    mkdirs(f);
+                    Files.createDirectories(f.toPath());


Also FilePath#mkdirs(File) calls filterNonNull().mkdirs(dir);, which checks permissions when doing remote operations. This removes that check. I would update the mkdirs method itself to use Files#createDirectories, and then you won't need to change all of its callers.

dwnusbaum · 2017-12-05T22:19:47Z

core/src/main/java/hudson/FilePath.java

@@ -1962,7 +1963,7 @@ public void renameTo(final FilePath target) throws IOException, InterruptedExcep
        act(new SecureFileCallable<Void>() {
            private static final long serialVersionUID = 1L;
            public Void invoke(File f, VirtualChannel channel) throws IOException {
-            	reading(f).renameTo(creating(new File(target.remote)));
+                Files.move(reading(f).toPath(), creating(new File(target.remote)).toPath(), StandardCopyOption.COPY_ATTRIBUTES, LinkOption.NOFOLLOW_LINKS);


Also, the documentation for Files#move doesn't include StandardCopyOption.COPY_ATTRIBUTES, so I'd remove it here. (I think most implementations of Files#move attempt to copy attributes by default.)

…he new API.

…time exceptions

KrishanBhasin · 2017-12-06T21:59:13Z

I had a crawl through how FilePath deletes files, and it looks like Util has methods that could be migrated to using the new Files.delete(Path) method?

dwnusbaum · 2017-12-06T22:02:35Z

@KrishanBhasin Yes definitely, I've been working on that in #3169 if you want to take a look and review it.

dwnusbaum

The current state looks good. If you want to clean up the rest of the code that creates directories, you could also update IOUtils#mkdirs which is called by FilePath#mkdirsE, and then FilePath#mkdirs which has one last call to File#mkdirs (around line 1168) that could be switched to use mkdirs(f). That would get rid of all calls to File#mkdirs and File#mkdir (after #3161 is merged) from FilePath, which would be great.

dwnusbaum · 2017-12-07T21:17:30Z

core/src/main/java/hudson/FilePath.java

-import java.nio.file.Files;
-import java.nio.file.InvalidPathException;
-import java.nio.file.StandardCopyOption;
+import java.nio.file.*;


nit: I personally prefer using single-class imports (especially for non-static imports) to make it obvious what classes are being used and avoid any name clashes if new classes are added to the package. I don't know if there is a preference either way in Jenkins in general.

move mkdirs() to using FilePath method instead of File method.

KrishanBhasin · 2017-12-08T11:10:48Z

@dwnusbaum I think I've covered off those two methods you pointed out.

I thought this Jira ticket was going to involve a lot more changes, it looked a lot more intimidating than it actually was!

oleg-nenashev

Just another File#toPath()

oleg-nenashev · 2017-12-10T05:03:39Z

core/src/main/java/hudson/util/IOUtils.java

-        } catch (InterruptedException e) {
-            // ignore
+            return Files.createDirectories(dir.toPath()).toFile();
+        } catch (UnsupportedOperationException e) {


Also https://docs.oracle.com/javase/7/docs/api/java/nio/file/InvalidPathException.html . @dwnusbaum has added a utility method for that recently, so you could replace dir.toPath() byt fileToPath(dir)

oleg-nenashev · 2017-12-10T05:04:45Z

core/src/main/java/hudson/FilePath.java

-import static hudson.Util.deleteFile;
-import static hudson.Util.fixEmpty;
-import static hudson.Util.isSymlink;
+import static hudson.Util.*;


I would recommend to avoid doing it in a production code to avoid ambiguity

I thought I had set Intellij to avoid doing this! I have fixed the settings & the code

oleg-nenashev

👍

oleg-nenashev · 2017-12-11T08:36:07Z

core/src/main/java/hudson/FilePath.java

@@ -2957,11 +2960,12 @@ private File deleting(File f) {
        return f;
    }

-    private boolean mkdirs(File dir) {
+    private boolean mkdirs(File dir) throws IOException {
        if (dir.exists())   return false;


I feel this method should check whether the file is a directory and throw the exception otherwise. It is not related to this change, I will follow-up in a pull request. Maybe we can kill the method at all to be on the safe side

dwnusbaum · 2017-12-12T19:26:25Z

core/src/main/java/hudson/util/IOUtils.java

-            // ignore
+            return Files.createDirectories(fileToPath(dir)).toFile();
+        } catch (UnsupportedOperationException e) {
+            throw new IOException(e);


I'm not entirely sure what Ant <mkdir> task was referring to here before. It would be closer to the original code to try again if an IOException is thrown, but I can't think of a common error that would actually be fixed by doing that, so I'm ok with getting rid of the sleep with this change.

oleg-nenashev · 2017-12-14T19:53:12Z

Thanks @KrishanBhasin !

KrishanBhasin · 2017-12-15T10:47:49Z

Thanks @oleg-nenashev and @dwnusbaum for all the feedback as I went! I'll keep any eye out for another low hanging issue that I can try picking up :)

Move an instance of renameTo() to Files.move()

16ed1e7

dwnusbaum self-requested a review December 4, 2017 21:54

KrishanBhasin commented Dec 4, 2017

View reviewed changes

Replace an instance of File.toURI() with an instance of Path.toUri()

fcbc1e4

KrishanBhasin commented Dec 4, 2017

View reviewed changes

Replace mkdirs() with Files.createDirectories()

a448164

Replace mkdir() with Files.createTempDirectory()

oleg-nenashev changed the title ~~Jenkins Issue 47327~~ [JENKINS-47324] - Reduce usage of renameTo() Dec 4, 2017

oleg-nenashev reviewed Dec 4, 2017

View reviewed changes

oleg-nenashev added needs-more-reviews Complex change, which would benefit from more eyes work-in-progress The PR is under active development, not ready to the final review labels Dec 4, 2017

dwnusbaum reviewed Dec 5, 2017

View reviewed changes

dwnusbaum self-requested a review December 5, 2017 22:20

KrishanBhasin added 4 commits December 6, 2017 21:03

Undo addition of createTempDirectory() as per review comments

431f7c8

Return to use of FilePath#mkdirs(File) and instead modify it to use t…

8c849f3

…he new API.

Undo addition of toPath() in a URI conversion as it brings no benefits.

e22ad1f

Replace new uses of toPath() with Util.fileToPath() to pre-handle run…

a881c10

…time exceptions

dwnusbaum reviewed Dec 7, 2017

View reviewed changes

KrishanBhasin added 3 commits December 8, 2017 10:52

Remove * import.

49630d6

move mkdirs() to using FilePath method instead of File method.

Make IOUtils.mkdirs(File) use Java7 API calls

eec77f9

Add back accidentally removed imports.

356b0d8

KrishanBhasin changed the title ~~[JENKINS-47324] - Reduce usage of renameTo()~~ [JENKINS-47324] - Reduce usage of File.mkdirs() Dec 8, 2017

oleg-nenashev removed the work-in-progress The PR is under active development, not ready to the final review label Dec 10, 2017

oleg-nenashev requested changes Dec 10, 2017

View reviewed changes

KrishanBhasin added 2 commits December 10, 2017 16:15

Fixed use of wildcard import

0ec9e38

Use utility method fileToPath() to handle potential exception thrown

eb06ecb

oleg-nenashev approved these changes Dec 11, 2017

View reviewed changes

dwnusbaum approved these changes Dec 12, 2017

View reviewed changes

oleg-nenashev merged commit 09bcc5d into jenkinsci:master Dec 14, 2017

oleg-nenashev added ready-for-merge The PR is ready to go, and it will be merged soon if there is no negative feedback and removed needs-more-reviews Complex change, which would benefit from more eyes labels Dec 14, 2017

KrishanBhasin deleted the feature/47324-java-7-files-api branch December 16, 2017 13:32

jglick mentioned this pull request Mar 21, 2018

[JENKINS-46386] Make SystemProperties safe to call from agent JVMs #3362

Merged

dwnusbaum mentioned this pull request Oct 5, 2018

[JENKINS-2111] Managed workspace indices jenkinsci/branch-api-plugin#129

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JENKINS-47324] - Reduce usage of File.mkdirs() #3173

[JENKINS-47324] - Reduce usage of File.mkdirs() #3173

KrishanBhasin commented Dec 4, 2017

KrishanBhasin commented Dec 4, 2017

KrishanBhasin Dec 4, 2017

dwnusbaum Dec 5, 2017

dwnusbaum Dec 5, 2017

KrishanBhasin Dec 4, 2017

oleg-nenashev Dec 4, 2017

oleg-nenashev Dec 4, 2017

oleg-nenashev Dec 4, 2017

dwnusbaum Dec 5, 2017

oleg-nenashev commented Dec 4, 2017 •

edited

Loading

dwnusbaum Dec 5, 2017

dwnusbaum Dec 5, 2017

dwnusbaum Dec 5, 2017

dwnusbaum Dec 5, 2017

KrishanBhasin commented Dec 6, 2017

dwnusbaum commented Dec 6, 2017

dwnusbaum left a comment

dwnusbaum Dec 7, 2017

KrishanBhasin commented Dec 8, 2017

oleg-nenashev left a comment

oleg-nenashev Dec 10, 2017

oleg-nenashev Dec 10, 2017

KrishanBhasin Dec 10, 2017

oleg-nenashev left a comment

oleg-nenashev Dec 11, 2017

dwnusbaum Dec 12, 2017

oleg-nenashev commented Dec 14, 2017

KrishanBhasin commented Dec 15, 2017

[JENKINS-47324] - Reduce usage of File.mkdirs() #3173

[JENKINS-47324] - Reduce usage of File.mkdirs() #3173

Conversation

KrishanBhasin commented Dec 4, 2017

Proposed changelog entries

Submitter checklist

Desired reviewers

KrishanBhasin commented Dec 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleg-nenashev commented Dec 4, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KrishanBhasin commented Dec 6, 2017

dwnusbaum commented Dec 6, 2017

dwnusbaum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KrishanBhasin commented Dec 8, 2017

oleg-nenashev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleg-nenashev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleg-nenashev commented Dec 14, 2017

KrishanBhasin commented Dec 15, 2017

oleg-nenashev commented Dec 4, 2017 •

edited

Loading