Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NoSuchFileException related to Intellij __jb_tmp__ safe write files #471

Open
kennyjwilli opened this issue Jun 14, 2016 · 18 comments
Open
Assignees
Labels

Comments

@kennyjwilli
Copy link

kennyjwilli commented Jun 14, 2016

I seem to be inconsistently getting this error when writing the target dirs after a reload. It goes away if I restart my boot process.

Writing main.cljs.edn...
Compiling ClojureScript...
• main.js
Writing target dir(s)...
java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: target/some/project/myns.cljs___jb_tmp___
      java.nio.file.NoSuchFileException: target/some/project/myns.cljs___jb_tmp___
    file: "target/some/project/myns.cljs___jb_tmp___"
sun.nio.fs.UnixException.translateToIOException               UnixException.java:   86
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  102
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  107
   sun.nio.fs.UnixFileSystemProvider.implDelete      UnixFileSystemProvider.java:  244
   sun.nio.fs.AbstractFileSystemProvider.delete  AbstractFileSystemProvider.java:  103
                                            ...                                       
                        boot.filesystem/delete!                   filesystem.clj:  163
                         boot.filesystem/patch!                   filesystem.clj:  178
                                            ...                                       
                clojure.core/apply/invokeStatic                         core.clj:  652
                        clojure.core/partial/fn                         core.clj: 2534
                                            ...                                       
      boot.core/fileset-syncer/fn/iter/fn/fn/fn                         core.clj:  930
            clojure.core/binding-conveyor-fn/fn                         core.clj: 1938
                                            ...                                       
Elapsed time: 0.162 sec

From the discussion on #boot on Slack it seems the __jb_tmp__ is an IntelliJ file that is semi-atomically created (using SafeFileOutputStream according to cfleming).

@arichiardi
Copy link
Contributor

arichiardi commented Jun 14, 2016

This would also solve the issue with emacs' lock files.

@kennyjwilli
Copy link
Author

Example project with error below.
https://github.com/kennyjwilli/boot-files-issue

Writing main.cljs.edn...
Compiling ClojureScript...
• main.js
Writing target dir(s)...
java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: target/boot_files/core.cljs___jb_tmp___
      java.nio.file.NoSuchFileException: target/boot_files/core.cljs___jb_tmp___
    file: "target/boot_files/core.cljs___jb_tmp___"
sun.nio.fs.UnixException.translateToIOException               UnixException.java:   86
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  102
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  107
   sun.nio.fs.UnixFileSystemProvider.implDelete      UnixFileSystemProvider.java:  244
   sun.nio.fs.AbstractFileSystemProvider.delete  AbstractFileSystemProvider.java:  103
                                            ...                                       
                        boot.filesystem/delete!                   filesystem.clj:  163
                         boot.filesystem/patch!                   filesystem.clj:  178
                                            ...                                       
                clojure.core/apply/invokeStatic                         core.clj:  652
                        clojure.core/partial/fn                         core.clj: 2534
                                            ...                                       
      boot.core/fileset-syncer/fn/iter/fn/fn/fn                         core.clj:  930
            clojure.core/binding-conveyor-fn/fn                         core.clj: 1938
                                            ...                                       
Elapsed time: 0.101 sec

@kennyjwilli
Copy link
Author

It appears that the exception is thrown in every boot process (e.g. I have boot web-dev running in one terminal and boot repl running in another. When the exception is thrown in the boot web-dev instance the exception also gets thrown in the boot repl terminal).

@micha micha added the Bug label Jul 30, 2016
@micha micha added this to the 2.7.0 milestone Jul 30, 2016
@micha micha added the ready label Jul 30, 2016
@micha micha self-assigned this Jul 30, 2016
@micha micha closed this as completed in e77ef6c Jul 30, 2016
@micha micha removed the ready label Jul 30, 2016
@kennyjwilli
Copy link
Author

This is still occuring for me. The exception is slightly different though. Related problem?

Uncaught exception in thread Thread-29:
                                  java.lang.Thread.run                   Thread.java:  745
                                                   ...                                    
                               boot.core/watch-dirs/fn                      core.clj:  751
                    boot.core/set-user-dirs!/on-change                      core.clj:  194
                             boot.core/sync-user-dirs!                      core.clj:  139
                                                   ...                                    
                                      boot.core/patch!                      core.clj:  724
                                   clojure.core/reduce                      core.clj: 6704
                           clojure.core.protocols/fn/G                 protocols.clj:   13
                             clojure.core.protocols/fn                 protocols.clj:   75
                    clojure.core.protocols/iter-reduce                 protocols.clj:   49
                               boot.core/patch!/merge'                      core.clj:  722
                                                   ...                                    
                                boot.filesystem/mktree                filesystem.clj:  113
                                                   ...                                    
                              boot.file/walk-file-tree                      file.clj:   59
                      java.nio.file.Files.walkFileTree                    Files.java: 2706
                     java.nio.file.FileTreeWalker.next           FileTreeWalker.java:  372
                    java.nio.file.FileTreeWalker.visit           FileTreeWalker.java:  276
            java.nio.file.FileTreeWalker.getAttributes           FileTreeWalker.java:  225
                    java.nio.file.Files.readAttributes                    Files.java: 1737
     sun.nio.fs.LinuxFileSystemProvider.readAttributes  LinuxFileSystemProvider.java:   99
      sun.nio.fs.UnixFileSystemProvider.readAttributes   UnixFileSystemProvider.java:  144
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes   UnixFileAttributeViews.java:   55
         sun.nio.fs.UnixException.rethrowAsIOException            UnixException.java:  107
         sun.nio.fs.UnixException.rethrowAsIOException            UnixException.java:  102
       sun.nio.fs.UnixException.translateToIOException            UnixException.java:   86
java.nio.file.NoSuchFileException: src/compute/ui_frontend/re_frame.cljs___jb_old___
    file: "src/compute/ui_frontend/re_frame.cljs___jb_old___"

@martinklepsch
Copy link
Member

@kennyjwilli thanks for reporting!

  1. Is the repro (https://github.com/kennyjwilli/boot-files-issue) still intact and can you reproduce the issue there?
  2. Could you please add some minimal information what to run to reproduce the issue?

@martinklepsch martinklepsch reopened this Jan 10, 2018
@martinklepsch martinklepsch removed this from the 2.7.0 milestone Jan 10, 2018
@kennyjwilli
Copy link
Author

I'm struggling to get it to reproduce outside of our large CLJS project. We have a project with 50+ CLJS namespaces and occasionally while recompiling, that error is thrown. That project takes 2s or more to compile compared to the 0.2s the boot-files-issue repo takes. I'll take a look at Boot's code to see if this is an easy fix.

@kennyjwilli
Copy link
Author

It seems to happen if there are rapid file changes while it is compiling the CLJS.

@kennyjwilli
Copy link
Author

It seems like these files are created from IntelliJ's "safe write" feature: https://stackoverflow.com/questions/23271895/temporary-jb-old-file-in-phpstorm-and-webstorm-causing-errors.

@kennyjwilli
Copy link
Author

Disabling "safe write" fixes the issue.

It's tough to debug when I can't get the Cursive debugger to break on the NoSuchFileException. I'd really like to know what params are being passed to each function so I at least can replicate the behavior in the REPL.

It's also interesting that I can only reliably reproduce the error when a CLJS REPL is running.

@kennyjwilli
Copy link
Author

kennyjwilli commented Jan 12, 2018

Best guess at what is happening:

  1. File is saved in editor with "safe write" enabled
  2. IntelliJ writes the changed file to its temp file (e.g. src/compute/ui_frontend/re_frame.cljs___jb_tmp___)
  3. Boot creates the filesystem tree which includes the jb_tmp file
  4. IntelliJ safely writes to the original file and deletes its temporary file.
  5. Boot attempts to walk the file tree it created in 1 and a java.nio.file.NoSuchFileException is thrown because the jb_tmp file was included in the file tree but deleted before the file tree could be walked.

This is a race condition which would explain why it only happens occasionally.

@kennyjwilli
Copy link
Author

I have added the regex .*\_\_\_$ to my .bootignore which should cause Boot to ignore any files ending with ___ but the issue still occurs.

@martinklepsch
Copy link
Member

There’s an outstanding PR (which will be part of the next release) that makes bootignore behavior more consistent: #663

You could try merging those changes and rebuilding boot locally to see if that makes bootignore work as expected.

I’m also wondering if there are ways to make IntelliJ write those files elsewhere? I roughly remember similar issues related to emacs storing temporary files in watched directories. (I think there’s some stuff in the wiki about this, will update with a link later.)

@martinklepsch
Copy link
Member

martinklepsch commented Jan 13, 2018

It seems there are other people also having issues with IntelliJ's safe write feature: cgrand/enlive#143 — the universal solution seems to be disabling safe write 😏

@martinklepsch martinklepsch changed the title Allow other programs to create files that boot doesn't know about NoSuchFileException related to Intellij __jb_tmp__ safe write files Jan 13, 2018
@martinklepsch
Copy link
Member

@kennyjwilli I updated the name if this ticket, feel free to change if you feel this does not accurately reflect the nature of this problem.

@martinklepsch
Copy link
Member

martinklepsch commented Jan 13, 2018

Rereading #663 it seems that this won't affect the issue at hand since .bootignore really just takes care of ignoring directories, not individual files.

@martinklepsch
Copy link
Member

martinklepsch commented Jan 13, 2018

Ideas for changes that could help future users avoid this kind of trouble:

  • Document it in the Cursive page in the wiki
  • Catch NoSuchFileExceptions occuring in boot.file/walk-file-tree and print additional information if the filepath contains __jb_old__ or similar

@witek
Copy link

witek commented Jan 31, 2018

I have the same problem using Spacemacs:

Exception in thread "Thread-15" java.nio.file.NoSuchFileException: src/tao/.#tests_spa.cljs at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.nio.file.Files.readAttributes(Files.java:1737) at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225) at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372) at java.nio.file.Files.walkFileTree(Files.java:2706) at boot.file$walk_file_tree.doInvoke(file.clj:59) at clojure.lang.RestFn.invoke(RestFn.java:425) at boot.filesystem$mktree.doInvoke(filesystem.clj:113) at clojure.lang.RestFn.invoke(RestFn.java:439) at boot.core$patch_BANG_$merge_SINGLEQUOTE___728.invoke(core.clj:722) at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49) at clojure.core.protocols$fn__7839.invokeStatic(protocols.clj:75) at clojure.core.protocols$fn__7839.invoke(protocols.clj:75) at clojure.core.protocols$fn__7781$G__7776__7794.invoke(protocols.clj:13) at clojure.core$reduce.invokeStatic(core.clj:6748) at clojure.core$reduce.invoke(core.clj:6730) at boot.core$patch_BANG_.doInvoke(core.clj:724) at clojure.lang.RestFn.invoke(RestFn.java:464) at boot.core$sync_user_dirs_BANG_.invoke(core.clj:139) at boot.core$set_user_dirs_BANG_$on_change__459.invoke(core.clj:194) at boot.core$watch_dirs$fn__737.invoke(core.clj:751) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748)

@larkery
Copy link

larkery commented Nov 7, 2018

Whilst waiting for this problem to go away I did this hack:

(require 'boot.filesystem)
(in-ns 'boot.filesystem)
(let [mkvisitor-0 mkvisitor]
  (defn mkvisitor [^Path root tree & {:keys [ignore]}]
    (let [^SimpleFileVisitor v0 (mkvisitor-0 root tree :ignore ignore)]
      (proxy [SimpleFileVisitor] []
        (postVisitDirectory [t e] (.postVisitDirectory v0 t e))
        (preVisitDirectory [path attr] (.preVisitDirectory v0 path attr))
        (visitFile [path attr] (.visitFile v0 path attr))
        (visitFileFailed [path exc] FileVisitResult/CONTINUE)))))
(in-ns 'boot.user)

This appears to make this class of issue go away for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants