Skip to content

Building the AST is not practical because it runs tests - how to prevent that? #426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Andre0991 opened this issue Jul 7, 2018 · 10 comments
Closed

Building the AST is not practical because it runs tests - how to prevent that? #426

Andre0991 opened this issue Jul 7, 2018 · 10 comments
Labels
enhancement

Comments

@Andre0991
Copy link

cljr-find-usages and friends have to create the AST for the whole project.

This is not practical for my use case, as it would need to run all the tests, which might take a long time. Also, some tests actually break the AST creation with Method code too large!.

Is there are way to configure clj-refactor so that it ignore certain namespaces when building the AST?
@expez
Copy link
Member

expez commented Jul 8, 2018

The easiest way to solve this would be wrap your tests in a function, which your test-runner calls, instead of having (run-tests!) as a top-level form. That way your tests won't run as part of the analysis step performed by tools.analyzer.

There's code in place to ignore certain paths that's already been written on the middleware side of things. It would be easy to expose that to the client as a config setting, and we should probably do that.

However, if you ignore your test directory, then renamings using cljr-rename-symbol won't be propagated to these namespaces because they haven't been analyzed! This could be a source of both breakage and confusion :(

In other words, you'd unfortunately have to refactor a little even with a new config setting, e.g. to have a file with nothing except a require form and (run-tests!) as your the form.

expez added a commit that referenced this issue Jul 8, 2018
@expez
[Fix #426] Expose variable to ignore paths in the middleware
a5888e4 
This makes it possible to easily exclude certain files and directories
for analysis.
expez added a commit that referenced this issue Jul 8, 2018
@expez
[Fix #426] Expose variable to ignore paths in the middleware
4c9d077 
This makes it possible to easily exclude certain files and directories
for analysis.
expez added a commit that referenced this issue Jul 8, 2018
@expez
[Fix #426] Expose variable to ignore paths in the middleware
fbd2f16 
This makes it possible to easily exclude certain files and directories
for analysis.
@expez
Copy link
Member

expez commented Jul 8, 2018

You should be able to apply this minimal patch manually, if you're so inclined, to see if it solves your problem @Andre0991: #427

expez added a commit that referenced this issue Jul 8, 2018
@expez
[Fix #426] Expose variable to ignore paths in the middleware
55b72af 
This makes it possible to easily exclude certain files and directories
for analysis.
@Andre0991
Copy link
Author

Andre0991 commented Jul 8, 2018 •
edited
Loading

The easiest way to solve this would be wrap your tests in a function, which your test-runner calls, instead of having (run-tests!) as a top-level form. That way your tests won't run as part of the analysis step performed by tools.analyzer.

The issue is that I use Midje, which runs the test when the test form is evaluated. I actually like this behaviour, it makes testing more interactive - but it does make evaluating the whole project impractical.

However, if you ignore your test directory, then renamings using cljr-rename-symbol won't be propagated to these namespaces because they haven't been analyzed! This could be a source of both breakage and confusion :(
In other words, you'd unfortunately have to refactor a little even with a new config setting, e.g. to have a file with nothing except a require form and (run-tests!) as your the form.

You're right, that would be confusing. I'm not sure if there's a good solution for this using Midje :(

Thanks for the patch. I manually applied it, but I got the following error:

error in process filter: cljr--get-error-value: Error in nrepl-refactor: java.util.concurrent.ExecutionException: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.regex.Pattern
 at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    java.util.concurrent.FutureTask.get (FutureTask.java:192)
    clojure.core$deref_future.invokeStatic (core.clj:2292)
    clojure.core$future_call$reify__8097.deref (core.clj:6894)
    clojure.core$deref.invokeStatic (core.clj:2312)
    clojure.core$deref.invoke (core.clj:2298)
    refactor_nrepl.find.find_symbol$find_symbol.invokeStatic (find_symbol.clj:242)
    refactor_nrepl.find.find_symbol$find_symbol.invoke (find_symbol.clj:235)
    clojure.lang.Var.invoke (Var.java:381)
    refactor_nrepl.middleware$find_symbol_reply.invokeStatic (middleware.clj:88)
    refactor_nrepl.middleware$find_symbol_reply.invoke (middleware.clj:85)
    refactor_nrepl.middleware$wrap_refactor$fn__22106.invoke (middleware.clj:203)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_undef$fn__17917.invoke (nrepl.clj:484)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_ns$fn__17837.invoke (nrepl.clj:297)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_debug$fn__17787.invoke (nrepl.clj:181)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_enlighten$fn__17795.invoke (nrepl.clj:207)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_inspect$fn__17821.invoke (nrepl.clj:244)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    clojure.tools.nrepl.middleware.pr_values$pr_values$fn__16887.invoke (pr_values.clj:22)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_test$fn__17893.invoke (nrepl.clj:435)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl.middleware.pprint$handle_pprint_fn.invokeStatic (pprint.clj:49)
    cider.nrepl.middleware.pprint$handle_pprint_fn.invoke (pprint.clj:44)
    clojure.lang.Var.invoke (Var.java:385)
    cider.nrepl$wrap_pprint_fn$fn__17725.invoke (nrepl.clj:89)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    com.billpiel.sayid.nrepl_middleware$wrap_sayid$fn__25160.invoke (nrepl_middleware.clj:577)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_trace$fn__17901.invoke (nrepl.clj:455)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_tracker$fn__17909.invoke (nrepl.clj:473)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    clojure.tools.nrepl.middleware.session$session$fn__17039.invoke (session.clj:192)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    clojure.tools.nrepl.server$handle_STAR_.invokeStatic (server.clj:19)
    clojure.tools.nrepl.server$handle_STAR_.invoke (server.clj:16)
    clojure.tools.nrepl.server$handle$fn__17112.invoke (server.clj:28)
    clojure.core$binding_conveyor_fn$fn__5476.invoke (core.clj:2022)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:266)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.regex.Pattern
 at clojure.core$re_matcher.invokeStatic (core.clj:4789)
    clojure.core$re_find.invokeStatic (core.clj:4838)
    clojure.core$re_find.invoke (core.clj:4838)
    refactor_nrepl.core$ignore_dir_on_classpath_QMARK_$fn__21777.invoke (core.clj:38)
    clojure.lang.PersistentVector.reduce (PersistentVector.java:341)
    clojure.core$reduce.invokeStatic (core.clj:6747)
    clojure.core$reduce.invoke (core.clj:6730)
    refactor_nrepl.core$ignore_dir_on_classpath_QMARK_.invokeStatic (core.clj:37)
    refactor_nrepl.core$ignore_dir_on_classpath_QMARK_.invoke (core.clj:34)
    refactor_nrepl.core$dirs_on_classpath$fn__21783.invoke (core.clj:59)
    clojure.core$complement$fn__5391.invoke (core.clj:1433)
    clojure.core$filter$fn__5614.invoke (core.clj:2813)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.RT.seq (RT.java:528)
    clojure.core$seq__5124.invokeStatic (core.clj:137)
    clojure.core$map$fn__5587.invoke (core.clj:2738)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.RT.seq (RT.java:528)
    clojure.core$seq__5124.invokeStatic (core.clj:137)
    clojure.core$apply.invokeStatic (core.clj:652)
    clojure.core$mapcat.invokeStatic (core.clj:2775)
    clojure.core$mapcat.doInvoke (core.clj:2775)
    clojure.lang.RestFn.invoke (RestFn.java:423)
    refactor_nrepl.core$find_in_project.invokeStatic (core.clj:179)
    refactor_nrepl.core$find_in_project.invoke (core.clj:176)
    refactor_nrepl.find.find_macros$find_macro_definitions_in_project.invokeStatic (find_macros.clj:85)
    refactor_nrepl.find.find_macros$find_macro_definitions_in_project.invoke (find_macros.clj:82)
    refactor_nrepl.find.find_macros$find_macro.invokeStatic (find_macros.clj:212)
    refactor_nrepl.find.find_macros$find_macro.invoke (find_macros.clj:207)
    refactor_nrepl.find.find_symbol$find_symbol$fn__110003.invoke (find_symbol.clj:237)
    clojure.core$binding_conveyor_fn$fn__5476.invoke (core.clj:2022)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:266)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    java.lang.Thread.run (Thread.java:748)

error in process filter: Error in nrepl-refactor: java.util.concurrent.ExecutionException: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.regex.Pattern
 at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    java.util.concurrent.FutureTask.get (FutureTask.java:192)
    clojure.core$deref_future.invokeStatic (core.clj:2292)
    clojure.core$future_call$reify__8097.deref (core.clj:6894)
    clojure.core$deref.invokeStatic (core.clj:2312)
    clojure.core$deref.invoke (core.clj:2298)
    refactor_nrepl.find.find_symbol$find_symbol.invokeStatic (find_symbol.clj:242)
    refactor_nrepl.find.find_symbol$find_symbol.invoke (find_symbol.clj:235)
    clojure.lang.Var.invoke (Var.java:381)
    refactor_nrepl.middleware$find_symbol_reply.invokeStatic (middleware.clj:88)
    refactor_nrepl.middleware$find_symbol_reply.invoke (middleware.clj:85)
    refactor_nrepl.middleware$wrap_refactor$fn__22106.invoke (middleware.clj:203)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_undef$fn__17917.invoke (nrepl.clj:484)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_ns$fn__17837.invoke (nrepl.clj:297)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_debug$fn__17787.invoke (nrepl.clj:181)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_enlighten$fn__17795.invoke (nrepl.clj:207)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_inspect$fn__17821.invoke (nrepl.clj:244)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    clojure.tools.nrepl.middleware.pr_values$pr_values$fn__16887.invoke (pr_values.clj:22)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_test$fn__17893.invoke (nrepl.clj:435)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl.middleware.pprint$handle_pprint_fn.invokeStatic (pprint.clj:49)
    cider.nrepl.middleware.pprint$handle_pprint_fn.invoke (pprint.clj:44)
    clojure.lang.Var.invoke (Var.java:385)
    cider.nrepl$wrap_pprint_fn$fn__17725.invoke (nrepl.clj:89)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    com.billpiel.sayid.nrepl_middleware$wrap_sayid$fn__25160.invoke (nrepl_middleware.clj:577)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_trace$fn__17901.invoke (nrepl.clj:455)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    cider.nrepl$wrap_tracker$fn__17909.invoke (nrepl.clj:473)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    clojure.tools.nrepl.middleware.session$session$fn__17039.invoke (session.clj:192)
    clojure.tools.nrepl.middleware$wrap_conj_descriptor$fn__16688.invoke (middleware.clj:22)
    clojure.tools.nrepl.server$handle_STAR_.invokeStatic (server.clj:19)
    clojure.tools.nrepl.server$handle_STAR_.invoke (server.clj:16)
    clojure.tools.nrepl.server$handle$fn__17112.invoke (server.clj:28)
    clojure.core$binding_conveyor_fn$fn__5476.invoke (core.clj:2022)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:266)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.regex.Pattern
 at clojure.core$re_matcher.invokeStatic (core.clj:4789)
    clojure.core$re_find.invokeStatic (core.clj:4838)
    clojure.core$re_find.invoke (core.clj:4838)
    refactor_nrepl.core$ignore_dir_on_classpath_QMARK_$fn__21777.invoke (core.clj:38)
    clojure.lang.PersistentVector.reduce (PersistentVector.java:341)
    clojure.core$reduce.invokeStatic (core.clj:6747)
    clojure.core$reduce.invoke (core.clj:6730)
    refactor_nrepl.core$ignore_dir_on_classpath_QMARK_.invokeStatic (core.clj:37)
    refactor_nrepl.core$ignore_dir_on_classpath_QMARK_.invoke (core.clj:34)
    refactor_nrepl.core$dirs_on_classpath$fn__21783.invoke (core.clj:59)
    clojure.core$complement$fn__5391.invoke (core.clj:1433)
    clojure.core$filter$fn__5614.invoke (core.clj:2813)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.RT.seq (RT.java:528)
    clojure.core$seq__5124.invokeStatic (core.clj:137)
    clojure.core$map$fn__5587.invoke (core.clj:2738)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.RT.seq (RT.java:528)
    clojure.core$seq__5124.invokeStatic (core.clj:137)
    clojure.core$apply.invokeStatic (core.clj:652)
    clojure.core$mapcat.invokeStatic (core.clj:2775)
    clojure.core$mapcat.doInvoke (core.clj:2775)
    clojure.lang.RestFn.invoke (RestFn.java:423)
    refactor_nrepl.core$find_in_project.invokeStatic (core.clj:179)
    refactor_nrepl.core$find_in_project.invoke (core.clj:176)
    refactor_nrepl.find.find_macros$find_macro_definitions_in_project.invokeStatic (find_macros.clj:85)
    refactor_nrepl.find.find_macros$find_macro_definitions_in_project.invoke (find_macros.clj:82)
    refactor_nrepl.find.find_macros$find_macro.invokeStatic (find_macros.clj:212)
    refactor_nrepl.find.find_macros$find_macro.invoke (find_macros.clj:207)
    refactor_nrepl.find.find_symbol$find_symbol$fn__110003.invoke (find_symbol.clj:237)
    clojure.core$binding_conveyor_fn$fn__5476.invoke (core.clj:2022)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:266)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    java.lang.Thread.run (Thread.java:748)

I set the variable as follows:

  (setq cljr-middleware-ignored-paths '("test\.*"))

@expez
Copy link
Member

expez commented Jul 9, 2018

Thanks, think we're missing a coercion step in the middleware to turn these into regexes.

@expez
Copy link
Member

expez commented Jul 23, 2018

The new snapshot is out @Andre0991. Let please me know if this is solved now.

@expez expez closed this as completed in 46f5419 Jul 25, 2018
@Andre0991
Copy link
Author

Andre0991 commented Oct 2, 2018 •
edited
Loading

Thanks @expez .
I haven't been able to make it ignore my tests files - is my regex wrong or something?

Caused by: java.lang.IllegalStateException: refactor-nrepl is unable to build an AST for purgatory.logic.agreement.validation-test. tools.analyzer encountered the following problem: java.lang.RuntimeException: Method code too large!, compiling:(file:/Users/user/dev/project/test/project/logic/agreement/validation_test.clj:245:3)

Those are my attempts:

(setq cljr-middleware-ignored-paths '("test\\.*"))
(setq cljr-middleware-ignored-paths '("\\.*_test.clj"))

@expez
Copy link
Member

expez commented Oct 2, 2018

@Andre0991 change the \\ to \ in either of those and you should be good to go (java regex)

@Andre0991
Copy link
Author

@expez, for some reason it's always triggering the same error. How can I debug that? 🤔

It complains about file:/Users/myUser/dev/nu/purgatory/test/purgatory/logic/agreement/validation_test.clj:245:3.

My clj-refactor version is 20180826.2149.

I basically set cljr-middleware-ignored-paths in my dotfile as above (I removed the slash and tried to use other formats too).

@expez
Copy link
Member

expez commented Oct 3, 2018

Looks like when this feature was implemented it was only meant to be used to ignore directories. Guess that could've been made a little clearer.

Honestly we should fix that, so any arbitrary file can be excluded.

Try again with "/test/" to ignore match on the test directory in the path.

@Andre0991
Copy link
Author

Oh, that makes sense.

Ignoring all paths with /test/ would be OK for my current use case too.
But still, it won't work with "test" either :/

I'm suspecting nrepl is not actually considering it.

Let me know if you need more information.

@mbuczko mbuczko mentioned this issue Dec 15, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@expez @Andre0991