Unexpected cider-undef-all outcome wrt. java.lang classes

[codeasone]

I've hit upon an issue with cider-undef-all

Minimal context:

{:paths ["src"]
 :deps  {org.clojure/clojure {:mvn/version "1.10.3"}}}

with src/core.clj

(ns core)

(print (System/getenv "HOME"))

Custom binding I established to evaluate the new undef-all functionality:

:bind (:map cider-mode-map
            ("C-c C-k" . (lambda () (interactive)
                           (cider-undef-all)
                           (cider-load-buffer)
                           (message "Reset vars in namespace")))
            ...)

Expected behavior

Invoking C-c C-k should dwim and I should see the message in the mini-buffer.

Actual behavior

Invoking C-c C-k gives rise to:

  Show: Project-Only All
  Hide: Clojure Java REPL Tooling Duplicates  (33 frames hidden)

2. Unhandled clojure.lang.Compiler$CompilerException
   Error compiling src/core.clj at (3:8)

   No such namespace: System

                 Util.java:  221  clojure.lang.Util/runtimeException
             Compiler.java: 7388  clojure.lang.Compiler/resolveIn
             Compiler.java: 7362  clojure.lang.Compiler/resolve
             Compiler.java: 7323  clojure.lang.Compiler/analyzeSymbol
             Compiler.java: 6772  clojure.lang.Compiler/analyze
             Compiler.java: 6749  clojure.lang.Compiler/analyze
             Compiler.java: 3824  clojure.lang.Compiler$InvokeExpr/parse
             Compiler.java: 7113  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6793  clojure.lang.Compiler/analyze
             Compiler.java: 6749  clojure.lang.Compiler/analyze
             Compiler.java: 3892  clojure.lang.Compiler$InvokeExpr/parse
             Compiler.java: 7113  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6793  clojure.lang.Compiler/analyze
             Compiler.java: 6749  clojure.lang.Compiler/analyze
             Compiler.java: 6124  clojure.lang.Compiler$BodyExpr$Parser/parse
             Compiler.java: 5471  clojure.lang.Compiler$FnMethod/parse
             Compiler.java: 4033  clojure.lang.Compiler$FnExpr/parse
             Compiler.java: 7109  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6793  clojure.lang.Compiler/analyze
             Compiler.java: 7178  clojure.lang.Compiler/eval
             Compiler.java: 7640  clojure.lang.Compiler/load
                      REPL:    1  core/eval9167
                      REPL:    1  core/eval9167
             Compiler.java: 7181  clojure.lang.Compiler/eval
             Compiler.java: 7136  clojure.lang.Compiler/eval
                  core.clj: 3202  clojure.core/eval
                  core.clj: 3198  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1977  clojure.core/with-bindings*
                  core.clj: 1977  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  218  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  217  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  833  java.lang.Thread/run

Seems like undef-all (the use of it I've presented in any case) breaks the following:

All classes in java.lang are automatically imported to every namespace

from https://clojure.org/reference/java_interop

Environment & Version information

CIDER version information

;; CIDER 1.4.0 (Kyiv), nREPL 0.9.0
;; Clojure 1.10.3, Java 17.0.2

Emacs version

GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0) of 2022-03-17

Operating system

Linux mark-pc 5.13.0-39-generic #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[vemv]

Thanks for the issue!

This is what undef does: https://github.com/clojure-emacs/cider-nrepl/blob/0310f1cdb1151e980147d75baa04520ec5e358e8/src/cider/nrepl/middleware/undef.clj#L30-L38

Not sure if undef-all is the piece at fault. Couldn't it be as well (cider-load-buffer) that isn't re-importing classes as you expect?

Which could be either an actual bug, or a slightly unidiomatic choice (there might be a better approach than (cider-load-buffer)), IDK.

[jumarko]

I stumbled upon the same thing: https://clojurians.slack.com/archives/C0617A8PQ/p1651566926013869?thread_ts=1651525142.818839&cid=C0617A8PQ

Perhaps it's the ns macro that makes the java symbols available for the first time but doesn't re-import them after cider unmaps them?

[jumarko]

It's the clojure.lang.Namespace constructor that automatically adds mappings for java.lang classes: https://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/jvm/clojure/lang/Namespace.java#L34

[vemv]

Good find, thanks!

Looks like https://github.com/clojure-emacs/cider-nrepl/blob/0310f1cdb1151e980147d75baa04520ec5e358e8/src/cider/nrepl/middleware/undef.clj#L30-L38 could perform a (remove-ns ns) so that subsequent code loadings will force triggering again the code you have linked to.

It does work, from what I see in the repl.

Anyone willing to give that a spin is welcome to hack on cider-nrepl (see https://github.com/clojure-emacs/cider-nrepl/tree/c0efb99c3cb8fe553d68125ed61090a147042d89#using-the-makefile) and give it a QA session.

If you see it working with your local CIDER, PR welcome.

(I don't have that much time nowadays to do it myself)

[vemv]

@yuhan0: I see you authored undef-all clojure-emacs/cider-nrepl#698 , maybe you could give this one a spin?

[yuhan0]

Thanks! I was mucking around a little with either having undef-all filter out the classes from clojure.lang.RT/DEFAULT_IMPORTS or having load-buffer auto-import them.

Of course using remove-ns is more elegant, and semantically what the op is really about - removing all stale and hidden state from the ns without having to restart the REPL. I wondered if calling it clear-ns would have been better, but there's already the confusingly named ns-refresh and ns-reload commands.

It's not too much a stretch I think for a command named "undef-all" to also delete the ns, after all it should be totally empty by the time the command is done. Surely no one does things like keep reference objects in the ns metadata that would be affected by this?

[yuhan0]

I'm happy to submit a PR to cider-nrepl, just trying to make sure there won't be any further unintended consequences - evidently I don't use Java interop enough to have noticed this obvious bug in all the time I've been using it locally

[vemv]

Surely no one does things like keep reference objects in the ns metadata that would be affected by this?

Yeah it would seem a highly unidiomatic thing to do.

CIDER in particular always performs a fresh (non-cached) query for things like ns/var metadata since that is extremely fast in an already-running JVM.

I'm happy to submit a PR to cider-nrepl

Yes, please go ahead 🙏 it would be greatly appreciated if you can repro now and then check that the issue is gone with the PR (and that undef-all remains otherwise unbroken)

[jumarko]

@yuhan0

undef-all filter out the classes from clojure.lang.RT/DEFAULT_IMPORTS

I think that would be a fine solution.
I guess the user don't have any option how to modify those classes anyway.

I use remove-ns every now and then but it's not without issues.
The other namespaces that refer to such a removed namespace might break transitively and all of them might have to be removed and re-evalted too.

[vemv]

I guess the user don't have any option how to modify those classes anyway.

Users can use ns-unmap. So I'd avocate for remove-ns since it gives one a clean slate. One cannot know if users (or other tools!) used ns-unmap in a way that contradicted our expectations.

The other namespaces that refer to such a removed namespace might break transitively and all of them might have to be removed and re-evalted too.

That seems intrinsic to undef-all? Namespaces don't depend on other namespaces directly (at least off the top of my head), but on the vars of those namespaces.

And undef-all obliterates those vars, so it breaks other namespaces no matter what.

[jumarko]

@vemv about this

The other namespaces that refer to such a removed namespace might break transitively and all of them might have to be removed and re-evalted too.

I cannot give you a deep explanation for this but it's not related to undef-all at all.
I've been using remove-ns for a few years and it sometimes works fine - such as when you use it in for a test ns that is not referred anywhere else.
However, it tends to break quite often if I use it for an application namespace that is required elsewhere.

Consider this example: https://github.com/jumarko/clojure-experiments/blob/master/src/clojure_experiments/aws/logs.clj#L1

• I start by loading the buffer linked above
• I then jump to clojure-experiments.stats.descriptive that is used by the logs ns
• I call remove-ns

clojure-experiments.stats.descriptive> (remove-ns (.getName *ns*))
#namespace[clojure-experiments.stats.descriptive]

• I jump back to the logs ns and try to load the whole buffer => ERROR

Syntax error compiling at (src/clojure_experiments/aws/logs.clj:1:1).
namespace 'clojure-experiments.stats.descriptive' not found

[vemv]

remove-ns has to be used in conjunction with (dosync (alter @#'clojure.core/*loaded-libs* disj your-ns)). Which is exactly what tools.namespace does - it performs both:

https://github.com/clojure/tools.namespace/blob/7a171416beae284c93cea159af5b41aa5f4250eb/src/main/clojure/clojure/tools/namespace/reload.clj#L15-L19

In all fairness, I didn't say that at first so thanks for the scrutiny!

(That would be a good thing to add to the PR + QAing, @yuhan0 )

[yuhan0]

I'm not sure I understand the issue or failure case here - @jumarko did you reload the clojure-experiments.stats.descriptive file after removing the ns?
Or do you expect the logs ns to be able to relocate it on the classpath?

I hope I'm not just reinventing things already found in c.t.n or elsewhere, but must admit I've never used it or any of the various "reloaded" workflows, or understood what they were doing beneath the covers.

[yuhan0]

The few times I tried using Cider's cider-ns-refresh command it put my entire project in an unusable state and I had to restart the REPL altogether.

undef-all is just part of my primitive workflow when dealing with leftover vars or outdated defmultis / protocols hanging around in the REPL state -- just unmap everything in a ns to start it off from a clean slate, followed by a manual cider-load-file.

No dependency graphs or auto refreshing to worry about - if other namespaces are affected I just manually reload them too.

[vemv]

Reimplenting tools.namespace would indeed not make a lot of sense. People interested in it can just use it.

I simply quoted https://github.com/clojure/tools.namespace/blob/7a171416beae284c93cea159af5b41aa5f4250eb/src/main/clojure/clojure/tools/namespace/reload.clj#L15-L19 as a reliable source of how to completely obliterate a ns.

I think the workflow you describe should keep working with that additional cleanup, while also solving #3194