Idea: use any comment block as test

[borkdude]

It would be great if RCF could use any comment block as test, without having to add special syntax or a dependency for a library.

(ns my-lib)

(defn add [a b] (+ a b))

^:hyperfiddle/rcf
(comment
 (add 1 2) ;;=> 3
)

;;; rest of library namespace

This might require a different approach though and I could see that this would not be within RCF's scope. Just wanted to share the idea and see what you think of it. Feel free to migrate this issue to "Github Discussions" once you activate that in this repo, or close this issue.

[ieugen]

Having comment blocks as test without having to require a library in the namespace would be huge IMO.
I have often thought of comment blocks as examples / documentation on how to use the functions and what kind of inputs it expects, even as tests.

But their usability as tests is limited that you have to run them one by one.
I would love to be able to have some tests there without having to pull in the test dependency in production builds.
I think this is where the tricky part lies: How to require things only in comments in a way that works with rcf and tooling.

rcf is a great initiative but I wouldn't want to pull in test libraries / dependencies in production builds.
Maybe it's just me but I think having a reduced surface area in production deployments is important.

[borkdude]

A similar idea here:

https://github.com/lread/test-doc-blocks/blob/main/doc/01-user-guide.adoc#introduction

It takes your code sections from markdown and interprets:

(+ 1 2)
;;=> 3

as a test assertion.

[borkdude]

I might do a little prototype of such a test runner using rewrite-clj.

[dustingetz]

Thank you for sharing your idea! Parsing comment is tricky:

• Async tests have additional syntax - rcf/% and rcf/!
• ClojureScript support, for each way to build ClojureScript (Shadow comes with its own module loader)
• Support for all possible ways to evaluate a test form
  • Support for naive "send buffer to REPL" editor commands in all editors
  • Support for sending a single test form to the REPL
  • Support for running tests in transitive dependencies when file is loaded at the REPL
• Support for bypassing tests on REPL initial startup so REPL starts up as fast as possible
• Support for CI
• support for pattern matching and future syntax extensions like destructuring test results
• support for custom test macros
• support for deep assertions like (tests (let [x 1] (inc x) := 2))
• support for syntax highlighting and structural editing
• etc

There is also the question of is it moral, what are the risks, is it worth it? What would RH do?

[borkdude]

Whether if it is moral, I think the question should be: should it be explicit? I think yes. That is why I put an explicit ^:hyperfiddle/rcf value on the comment form.

I just posted this idea for a lack of "Github Discussions", I think that is a much better fit, as this idea isn't really actionable.

[dustingetz]

Ah, thank you for pointing out that you tag the comment for RCF evaluation, I missed that!

[dustingetz]

rcf is a great initiative but I wouldn't want to pull in test libraries / dependencies in production builds. Maybe it's just me but I think having a reduced surface area in production deployments is important.

Thank you @ieugen for the feedback! You can still separate your RCF tests into a tests folder, we merely remove the requirement. At Hyperfiddle we like to put 2-3 lightweight "pure function" tests next to the defn as example uses, and heavier test coverage and/or tests requiring setup and dependencies in separate files that only work under an alias. We're still debating if "heavy tests" need to be split out into a parallel folder (as that seems to be a historical requirement of legacy rest runners that watch the filesystem and rerun tests on file changes – which RCF's REPL-first design does not do. You can still separate a test folder if you want.)

[borkdude]

@dustingetz Maybe going a bit off-topic here, but how does the infix notation play well with the REPL?

E.g.: (is (= 1 2)) you can eval in the REPL, but 1 := 2 not so much?

[dustingetz]

(tests 1 := 2)

[borkdude]

:-) But that's not how you normally work with rich comment forms?

[dustingetz]

put your cursor on the 1 and send form to repl? I am confused

[ggeoffrey]

For RCF to use any comment block as test, without having to add special syntax or a dependency, I assume we would have to hook onto the clojure reader. I’m not sure how, but if it’s possible it would be big.

If changing ;; => to => is acceptable, then data readers could work:

#rcf/tests
(comment
 (add 1 2) => 3
)

It saves a require and the #rcf/tests line can be commented out easily. Not sure if it’s a good tradeoff 🤔

[borkdude]

@ggeoffrey I think you would have to go with an approach using rewrite-clj, or perhaps edamame or clojure.tools.reader, to read top level comments and process them differently.

See here for such an approach: #49 (comment)

[borkdude]

Using data readers would still require you to register a data reader using a dependency (or no-op).

[ggeoffrey]

Thank you for the link. It seems these approaches are unfortunately not compatible with a live REPL. As far as I can tell, the Clojure LispReader can’t be extended or overloaded. So it would force us into a completely different workflow (parsing files vs live coding).

[ggeoffrey]

:-) But that's not how you normally work with rich comment forms?

The idea is, in both of these forms what’s on right is like an annotation. We usually evaluate the left hand side at the REPL.

(comment
  1 ;; => 1
  )

(tests
  1 := 1)

IMO infix vs prefix should not be a constraint. Infix should just be sugar, meaning prefix could work too.

[borkdude]

Makes sense now, thanks.

[borkdude]

I think Stuart Halloway's transcriptor is a nice alternative approach:

https://github.com/cognitect-labs/transcriptor

(+ 1 2 3) ;; statement
(check! #{6}) ;; assertion

These appear in successive order and can be evaluated separately (to check if the assertion is valid during dev time for example).

[dustingetz]

Thanks for pointing that out – I don't think we considered that style. What do you think of

(require '[hyperfiddle.rcf :as rcf :refer [=> tests]])
(tests
  (map inc [1 2 3])
  (=> '(2 3 4)))

(tests (map inc [1 2 3]) (=> '(2 3 4)))

clojure.core/=>
Syntax error compiling at (REPL:0:0).
No such var: clojure.core/=>

[borkdude]

Seems like an improvement to me!

[borkdude]

It seems https://github.com/matthewdowney/rich-comment-tests ran with this idea! :)

[marksto]

It also worth noting that neither of these two brilliant libs (RCF nor RCT) eliminate the problem with dangling regular (uninstrumented) comment forms, that tend to "rot" over time because they are not called on frequently enough. An SCA tool, i.e. clj-kondo, is the only way to keep such comments up to date and workable. However, things go sour when you tag these forms with ^:clj-kondo/ignore to make REPL-based development easier. We've abandoned this practice and re-configured clj-kondo on one of our projects to prevent this issue.