Add test generator #728

ErikSchierboom · 2025-01-09T20:46:22Z

@tasxatzial This is a very basic test generator prototype. The way it works is that you create a generator.template file in the exercise's .meta directory and then run bin/generate-tests <slug>. This then:

Fetches the problem-specifications repo locally (or pulls the latest when it is already there)
Parses the exercise's canonical-data.json file to find the test cases
Parses the exercise's tests.toml file to exclude test cases that are set to include = false in the tests.toml file (this happens automatically for reimplemented test cases)
Renders the generator.template file using the fetched test cases

The generator itself is located in the generators directory and uses a deps.edn file to manage its dependency.
The generator doesn't yet support any pre-processing of canonical data, but I imagined that to be done via an optional generator.clj file in the .meta directory.

exercises/practice/collatz-conjecture/.meta/tests.clj

ErikSchierboom · 2025-01-10T14:38:13Z

I'll review the actual templates from the PR before we'll merge, as they should be added in separate PRs

exercises/practice/isogram/.meta/generator.template

ErikSchierboom · 2025-01-10T15:04:08Z

exercises/practice/roman-numerals/test/roman_numerals_test.clj


-(deftest one
-  (is (= "I" (roman-numerals/numerals 1))))
+(deftest roman-numerals_test_1


If the tested function has dashes in it (like roman-numerals), do we want to keep that in the function name or would you rather replace it with underscores?

I don't have any strong preference. We can keep them to indicate more clearly the tested function, and use underscores for the _test_1 part. Or, we can use dashes everywhere so that we follow the language conventions. Or, we can use all underscores. Do you have any preference or suggestion?

I don't. I'm happy to go with what you prefer

Let's go with your suggestion then. Dashes for the function name, underscores for the _test_1 part.

I'll also fix the names in my open PRs. Should be trivial to update them all. Will notify when they are done.

@ErikSchierboom All my open PRs are finished, but we both have PRs for pangram and difference-of-squares. Take a look at the pangram PR to see how the tests should be written in that case. We should be explicitly checking for true or false return values. The reason for it is because without explicit checks, someone can pass the tests without actually returning true or false. And given the function ends in ?, we are expecting only true and false to be returned.

I'll go through your PRs later and approve them. Once they're merged I'll update my PRs to just add the template.

And I agree with true? and false?

ErikSchierboom · 2025-01-10T16:34:56Z

I'll check to see if there is a nice handlebars library, as I think we'll be needing conditionals

ErikSchierboom · 2025-01-10T18:31:58Z

I'm switching to hbs for handlebars compatible templates which are nicer to work with and extensible. I'll do some work on more involved exercises

tasxatzial · 2025-01-11T01:39:39Z

I'll review the actual templates from the PR before we'll merge, as they should be added in separate PRs

If you also post the isogram tests in a new PR, i'll close #705

Edit: Actually, i can't close it as i've updated more than the tests. So for isogram, only the template needs to go in a new PR.

ErikSchierboom · 2025-01-11T11:29:55Z

generators/src/templates.clj

+(defn- transform [slug test-cases]
+  (let [transform-file (paths/generator-clojure-file slug)]
+    (if (.exists transform-file)
+      (let [generator-ns (symbol (str slug "-generator"))]
+        (load-file (str transform-file))
+        (if-let [transform-fn (ns-resolve generator-ns (symbol "transform"))]
+          (transform-fn test-cases)
+          test-cases))
+      test-cases)))


This now allows for a generator.clj file next to the generator.template file that can be used to customize the data before it is passed to the template.

ErikSchierboom · 2025-01-11T11:52:41Z

I've happy with what I have now, and it should be enough for a lot of exerciss.

ErikSchierboom · 2025-01-11T12:31:15Z

Open question before we'll merge PRs regarding the template: do we want generator.template or generator.tpl files?

ErikSchierboom · 2025-01-11T14:09:07Z

I think I'll go with .tpl

generators/src/templates.clj

Co-authored-by: Anastasios Chatzialexiou <16361161+tasxatzial@users.noreply.github.com>

ErikSchierboom · 2025-01-12T07:21:51Z

@tasxatzial should be good now!

ErikSchierboom · 2025-01-12T08:06:00Z

@tasxatzial I'll wait for this PR to be merged and then I'll double check all the other PRs

bin/add-practice-exercise

tasxatzial · 2025-01-12T14:22:17Z

The sum of multiples template has been added here accidentally.

tasxatzial · 2025-01-12T14:31:30Z

@tasxatzial I'll wait for this PR to be merged and then I'll double check all the other PRs

Ok, i'll do it as well.

ErikSchierboom · 2025-01-12T14:56:36Z

The sum of multiples template has been added here accidentally.

Fixed!

tasxatzial

Ok, let's proceed then. I believe this is good to go. Feel free to merge.

ErikSchierboom commented Jan 9, 2025

View reviewed changes

exercises/practice/collatz-conjecture/.meta/tests.clj Outdated Show resolved Hide resolved

ErikSchierboom force-pushed the generator-sample branch 2 times, most recently from e9faf34 to 2edba7c Compare January 10, 2025 13:41

ErikSchierboom changed the title ~~Some attempts~~ Add test generator Jan 10, 2025

ErikSchierboom requested a review from tasxatzial January 10, 2025 13:45

ErikSchierboom force-pushed the generator-sample branch 7 times, most recently from 9cd4520 to 328fc65 Compare January 10, 2025 14:36

ErikSchierboom force-pushed the generator-sample branch 5 times, most recently from 490df82 to 66cb526 Compare January 10, 2025 14:43

ErikSchierboom marked this pull request as ready for review January 10, 2025 14:44

ErikSchierboom commented Jan 10, 2025

View reviewed changes

exercises/practice/isogram/.meta/generator.template Outdated Show resolved Hide resolved

ErikSchierboom commented Jan 10, 2025

View reviewed changes

ErikSchierboom added 6 commits January 11, 2025 12:23

Some attempts

66f711d

Move to project

642902c

Use tests.toml

745196f

Lots more work

93217a5

Allow rendering

b0d636e

More templates cleanup

6d72b0d

ErikSchierboom commented Jan 11, 2025

View reviewed changes

This was referenced Jan 11, 2025

isogram: add generator template #732

Merged

two-fer: add generator and regenerate tests #733

Merged

This was referenced Jan 11, 2025

roman-numerals: add generator and regenerate tests #734

Merged

difference-of-squares: add generator and regenerate tests #735

Merged

Change extension

af791cb

tasxatzial added the x:rep/massive Massive amount of reputation label Jan 11, 2025

Use root for prob-specs repo

9c27978

This was referenced Jan 11, 2025

Update add-practice-exercise script to reference test generator #738

Merged

Add generator docs #739

Merged

tasxatzial reviewed Jan 12, 2025

View reviewed changes

generators/src/templates.clj Outdated Show resolved Hide resolved

Update templates.clj

79a5845

Co-authored-by: Anastasios Chatzialexiou <16361161+tasxatzial@users.noreply.github.com>

ErikSchierboom added 2 commits January 12, 2025 08:40

Fix extension

17747ea

Add helpers

7fa9f91

Update add-practice-exercise script to reference test generator (#738)

f6bcb12

ErikSchierboom commented Jan 12, 2025

View reviewed changes

bin/add-practice-exercise Outdated Show resolved Hide resolved

ErikSchierboom added 2 commits January 12, 2025 14:59

Update add-practice-exercise

92a3cb2

Add generator docs (#739)

8bdb911

Delete exercises/practice/sum-of-multiples/.meta/generator.tpl

432d82d

tasxatzial approved these changes Jan 12, 2025

View reviewed changes

ErikSchierboom merged commit 89aec85 into main Jan 12, 2025
2 checks passed

ErikSchierboom deleted the generator-sample branch January 12, 2025 15:05

Uh oh!

Add test generator #728

Add test generator #728

Uh oh!

Conversation

ErikSchierboom commented Jan 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ErikSchierboom commented Jan 10, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ErikSchierboom commented Jan 10, 2025

Uh oh!

ErikSchierboom commented Jan 10, 2025

Uh oh!

tasxatzial commented Jan 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ErikSchierboom commented Jan 11, 2025

Uh oh!

ErikSchierboom commented Jan 11, 2025

Uh oh!

ErikSchierboom commented Jan 11, 2025

Uh oh!

Uh oh!

ErikSchierboom commented Jan 12, 2025

Uh oh!

ErikSchierboom commented Jan 12, 2025

Uh oh!

Uh oh!

tasxatzial commented Jan 12, 2025

Uh oh!

tasxatzial commented Jan 12, 2025

Uh oh!

ErikSchierboom commented Jan 12, 2025

Uh oh!

tasxatzial left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ErikSchierboom commented Jan 9, 2025 •

edited

Loading

tasxatzial commented Jan 11, 2025 •

edited

Loading