cabal init --tests does not create any tests if not using --lib{andexe}

[kindaro]

Describe the bug

cabal init --tests does not create any tests.

To Reproduce

% cabal init --tests
Warning: The package list for 'hackage.haskell.org' is 23 days old.
Run 'cabal update' to get the latest list of available packages.

Guessing dependencies...

Generating LICENSE...
Warning: unknown license type, you must put a copy in LICENSE yourself.
Generating CHANGELOG.md...
Generating app/Main.hs...
Generating example.cabal...

Warning: no synopsis given. You should edit the .cabal file and add one.
You may want to edit the .cabal file and add a Description field.
% tree
.
├── app
│   └── Main.hs
├── CHANGELOG.md
└── example.cabal

1 directory, 3 files

Expected behavior

A directory is created with some tests. The tests are wired into example.cabal as appropriate.

System information

% cabal --version
cabal-install version 3.4.0.0
compiled using version 3.4.0.0 of the Cabal library 
% uname
Linux

[fgaz]

This issue is happening because by default cabal initializes a package with only an executable, and there is no way to wire tests directly to an executable without a library in the middle, as far as I can see.

Pheraps --tests without --lib or --libandexe should error out and point to those flags, or it should imply them. Or it could generate a non-wired testsuite, though that's probably less useful.

As a workaround, cabal init --libandexe --tests does generate a test.

If on the other hand you meant to initialize tests in an already existing package, this is not currently possible, though there are plans to allow editing .cabal files with cabal commands: #7337.

[kindaro]

This issue is happening because by default cabal initializes a package with only an executable, and there is no way to wire tests directly to an executable without a library in the middle, as far as I can see.

However, observe that you can have a test suite without either a library or an executable, and the default test suite being generated by cabal init does not depend in any way on either. Simply run cabal init --lib --tests and edit out the library stanza — the test suite still builds and runs!

That is to say, the test suite is not and does not need to be wired to any other stanza.

[fgaz]

That is true, but what would such a testsuite test? Answering that myself: other packages in a project, maybe in combination. Or there could be an overlap in the source directories, though I think that's discouraged.

So maybe it would be best to generate the testsuite but also point out in a warning/notice that the user may want to use --libandexe to test the code in the same package (without rebuilding stuff due to overlapping src-dirs).

/cc @emilypi @ptkato

[fgaz]

and the default test suite being generated by cabal init does not depend in any way on either

this may be another issue. when generating a lib+exe, the exe is made to depend on the lib since that's what one usually wants, so maybe the same should apply to lib+test

[kindaro]

My case is such that I needed a package with exactly one test-suite stanza. I was having an issue with property checks in a big project, and I wanted to write some small examples separately. That is to say, a feature of generating a standalone test suite would not be without use.

[emilypi]

Hmmm. First off, I would not call this a bug @fgaz, since it's explicitly the behavior of the init script to disallow creating test suite stanzas without an associated library. Primarily because init is for initializing new projects in the sense of libraries or executables. Second, I can see this behavior being legitimate, but it is very non-standard. We'd have to add support for standalone test-suites, since --tests, despite its historical naming convention, only provides a flag for initializing test suites. It doesn't tell the script that it wants to create a test stanza as a project type. To get this behavior, we probably need to add a new flag.

Ideally, something like this:

λ P temp → λ git master* → ../dist-newstyle/build/x86_64-osx/ghc-8.10.4/cabal-install-3.5.0.0/x/cabal/noopt/build/cabal/cabal init --simple --test-suite --minimal

Using cabal specification: 3.0

Warning: unknown license type, you must put a copy in LICENSE yourself.
Creating CHANGELOG.md...
Creating file CHANGELOG.md...
Creating directory ./test...
Creating file test/Main.hs...
Creating file temp.cabal...

Warning: no synopsis given. You should edit the .cabal file and add one.
You may want to edit the .cabal file and add a Description field.
λ P temp → λ git master* → tree
.
├── CHANGELOG.md
├── cabal.tix
├── temp.cabal
└── test
    └── Main.hs

1 directory, 4 files
λ P temp → λ git master* → rm cabal.tix 
λ P temp → λ git master* → tree        
.
├── CHANGELOG.md
├── temp.cabal
└── test
    └── Main.hs

1 directory, 3 files
λ P temp → λ git master* → cat temp.cabal 
cabal-version:   3.0
name:            temp
version:         0.1.0.0
license:         NONE
build-type:      Simple
extra-doc-files: CHANGELOG.md

test-suite temp-test
    default-language: Haskell2010
    type:             exitcode-stdio-1.0
    hs-source-dirs:   test
    main-is:          Main.hs
λ P temp → λ git master* → 

Is this what you're going for @kindaro?

[kindaro]

Please understand: I use Cabal as a tool to get the job done. Not a guru developer, not an occasional user either. So:

• It is not mine to have opinions as to how command line flags should be interpreted. I cannot account for backwards compatibility, difficulties of implementation, and so on. I only ever used maybe a tenth of all the variety of flags Cabal provides overall in its plethora of sub-commands.
• I can see when a given flag does not make sense — it makes me spend time adding other flags and reading help messages. This waste of time can be justified, say by demands of backwards compatibility, but it cannot be said to be good. It might be a wontfix, but it is still a bug. It being documented, briefly or extensively, does not make it any less a fault.

If you show me the results of «hallway testing» where people figure out to invoke cabal init --simple --test-suite --minimal sooner than cabal init --tests when they want Cabal to initialize a test suite, I shall be surprized, but I shall accept my experience to be an exception. (Or re-do the study.) But so far I am your «hallway».

I do not mean this as saying that I am somehow special and entitled. If you have other means of measuring the goodness of user experience, please go for it. I am fine with any behaviour as long as it is clear and consistent. All I can say is that, when I type cabal init --tests and it gives me an executable, it is neither clear nor consistent. (Compare with cabal init --lib which initializes a library!)

Cabal already has a myriad of flags that work or do not work depending on their exact combination. Especially since cabal init is a convenience feature, it better be convenient!

[emilypi]

Ah no I think maybe I wasn't clear - I provided a minimal example as a proof of concept for what this workflow might look like if we added --test-suite as a flag. It doesn't exist currently, and I'm suggesting we can add it to handle your use case 😄

The --minimal flag is just to cut down on comments in the .cabal, which makes it easier to read, and --simple was used to shortcut the prompting process.

Just for background, what i mean when i say "initializing a test suite", I mean that the --tests flag is a boolean value which tells the script to generate a test suite if and only if a library is created. It is a confusing flag which doesn't quite make sense, since it's dependent upon the other package type flags, which tell cabal what kind of package to generate. The solution here would be to introduce a new package type called "Test Suite" into the list of available package types, and then to provide a flag which tells cabal to generate a package of that type. This is what --lib is doing, for example, and this is what I mean when i say "introduce a --test-suite flag", which would act like --lib and --exe.

I agree that what we have now is poor, ergonomically. I'm agreeing with you that this is could be improved!

[kindaro]

Is this what you're going for @kindaro?

… I provided a minimal example as a proof of concept for what this workflow might look like if we added --test-suite as a flag. It doesn't exist currently, and I'm suggesting we can add it to handle your use case 😄

Sure, the result looks right! If there is a way to put it into Cabal in such a way that it does not increase confusion, that would be good.

[emilypi]

There are one of two ways I can see this going:

1. Re-purpose --tests to indicate that you either want a standalone test suite if you don't specify a package type, or that you at least want a test suite when you specify a different package type. This means all other package types take precedent over test suites, which may be correct - we'd have to debate it. This may cause confusion, but I doubt many people would care about this small change in semantics. The problem with this is that you can't differentiate between workflows between you wanting only a test suite vs. adding a test suite to another package type, and you end up with implicit invariants littered throughout the code and in the way people need to pass flags.

2. Introduce a new --test-suite flag that sets the package type flag, in the same way as --lib and --exe. This is easiest code-wise, but it does introduce a confusing --tests vs --test-suite dichotomy which I could see tripping people up.

3. Repurpose --tests to be a flag which specifies a package type, like --test-suite above, and then have a flag like --initialize-tests which is just for telling the script that you want tests along with your other package types.

I think for me, reconsidering, it may just be best to repurpose --tests with the above behavior, since we're introducing a redesign anyway with the 3.6 release. As long as we document it, it sounds fine. And like you said - your hallways tests would point you to that flag being the correct one to generate test suites, so maybe we need to consider that!

@ptkato what do you think?

[ptkato]

I'm fine with whatever option, as long there's an actual use case (which it does have), however I'm inclined towards supporting the first option more than the others.

[emilypi]

Okay, so just so spec that option out, here's my problem with 1:

• To say --tests it so specify "a package type with a test suite", but not "a package whose package type is 'test suite'". Imagine this scenario: you pass cabal init --simple --tests. How do you indicate here that you want a test suite vs. a package type like --lib with tests?
• Having the --tests flag set creates an ambiguity of intent: should we enter in the prompt and ask what people want for a package type, or do we take their word for it that they want just tests? This is semantically different than something like --lib-with-tests or --libexe-with-tests.
• How do we mark the difference in intent without coming up with a set of flags like --lib-with-tests in order to describe that intent? The difference in intent is a dependent type: the behavior of --tests depends on the package type.

So, a solution would go like this:

1. If package type is defined, --tests corresponds with a boolean value that toggles the "initialize test suite" behavior we already have.
2. If package type is not defined, --tests corresponds with the package type TestSuite.

And we'd modify the following workflows:

• Simple: iff --tests is supplied, we consider it to be a package type only, and do not prompt in that case. Here, the tradeoff is we now always prompt for tests on --lib and --libexe unless --tests is passed explicitly along with the package type. Before we could pass --tests and save ourselves a prompt in those workflows.
• Interactive: --tests corresponds with a test suite package type by default, and we now always prompt for tests on --lib and --libexe by default. Before, we could pass --tests and save ourselves a prompt.
• Non-interactive: The heuristics will need to cover the inferrance of test-suite only projects. I leave this one to you.

[ptkato]

• Non-interactive: The heuristics will need to cover the inferrance of test-suite only projects. I leave this one to you.

That boils down to assuming defaults and checking the working directory for existing files. I believe with minimal alterations, what already is in place could work great.

[tonyday567]

I have to separate tests out all the time as users complain about the extra dependencies that it adds to the package.

How about --tests-only?

You won't find it by intuition, but coming across it in the docs, the name would help clarify the usage provisos involved, and listed near --lib, would help distinguish this different use-case.