-
Notifications
You must be signed in to change notification settings - Fork 704
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
doctest integration for Cabal #2327
Comments
What I can see, than the |
CC'ing @sol here. We need to know a little more about how doctest's command line runner works before we can work something out here. I think the main question is whether or not doctest should be implemented like a virtual component (as @phadej suggests), or something more like Haddock, which is not really a component but just another mode you can run your project in. Based on my reading of https://github.com/sol/doctest/blob/master/src/Run.hs it seems that doctests accepts exactly the same arguments that GHC accepts (have you considered making it a frontend plugin? :) https://downloads.haskell.org/~ghc/master/users-guide/extending_ghc.html#frontend-plugins ) with a little bit of extra smarts to handle options that people commonly forget to add. My key question: does doctest recompile the module under test, or does it assume that the module has already been compiled and installed? Based on the README, it seems it must be the former. In that case, it seems like a virtual component ala @phadej is highly inappropriate, because you're going to need to also pass in all the flags that you would have needed to build the package in the first place. Here's my question: why can't doctest be implemented as a wrapper around GHC, passed to Cabal via |
I was thinking more of test-suite with type As far as I understand, doctest drives GHCi, so library doesn't need to be compiled, except that e.g. with |
@phadej Even if the library is interpreted, we still need to pass a sufficient set of |
not exactly, the superset I refine my proposal. Yes it makes more sense to have it as a virtual command:
module MyPkg where
-- $setup
-- import MyFancyExtrasCombinators
-- | ...
--
-- >>> fancyHelperFindFixpoint fun
-- 42
fun :: Int -> Int
fun ...
I stress, this is important. Cabal should be able to manage the Related: actually Cabal should be able to manage
|
Sure, that's fair. OK, so let's talk about how we actually get our hands on a doctest executable. Cabal gets away with not having to manage haddock because it's usually distributed with GHC, and when we attempt to detect ghc/ghc-pkg/etc, we also try to detect the correct Haddock for version of GHC. Haddock is tightly bound to the version of GHC in any case. Of course, doctest isn't shipping with GHC. So what can we do? If our clients are using new-build, we're actually in pretty good shape in HEAD: all we need to do is set things up so we get an executable dependency on doctest when doctests are enabled, and new-build will go off and build doctest (against the ghc API which will be the same version as what we are compiling) and then add it to the PATH. We'll build multiple copies of doctest per version of GHC because the Nix hash will vary depending on which one we use. I'm still not sure if we want to go the doctest stanza route; in principle, doctest can be applied to any stanza, not just library (libraries! with convenience libraries...) What if we had a magic flag that toggled doctest? |
Yes, I was thinking that with I think stanza is clearer. Library stanza is about how to build library, not how to test it. |
Found the package where |
@ezyang Doctest does not compile anything, it uses the GHC API to extract haddock comments (parsing only) and GHCi to check the code examples. It accepts the same command-line options as I think it should be a subcommand not a stanza for the same reasons haddock is a subcommand and not a stanza. Contrary to popular believe, Doctest is not a tool for testing code, it's a tool for testing documentation. You want to run Doctest on exactly the same code as Haddock, to make sure that the code examples in your documentation do actually work. For that reason I think it is natural to treat it the same way as Haddock. The only difference I see is that Doctest may need QuickCheck in addition to the library dependencies. So in general, Cabal would need to make sure that both the
For the rather uncommon case that somebody needs more additional dependencies for doctest, it may make sense to allow to specify additional dependencies somehow. But even that is not crucial from my perspective, as it is not very common. Regarding having multiple doctest stanzas for one cabal project, I don't see how this is crucial. Specifically, using doctest to test documentation for executables is as useful (or useless for that matter) as running Haddock for executables. If we want to support this, we could add the same command-line flags we already have for As a side note: Peoples who use Doctest for testing code (as opposed to documentation) do something fundamentally wrong. I don't think it's crucial that Cabal supports this. Other tools (e.g. Hspec) ar far superior when it comes to testing code. |
Yes, but if we are invoking GHCi, we have to deal with essentially the same problems that we face when compiling code: what modules should we have in scope when someone asks for an import. The reason why doctests are often said to be "fragile" is because the basic usage, I guess this also means that
I think the reason we are thinking about how to handle extra dependencies is not because we expect people to want this in general, but because we feel a bit uncomfortable when something like QuickCheck is hardcoded; in general we want to let people to use what they want to use, and not force them to something in particular. I think it would be reasonable, as a first cut, to add a |
Both work. You have to run Regarding non-exported entities, it's exactly as with But of course yes, you still can't use non-exposed things from third-party libraries.
Not saying you should do something different here, but just as a clarification: QuickCheck is actually hardcoded in Doctest itself. It's used when ever you use something like
in your Haddock comments. Thats it from my side. Due to medical reasons I'll leave the rest to you guys. Fare well. |
To clarify. Yes,
|
What about generating an output file from normal GHC compilation that just has the doc-comments, in some appropriate format that a doctest tool could process, rather than processing the original source. Subject to a flag being passed to GHC. |
@alanz the problem is not reading the source, but providing the proper ghc flags to |
Ok, so doctest is still a mess to get working in 2021. |
@hasufell: anything in particular? I've recently made my doctests work, so I thought it works fine. The trick is to do |
@Mikolaj It started with trying to run it plain, which didn't work due to missing modules. Modules used in setup stage had to be added to the module itself, although that doesn't use it. Then you have to create a I don't consider this good UX. |
There is also docspec |
That looks similar in complexity. I have to provide everything:
|
Well, that's just the "do everything manually because there's no cabal" example. In most cases, all that info will be extracted from plan.json |
Oh yes, I forgot I had to write this monstrosity: https://github.com/LambdaHack/LambdaHack/blob/master/test/doctest-driver.hs |
I think we have improved doctest support On The Roadmap for future cabal-install. We need to nail down a good spec we're comfortable with for this, which will take some careful design work (and require someone both familiar with existing doctest machinery and cabal internals). We should setup a new thread for discussion, but until then: One strawman idea -- what if instead of a doctest stanza as such, we extended the test stanza with a code generation step that invoked an executable that took the code directory itself as an argument? So we have a test stanza, with a build-tools depends on |
You are taking the most complicated example possible, My bad though, I should written the simple example first. Which is ridiculously simple for normal packages:
|
Oh, I did. It's right there
|
@Mikolaj something like that, but instead of needing to pass all the stuff in a directive in a stub driver file, one would simply give the executable name directly as clause in the test stanza. So something like
and nothing else. |
So, I get 3 different results with 3 tools:
|
With following straight forward diff:
the cabal-docspec runs plenty:
The extra cabal field is documented: https://github.com/phadej/cabal-extras/blob/master/cabal-docspec/MANUAL.md#cabal-fields as well as reasoning for not importing modules https://github.com/phadej/cabal-extras/blob/master/cabal-docspec/MANUAL.md#q-why-cabal-docspec-doesnt-import-modules-automatically I think that three errors left are also fixed easily with adding more imports to Yes, it's not "out of the box - just works", but it works with acceptable amount of extra work, I doubt Obviously, I'd like approach like |
@phadej I tried fixing all the imports and now
Branch with the code: https://github.com/hasufell/abstract-filepath/tree/doctest |
Yes, the extensions in files are not enabled either. https://github.com/phadej/cabal-extras/blob/master/cabal-docspec/MANUAL.md#using-ghc-extensions |
I don't understand, what extension am I supposed to pass? Even |
Looks very much like you need |
@phadej that finally worked. But can you feel my pain? Now I have to add special sections into my cabal file and still need to pass flags on the command line. Since cabal-docspec isn't on hackage I also can't easily add it to |
You can specify the command line parameters inside the The I also don't yet see any conceptual challenges why |
So let's reopen the ticket i guess? |
I'd open a ticket only when there is some concrete plan, there are discussions for getting consensus (and actually RFCs). |
Another point is that there's For example, I considered making a variant of That's an argument of not integrating anything too closely with This reminds me that I should make |
Have to note that only stack and cabal-docspec work.
At this point, most users would have given up. So my impression is that it's a problem with |
I would advocate not reopening the ticket. We have a separate ticket for |
(also @hasufell it looks like to get |
That doesn't fix it. |
New summary by @ezyang.
The goal of this task is to add a new command
Setup doctest
, which runs the doctest command on all of the modules of each component in a package, in much the same waySetup haddock
runshaddock
on all the modules of each component in a package.There are a number of steps to implementing this:
You have to add the boilerplate for setting doctest up as a new command. Grepping for occurrences of haddock should give you an idea for what you have to edit; the key files are
Cabal/Distribution/Simple/Setup.hs
, adding a case todefaultMainHelper
inCabal/Distribution/Simple.hs
, and a new file to actually contain the logic. I'd recommend against putting too many flags intoDoctestFlags
when starting off.You need to add
doctest
to the program database, so that you can invoke it (and end-users can customize which binary to use, extra options they want to pass, etc.) The relevant logic is in Cabal/Distribution/Simple/Program/Builtin.hs You should create create something like doctestProgram.The overall logic of the driver should be something like this:
Between component, lbi and clbi, you will have all the information necessary to invoke doctest. You can look at https://github.com/haskell/cabal/blob/master/Cabal/Distribution/Simple/Haddock.hs for an example of all of the parts being put together.
Original description below:
@sol suggested this would make more sense here than the doctest repository.
A
cabal doctest
or some manner of Cabal auto-magic is needed.Should do discovery and running of doctests automatically.
It's extremely hard for people to get doctests working with a Cabal project. It took be 3 days, 10-12 hours apiece just to get doctests working for Bloodhound. Now I realize I'm a fucking moron, but I was also getting help in IRC from people like @ekmett who uses doctests for
lens
and he had no idea what was causing things to break either.The things that make doctests hard to use fall into three categories:
This notion fixes 1 and 2, but not 3. However, if users have the reasonable belief that 1 and 2 are okay and should be working, they'll spend less time trying to diagnose the wrong things when they've messed up number 3.
I filed this issue long after having gotten Bloodhound's doctest integration working because I ran into somebody I know on IRC struggling for multiple days with this again.
I'm tired of having to tell people to copy-paste what I have in Bloodhound to make it work and people hate having to cargo-cult this anyway. Doctests have caused problems for the NICTA course as well.
The text was updated successfully, but these errors were encountered: