doctest's argument format does not match GHC's

[ezyang]

I was looking into doctest integration for Cabal in haskell/cabal#2327 and I was hoping that doctest accepted exactly the same set of arguments as GHC, so that I could reuse Cabal's logic for generating GHC flags for doctest. However, this does not seem to be the case.

All tests against doctest HEAD compiled against GHC 7.10.

The first problem I noticed was that -package-db foo was not supported.

ezyang@sabre:~/Dev/labs/doctest$ /srv/code/doctest/dist-newstyle/build/x86_64-linux/ghc-7.10.3/doctest-0.11.2/c/doctest/build/doctest/doctest -package-db /home/ezyang/.cabal/store/ghc-7.10.3/package.db
doctest: <no location info>: missing argument for flag: -package-db
Try `doctest --help' for more information.
ezyang@sabre:~/Dev/labs/doctest$ /srv/code/doctest/dist-newstyle/build/x86_64-linux/ghc-7.10.3/doctest-0.11.2/c/doctest/build/doctest/doctest -package-db=/home/ezyang/.cabal/store/ghc-7.10.3/package.db
Examples: 0  Tried: 0  Errors: 0  Failures: 0

The second problem is doctest behaves very strangely when -outputdir is set:

ezyang@sabre:~/Dev/labs/doctest$ doctest -outputdir /srv/code/labs/doctest/dist-newstyle/build/x86_64-linux/ghc-7.10.3/a-1.0/build A
*** Exception: /srv/code/labs/doctest/dist-newstyle/build/x86_64-linux/ghc-7.10.3/a-1.0/build/autogen/Paths_a.hs/A.o: getFileStatus: inappropriate type (Not a directory)
Examples: 0  Tried: 0  Errors: 0  Failures: 0

Is there a good reason for these divergences? At least from Cabal's perspective, it would be a lot easier to integrate if the semantics matched up exactly (perhaps this should be a separate mode?)

[sol]

I can't really help with this in terms of coding, but if changes to Doctest are needed, I'll make sure to review them in a timely manner and release a new version to Hackage.

Here are some notes:

• Doctest runs in two phases

  1. Extract Haddock comments, using the GHC API
  2. Verify examples by shelling out to ghc --interactive

• Extracting uses handleStaticFlags and handleDynamicFlags, so all arguments recognized by these functions should be supported (see withGhc in src/GhcUtil.hs)

• Verifying uses ghc (the executable), so all arguments recognized by that executable should work (see src/Interpreter.hs and ghci-wrapper/src/Language/Haskell/GhciWrapper.hs)

• doctest (the executable) which is implemented in terms of Run.doctest, applies all kinds of heuristics and magic that may interfere with cabal

• There is a version of Run.doctest that does not apply any magic: Run.doctest_. It may give more deterministic results. We may want to add a --no-magic-flag that exposes the behavior of Run.doctest_.

• Run.doctest uses various ways to discover and use sandboxes and package DBs

• Not sure what the issues is with -package-db, patches welcome

• Regarding -outputdir, there are two things that may be related (both when we extract Haddock comments)

  1. We modify the output directory, see Extract.withTempOutputDir
  2. We run in Haddock mode, see GhcUtils.setHaddockMode

I hope this will be of some help.

[angerman]

Just wanted to leave this here: the -package-db issue is essentially running into expandDirs, which expands each directory on the commandline into all it's contained hs files. If it wasn't a valid directory, we'd just return it as is. However if you provide -package-db <dir> it clearly is a valid directory. We then proceed by trying to find all haskell files in that directory, and substitute <dir> for the result. In the case of a package database, we won't find any haskell files, however a valid dir. This case then results in an empty substitution, which leaves -package-db without an argument.

Thus, if we want to allow arbitrary arguments passed to doctest to be passed verbatim down to say ghc, we need to figure out a way to prevent directory expansion in those cases.

I believe a --no-magic should simply sidestep getPacakgeDBArgs and expandDirs. This however would mean, that with --no-magic one would have to expand all directories by hand.

[ezyang]

OK. Is it an acceptable outcome if Cabal's native doctest support only supports doctests that understand --no-magic? If not, we'll still have to work around this.

[angerman]

I believe having Cabal's native doctest only support doctest 0.12 and up, (or on what ever version we decide after the PR).

I just want to have this work with the patched doctest in Cabal, before submitting a final PR.

[sol]

doctest-0.11.3 adds support for --no-magic.

I believe this solves this issue, closing.