Scan include directories for odoc files eagerly

[jonludlam]

Goes some way to addressing #148.

This is an RFC, particularly the error handling. As is, this still results in a nasty backtrace spewed to the terminal so I don't think it should be merged yet. Before I just wrapped the invocations of Env.create in an exception handler I thought I'd better ask whether there was a better plan in mind for this sort of non-recoverable one?

[aantron]

We do eventually plan to have more disciplined error handling at the top level, but so far we've been working "bottom-up" by introducing proper error handling in the parser first, and gradually propagating new error handling up to the command-line interface level. This process has reached only the parser and parts of the loader so far, however.

I'd say to do whatever you think is easiest here :) It could be the right solution for the long term, but even if not, that's okay, because then we will still address it later :)

[jonludlam]

OK. I'll stick with raising the exception from Model.Errors and catch it right at the top.

I'll need to stick in some more error handling around the eager loading too, since having just tested it on Coq I find it's failing with End_of_file, because dune is running a whole bunch of odoc instances in parallel all writing out odoc files and hence we can't guarantee to be able to read them.

[lpw25]

because dune is running a whole bunch of odoc instances in parallel all writing out odoc files and hence we can't guarantee to be able to read them.

You should be able to use the imports list to restrict the eager scanning to just those files we will actually use. This should remove issues with parallelism and also prevent non-reproducible documentation. Essentially you should only include files whose name matches those in the import list.

There could still be problems if there are two files with the same name, and dune thinks only one of them is a dependency. So some protection for End_of_file should still be there, but the problem should then be extremely rare and also a symptom of an actual problem with how the build system is handling dependencies.

[jonludlam]

Ah, this sounds much better. I'll have a look.

[jonludlam]

This is the wrong solution.

[aantron]

@jonludlam Could you link to something, or state in one brief sentence what the core of the reasoning is?

[jonludlam]

I was planning on doing so in the follow-up PR, which I have cooking. I just didn't want to waste anyones time reading this when I'm intending to do something different

[aantron]

Fair enough, thanks.