manual documentation uploads are not documented

[RossPaterson]

I understand that maintainers can upload docs if the builder can't build them, via

http://beta.hackage.haskell.org/api#documentation-core

but several points aren't documented:

• The flags that should be given to cabal to get the links right.
• The directory that needs to be tarred.
• How the tar file should be uploaded.

[dcoutts]

Some info, since I was working on a related bit of the code. Just need to collect it and turn it into a user-oriented page.

The flags that should be given to cabal to get the links right.

One can find this from the build client code. It invokes cabal-install with the appropriate args.

Though perhaps better to explain it as a cabal haddock invocation.

The directory that needs to be tarred.

Tarball content needs to be like:

zlib-0.5.3.1-docs/
zlib-0.5.3.1-docs/src/
zlib-0.5.3.1-docs/src/Codec-Compression-Zlib.html

... etc

How the tar file should be uploaded.

Like so:

curl -X PUT \
 -H "Content-Type: application/x-tar" \
 -H "Content-Encoding: gzip" \
 http://user:passwd@beta.hackage.haskell.org/package/zlib-0.5.3.1/docs \
 --data-binary @zlib-0.5.3.1-docs.tar.gz

[jeremyjh]

Hi, when I try to do this using your command @dcoutts I get an error message:

 Invalid documentation tarball: Archive is in the GNU tar format

I'm creating the tarball with:

tar -czf my_dir

[gracenotes]

Most tar programs read/write in the portable ustar format. "--format=ustar" should do the trick here.

[jeremyjh]

Thanks @gracenotes - that seems to work - at least I do not get any error message. I don't see my docs showing up either though.

[gracenotes]

Okay, I took a look at /package/higher-leveldb-0.1.0.0/docs.tar. In the tarball, the higher-leveldb-0.1.0.0-docs/src/ directory is meant for Haddock coloured source files; the top-level directory (right under higher-leveldb-0.1.0.0-docs/) is meant for the documentation itself. You'll also need ocean.css etc. in this top-level directory; some of the generation flags can be seen here: https://github.com/haskell/hackage-server/blob/2657af5/BuildClient.hs#L513

hackage-build itself should probably be easier to use for one-off documentation uploads.

[jeremyjh]

Great, that did it thanks!

[cartazio]

bump :)

[stbuehler]

I wrote a shell script to generate and upload docs for a package: https://gist.github.com/stbuehler/7068764

[cartazio]

this information needs to be more discoverable and easy to use until the broken docs problem is resolved

[Gabriella439]

I can confirm that stbuehler's script works. I was a bit suspicious at first because I didn't see any index.html file in the generated documentation, but I went ahead and tried it and it worked.

[dcoutts]

Note that we're now running a new doc builder client which should resolve some of these issues. For the things not resolved, it does at least generate build logs (see /package/foo-1.0/reports/$n/log) so we can see what's wrong. Mostly I expect this will now be missing C libs which we'll need to add to the builder machine.

[mruegenberg]

I just tried the process mentioned here. I managed to upload documentation, but it is actually not unpacked correctly.

The package is hydra-hs, the doc link (correctly) goes here, but the actual documentation is in a subdirectory here.
I then tried to create and upload the archive from the subdirectory that, when unpacked in the same way, would result in the correct paths, but that gave me an "Invalid documentation tarball" message.

[dcoutts]

@mruegenberg I think you didn't quite follow those directions (looks like you grabbed the docs from the installed location rather than from the inplace dist/ dir), but also those directions are also incomplete because they don't include the flag to get the correct links, so all the links to types from other packages would be broken.

[mruegenberg]

It worked now. I ran the script instead of following the steps manually.

I also had to replace the tar command by tar -c -v -z -H --format=ustar -f "${DDIR}.tar.gz" "${DDIR}". (This is probably because OS X uses bsdtar instead of GNU tar.)

[Fuuzetsu]

@dcoutts , what is the exact flag to get the cross-package linking?

[dcoutts]

@Fuuzetsu For the definitive answer, see the code for the doc builder:

https://github.com/haskell/hackage-server/blob/master/BuildClient.hs#L518

[liyang]

I did a release of true-name, but forgot to update some documentation, which I fixed in a post-release commit. Not wanting to make another release just to fix the documentation on Hackage, I tried to re-upload the docs manually.

The weird thing is: I'm still seeing the old version of the module documentation, but clicking on the Source link shows the updated comments. I've checked the …-docs.tar.gz I'm uploading and it definitely contains the new Unsafe-TrueName.html. I even tried deleting the existing docs via the maintenance pages and uploading my version before it has a chance to rebuild, but to no avail.

Can anyone help me understand what's going on?

[bitemyapp]

@liyang I'd just use http://hackage.haskell.org/package/neil to upload the docs.

Documented here: http://neilmitchell.blogspot.com/2014/10/fixing-haddock-docs-on-hackage.html

[liyang]

@bitemyapp Didn't know about @ndmitchell's eponymous tool… I was using my own Bash script to do the same.

Have just tried neil, and it seems to have done exactly the same thing—still the old module page, but updated (colourised) source.

I'm wondering if somewhere the old version is being cached somehow?

[bitemyapp]

@liyang if it was getting cached, I think that would be from the CDN, but it's extremely likely I'm wrong about that. The actual hackage page gets rendered directly from current docs tarball IIRC.

[sapek]

Is it still necessary to manually build/upload documentation? I've uploaded a new package bond and it appears that the build never ran. Is this a problem on the hackage side or something I'm doing wrong (it's my first package so quite likely)?

[lambda-fairy]

@sapek Have you already uploaded documentation for that package? The builder won't try to build a package if it already has docs.

[sapek]

Yes, today in the afternoon I gave up and uploaded docs manually using neil to mentioned above. But I waited almost 24 hours after publishing.

[cartazio]

There was some Haskell infrastructure related migrations a week or so ago,
has anyone checked on the health of the build queue?
Likewise, does the test upload facility interact with the build queue?

On Thursday, December 10, 2015, Adam Sapek notifications@github.com wrote:

Yes, today in the afternoon I have given up and uploaded docs manually
using neil to mentioned above. But I waited almost 24 hours after
publishing.

—
Reply to this email directly or view it on GitHub
#56 (comment)
.

[gbaz]

#553 should resolve this..

[gbaz]

since that's merged, closing this out.