Updating/Publishing the docs/ directory

[ryan-williams]

I only realized that the docs/ directory exists recently, when refactoring README.md (#763, #764).

@fnothaft implied on #764 that the docs there get distributed via released artifacts, but I don't see any in adam-distribution_2.10-0.17.0-bin.tar.gz.

It seems like there was a lot of somewhat-duplicated markdown living there, vs. what was in README; now the README is very different.

Are builds of docs/ available on the internet anywhere? How should we keep them updated / consistent with README.md? Is it up to anyone making changes to update the docs in both places?

Should we move towards setting up e.g. readthedocs.org builds of the docs for each version? Should the README point at those? I generally like that approach, though some way to also publish docs for random non-release SHAs/nightly-snapshots or something would also be good and has been lacking in other projects who've done something like this, e.g. Spark.

[massie]

The docs are published with the releases. I had hoped that by using simple markdown, the barrier to updating them would be low. Unfortunately, we haven't been updating them once created, e.g. git log -- docs.

I'm open to any suggestions about how to do documentation better. I haven't used readthedocs.org so I can't comment from experience. I hope what we choose needs to create standalone docs (e.g. PDF, html, etc) and not be dependent on services.

[ryan-williams]

Ah, thx @massie, I see the PDF and HTML downloads on the releases/ page now. Thanks for the additional context as well; I don't have any great ideas at the moment but I'll leave this here until inspiration strikes someone.

[fnothaft]

We should push this to our github pages.

[heuermh]

Fixed by #1461.