Make ddox the default doc for dlang.org menu

[wilzbach]

Starting with step 1 as outlined here

1. /library/ is promoted as the primary Phobos documentation in the site navigation.
2. /phobos/ is removed from search indexing.
3. /phobos/ is removed from site navigation.
4. /phobos/ is removed.

CC @s-ludwig

However I would love to love see the Disqus problem sorted out before. So let me summarize the three main options proposed so far:

Option 1: self-hosted commenting

I share a similar opinion as @MartinNowak:

I'd want to disable or replace discourse before we make it our official documentation. We could easily self-host some commenting functionality if deemed necessary, but adding an unmaintained communication channel isn't the best idea IMO.

Option 2: use DFeed

(from @CyberShadow)

I think it wouldn't be too hard to embed DFeed as a comment system. What do you think? Then anyone can subscribe to new comments through the usual means. The same solution can be applied to the blog.
I understand that the current bottleneck is someone adding new groups to the NNTP server. Can that be resolved?

Option 3: remove Disqus

Yeah, I'm not likely to ever use disqus but if it went through the same n.g./mailing list interface at least I'd consider talking at times (though I personally think comments in documentation should typically be used to just go back and improve the documentation rather than making readers actually wade through the out-of-date and repetitive comment thread...)

Note that removing Disqus has been attempted this summer, which was reverted as @MartinNowak wasn't available at this time.

[andralex]

@wilzbach are examples runnable just the same in both? any dead links? anything to worry about? In the past I ran quickly into some issue.

[andralex]

e.g. https://dlang.org/library-prerelease/ has a few oddities:

• Table has a wide first column due to large module names such as std.experimental.allocator.building_blocks.affix_allocator. I think we could insert a soft break before each dot in these names? What's HTML for soft break - i.e. like ­ but without outputting a hyphen?
• Those std.experimental.allocator.building_blocks.* don't have any description
• Description for some modules comes in heading font, e.g. "Assembling Your Own Allocator"
• The table "Access to plattform libraries is supported by specific D header files" is odd because it has a bunch under C99 and nothing everywhere else
• The table "Deprecated D header files." has only one "C" in it?

[andralex]

I also don't see any runnable test. Do we need to do some deployment? @CyberShadow is ddox part of the auto-deployment engine?

[wilzbach]

I also don't see any runnable test. Do we need to do some deployment? @CyberShadow is ddox part of the auto-deployment engine?

I haven't found time to work on this yet. It's a different setup.

are examples runnable just the same in both?

Well there are different two setups (since two years) :/

any dead links? anything to worry about?

Ddoc will stay for quite a while.

In the past I ran quickly into some issue.

Yep I just started the discussion to figure out the last blocking points...

[wilzbach]

@wilzbach are examples runnable just the same in both?

They should be soon: #1532

[CyberShadow]

@CyberShadow is ddox part of the auto-deployment engine?

Yes. Runnable tests are just not implemented for DDox right now.

[wilzbach]

So what is the decision here?

• Support to run the tests with the ddox build has been added with Allow to run ddox unittest examples #1532

• Do I understand it correctly that once the last glitches in ddox that Andrei has mentioned have been fixed, we can start the switch over?

• What is our consensus regarding Disqus / comment system? (@CyberShadow I think your opinion has a lot of weight here)

[andralex]

Copying from my older comments: "...e.g. https://dlang.org/library-prerelease/ has a few oddities: ..."

• Table has a wide first column due to large module names such as std.experimental.allocator.building_blocks.affix_allocator. I think we could insert a soft break before each dot in these names? What's HTML for soft break - i.e. like ­ but without outputting a hyphen?

Not fixed.

• Those std.experimental.allocator.building_blocks.* don't have any description

Not fixed.

• Description for some modules comes in heading font, e.g. "Assembling Your Own Allocator"

Not fixed.

• The table "Access to plattform libraries is supported by specific D header files" is odd because it has a bunch under C99 and nothing everywhere else

Not fixed.

• The table "Deprecated D header files." has only one "C" in it?

Not fixed.

[andralex]

I'd say let disqus be. Doesn't hurt and the dearth of comments is not a reflection on disqus as much as a lack of critical mass in viewership.

[adamdruppe]

On Wed, Feb 22, 2017 at 09:34:44AM -0800, Andrei Alexandrescu wrote: * Those `std.experimental.allocator.building_blocks.*` don't have any description

That's not ddox's fault, the source for those submodules is missing documentation on the module declaration. We should do a Phobos PR to add some description text to those modules.

* Description for some modules comes in heading font, e.g. "Assembling Your Own Allocator"

dpldocs.info stripped the header tags there, but it is arguably also the source module's fault - it should have a brief summary before the headers and details.

* The table "Access to plattform libraries is supported by specific D header files" is odd because it has a bunch under C99 and nothing everywhere else

Again, those aren't documented in the source. We should do a druntime PR to add doc comments there. Or something. I'm not sure it makes sense to do too much ddoc of C declarations, but it is a FAQ "why isn't this documented on dlang.org?" so we should do SOMETHING. So bottom line, let's do PRs on phobos and druntime to document these, and ddox should automatically pick up the benefit - and ddoc might too.

[andralex]

We should do a Phobos PR to add some description text to those modules.

Correct, those should ideally predate switching to ddox.

[JackStouffer]

I have to say, the dlang.org/library/ docs are a major regression in terms of scanability and readability. The default should not be changed until the following issues are fixed.

• The index page for /library/ sucks compared to /phobos/. Way less info and less organized.
• The package pages std.algorithm, std.range, and std.experimental.allocator are not linked
  anywhere but the side-bar under "Package Members". These need to be the default when clicking on std.algorithm on the side bar and linked on the index page. These pages give an essential overview of the whole module to new comers.
• Function/template name column is way too narrow, everything is squished.
• The way the new docs split the various module items by function/enum/structs/etc is a step backwards in readability, because now things like aliases for specific functions, e.g. std.algorithm.comparison.AllocateGC are no longer shown next to the function it's used for. There are many other examples of this.

Now, we get to the non-objective things. Feel free to disagree here, but I hope to provide a compelling case.

• The /library/ docs lack in information density. Compare https://dlang.org/phobos/std_algorithm_comparison.html#.among
  to https://dlang.org/library/std/algorithm/comparison/among.html. The new docs take up twice the space of the old ones. One reason for this is the absurd amount of space given to the function signatures. Also, why does /library/ repeat the docs for the function and the template of among?
• Things get repeated when they weren't before. Before, the template signiture for among was listed with the function overload. Now, in the module overview it's repeated in the templates section, but they both link to the same place. Does this add any extra value?
• I see very little value in giving every function its own page. Perhaps the argument could be made for structs and classes in order to encapsulate complex types. But I often go back and scan the entire std.range page because I know there's a function to do something, I just forgot the name of it. In this case, I would have to guess from the little information the auto generated module page gives, or click on each one individually.

I'll comment on more things as I think of them

[adamdruppe]

I see very little value in giving every function its own page.

It helps so, so, so much on search results, and having plenty of space means it is possible to format the information better (ddox does meh on it but still). Consider that so many search engine hits go to ddox already, it just highlights how much easier they are to link to.

[andralex]

This goes on to confirm what I like to joke about: whatever we do, someone will be against it. Page-per-artifact has been vociferously asked for by many folks. Growing pains! My approach in this: we can afford to offer both with ease, so we should. (We're not alone; see e.g. https://www.gnu.org/software/make/manual/ which offers several ways to view the same documentation.)

[JackStouffer]

As long as /phobos/ is still available, then I'm fine with the ddox version. However, I still think it's not good enough for the default.

[CyberShadow]

https://dlang.org/library/std/range/drop.html shows raw HTML

[s-ludwig]

The source looks off: https://github.com/dlang/phobos/blob/v2.074.1/std/range/package.d#L2880
Embedding DDOC within backticks is "strange" and the opening back tick has no closing match.

[CyberShadow]

Thanks. dlang/phobos#5494

[CyberShadow]

@s-ludwig Would it be possible to make DDox throw a hard error upon encountering such invalid syntax?

[s-ludwig]

@CyberShadow I'll remember to add that (dlang/ddox#154). I'd suggest to make this an opt-in switch, though, because it could break existing projects otherwise.

[CyberShadow]

Cheers. I guess we could grep the output in the makefile as well.

[CyberShadow]

I don't see any relevant warnings in dpl-docs output when running make -f posix.mak apidocs-prerelease, just some "Error parsing type" which seem unrelated. Perhaps there is no code to detect/warn on such malformed DDoc syntax yet?

[s-ludwig]

Indeed.. it currently silently uses the end of the paragraph to mark the end of the back tick code in this case.

[adamdruppe]

On Sun, Jun 18, 2017 at 01:13:37AM -0700, Sönke Ludwig wrote: Embedding DDOC within backticks is "strange" and the opening back tick has no closing match.

If it has no closing match on the line, the backtick is supposed to be output literally. The backtick code thing is not supposed to be able to span lines. I did that quite deliberately in the implementation for dmd because 1) linker output often does something like identifier `foo' and I wanted that to be copy/pastable and 2) github ignores it with no match too. See the explicit comment here: https://github.com/dlang/dmd/blob/master/src/ddmd/doc.d#L2166

[s-ludwig]

@adamdruppe: Fixed for DDOX 0.16.1: dlang/ddox#155

[adamdruppe]

On Sun, Jun 18, 2017 at 09:16:21AM -0700, Sönke Ludwig wrote: @adamdruppe: Fixed for DDOX 0.16.1: dlang/ddox#155

Excellent!

[CyberShadow]

Some broken documentation (example) here:

https://dlang.org/library/std/algorithm/searching/find_skip.html

Looks OK on DDoc:

https://dlang.org/phobos/std_algorithm_searching.html#findSkip

[s-ludwig]

This is an effect of the necessary workaround to extract the unit test code for the example. I've opened a ticket (dlang/ddox#156) and this should be solvable by employing dparse to tokenize the source code instead of blindly searching for the last closing brace.

[CyberShadow]

The prototype here is really broken:

http://dlang.org/library-prerelease/std/file/dir_entries.html

Screenshot:

• The first line seems to be justified, making the sole two terms appear on the opposite sides of the box.
• The second line has no whitespace or word wrap points, making it go out of bounds.
• There is something wrong with the emitted data, the prototype itself doesn't even look like D code.

On Chrome the justification will not be visible as we do not enable it for that browser (because it does not support automatic hyphenation).

[andralex]

Only regular flowing text should be justified, nothing else (not even tables) btw.

[CyberShadow]

Yeah, I guess problems with justification slip through the cracks because it's disabled for Chrome (because of hyphenation), and many contributors test with it exclusively.

[CyberShadow]

https://issues.dlang.org/show_bug.cgi?id=17624

Because DDox simply repeats the documentation for ditto'd symbols, but also sorts the properties alphabetically, in some cases it can apparently cause the impression that some fields were mis-documented by a copy/paste someone forgot to edit.

[s-ludwig]

#1526 (comment)

That must be one of the parse errors from above. I'll try to remember the text justification issue. BTW, is there any known reason why this is not simply restricted to <p>? Same for auto-hyphenation of course.

#1526 (comment)

Any suggestion how we should handle this? To me it seems to be a rather unclean use of ditto. There could be a logic that accumulates all ditto group members into the same table cell, but it kind of breaks down on the detailed member pages. We could list all member on each page, or could emit just one page, so that "line" in this case would be documented on the "file.html" page. Both are not really nice.

IMO, this one should be fixed by simply splitting up the sentence into two doc comments.

[CyberShadow]

Any suggestion how we should handle this?

How about listing members in declaration order, instead of alphabetical order? Then you can use rowspan for ditto.

[andralex]

How about listing members in declaration order, instead of alphabetical order?

That's a good idea that shifts the responsibility and freedom reasonably.

[s-ludwig]

It doesn't make that much sense to use declaration order once the declarations are grouped by kind, IMO. But I suggested to group declarations within the table already. The question is more how to proceed for the per-declaration page.

[CyberShadow]

The question is more how to proceed for the per-declaration page.

I guess the obvious answer is to somehow indicate that the documentation applies to other members too (listing them in a dimmed font or "(also: foo, bar)" OSLT). Another approach would be to decide to use ditto only for overloads, and fix other uses.

once the declarations are grouped by kind

BTW, I'm not really convinced that such grouping is useful. I know MSDN and many documentation generators do it, but... properties and paren-less function call syntax in general blur the line between fields and methods in D somewhat. Declaration order is likely to have things grouped together more consistently.

[CyberShadow]

There are commas missing in the description of the globMatch syntax:

https://dlang.org/phobos/std_path.html#globMatch
https://dlang.org/library/std/path/glob_match.html

In DDox, for the "Matches either of the specified strings." table row, the first column reads "{string1string2…}" instead of "{string1,string2,…}".

[veelo]

Disqus comments are identical for every function.

https://dlang.org/library/std/typecons/proxy.html
https://dlang.org/library/std/typecons/typedef_type.html
etc.

I would strongly argue for comments to be removed completely from documentation pages. I think comments have no place in documentation; we have the learn forum for asking questions (with excellent and timely response I might add) and the issue tracker for issues. I have yet to see a comment that is worth my time reading and in general comments just pollute the docs.

If a quick way of asking questions is thought to be needed then the following alternative can be considered: buttons at the bottom of the page for

• doing a search on the learn forum with an automated search string appropriate for the current page
• posting a question on the learn forum.

[wilzbach]

I would strongly argue for comments to be removed completely from documentation pages.

I absolutely agree - I even tried this one year ago:

#1363

However, it got reverted:

#1443

[wilzbach]

I absolutely agree - I even tried this one year ago:

Let's try this one more time: #2124

[quickfur]

I have to say, I agree with @JackStouffer that I personally prefer the current /phobos docs over the /library ones. Having one page per function is not necessarily bad, but it needs to be done with finesse rather than a hamfisted approach.

For example, the docs for std.algorithm.find contains 5 links at the top of the page, all named "function find", apparently 5 overloads, but the link text contains no information about which overload is which. This is followed by 5 blocks describing each of 5 overloads, but formatted in a way that upon looking at the first block, you wouldn't expect that it's just one of 5 blocks.

Then within each overload, far too much space is wasted displaying the function signature is an unusual formatting (compile-time parameters on one line, whereas runtime parameters forced into one line per parameter even if the whole thing actually fits on one line), extraneous space in the sig constraints line, and no syntax highlighting beyond type names. Worse yet, the constraints are then repeated again under a Constraints heading, which is redundant information, along with each parameter's full description in one-parameter-per-row table that seems excessive especially since we've already just seen each parameter on its own line.

It would have been much better if the function signature were more cleverly formatted, e.g., in a more visually-pleasing way, and in a way more closely resembling normal D style, such as:

InputRange find(alias pred, InputRange, Element)
               (InputRange haystack, scope Element needle)
if (isInputRange!InputRange &&
    is(typeof(binaryFun!pred(haystack.front, needle)) : bool));

I.e.:

1. Parameters should be kept on the same line if they fit on the same line;
2. If there are CT parameters and runtime parameters and together both are too long to fit on one line, wrap the RT parameters in such a way that it lines up with the CT parrameters;
3. If either CT or RT parameters don't all fit onto the same line, wrap them in a way that's nicely indented, with type names and parameter names lined up nicely, e.g. (for illustration purposes, assume they don't in the following, just to illustrate the point):

InputRange find(alias pred,
                InputRange,
                Element)
               (InputRange    haystack,
                scope Element needle)

4. Sig constraints should be split up by && if they are too long, and indented such that they line up nicely;
5. (Not shown here) Full syntax highlighting should be enabled, i.e., type names in a different style from function names, parameter names in a different style from both, and any default arguments in yet another style.

Overloads should be handled more deftly, by providing a more consistent way to navigate between overloads, or seeing an overview of all overloads that's more informative than "function find" repeated 5 times without any apparent difference between them. The current way of navigating between overloads in the /phobos docs is way better, IMO, even if simplistic (a list of overload numbers and links to go to the next/prev overloads). Sometimes simple is more functional than fancy-but-dysfunctional.

Then there's the matter of larger-scale navigation. The current navigation is functional, but there's one big gap: some visual indication of where the current function sits within the current module. Without this visual cue, one feels pretty lost at where in the forest of docs we're at right now. At least in the one page per module /phobos docs, the scrollbar serves as a crude indication of where you are (even though I'll be the first to say that I'm not a fan of huge long pages where scrolling by 1 pixel translates to more than a pageful of content -- Phobos docs are nowhere near that huge, but some of the larger modules are getting uncomfortably close to it). There's also the not-very-noticeable link in the overly large symbol itself (the link to std.algorithm.searching in std.algorithm.searching.find), which helpfully takes you back to the module overview, but:

1. It seems to operate on a different dimension from the hierarchical navigation on the left, which is confusing because one would expect the nav tree should expand to show the list of symbols within the module and where your current place in it is.

2. It takes you back to the top of the module overview, rather than to where find is listed in the cheat sheet or the list of functions. Again, this makes it hard to navigate; I may be browsing through the docs and click on find to see what's in there, then I want to go back to where I left off and continue browsing the list of functions, but the return link resets me back to the top of the page instead.

3. In the module overview, the cheat sheet table and the list of functions seem to be needless repetition of essentially the same information, only slightly different. Ideally, they should be unified. Not to mention that under the "Functions" section, the overloads of find use yet another formatting style (function name + parameters without types), and each overload occupies a new row. This is inconsistent. We need to choose a specific style for presenting a function signature and stick to it, instead of vacillating between who knows how many different ways of presenting the same information as if we're not sure how to best present it. On this front, IMO the cheat sheet table is far more effective than the (IMO poorly) machine-generated list of functions. Keep the cheat sheet and dump the list of functions, I say.

4. Also, specifically in std.algorithm.searching, why is find listed among "Functions" and canFind listed under "Templates"?! This is an implementation detail that's ridiculous to use as a basis for page organization. Surely the ddox generator ought to understand the concept of an eponymous template, and treat it accordingly! Since it's clear that the intent is that the user would call canFind as a function, regardless of how it's actually implemented under the hood.

So overall, the /library docs feel very raw and unfinished to me, and while they're certainly functional (in the sense that you can get from A to B, and you can find the function descriptions and other such information if you look hard enough), and arguably easier to link to and all that, there are a lot of details in the execution that scream "hamfisted" to me rather than "finesse". Until these issues are addressed, I just don't feel very inclined to use the /library docs very much, and would have a hard time recommending them to anyone.

[CyberShadow]

Worth noting that even though some other languages do choose to arrange their documentation with one page per symbol (PHP, Perl, .NET, Swift), this approach is not universal or even that popular - Java, Python, Ruby, Haskell, Erlang, Lua, Go, Rust all have multiple (related) symbols on one page.

[quickfur]

I'm not fundamentally against one page per function. But it does need to be done better in order to be usable.