How to document rakudo bugs?

[AlexDaniel]

I know that the first thought may be “let's not”, but hold your horses.

See this discussion: https://irclog.perlgeek.de/perl6/2017-09-01#i_15103501

The current situation is indeed LTA for the user, but linking all rakudobugs in the docs is way too LTA for us. We need some clever solution.

Any ideas? Like maybe some separate resource (“6aniuse”?) with a table of all features and the status in each implementation. Pretty much like https://perl6.org/compilers/features. Then maybe we can link to it automatically from the docs.

[AlexDaniel]

While we are discussing this to death, here's the plan: boldly document important bugs. Here's an example of a bug that needs to be mentioned.

There are more than 1500 open tickets, but not all of them deserve a mention in the docs. I am predicting that with ≈20 ticket mentions we can make sure that users see information about serious bugs that can cause a disaster. This works today, but once we have a better system we can always revert these changes.

[JJ]

Why don't you link some bugs that should be mentioned, and we try and document them?

[AlexDaniel]

Well, there's SEVERE tag that may be used for that. See RT and GH. But that tag is not being used consistently (IIRC I'm the only one using it), so there are not many tickets.

[JJ]

There are 6 in GH right now. Could we start there? And where would it go? An exclusive bug page? It would be better to keep track of them and eliminate once they have been squashed.

[jnthn]

If bugs are to be mentioned in the docs, and the concern is about knowing to remove the mentions when they are fixed (or include a "fixed from Rakudo YYYY.MM"), perhaps it would be possible to somehow have an "inverse test" associated with each one - that is, a test case that passes so long as the bug is present, and starts to fail when fixed. Or, perhaps less work, some reference to the bug's GitHub issue and a script run regularly that points out when said issues are closed.

[JJ]

I'd vote for the second option.

[AlexDaniel]

So basically a way to write a snippet that will eventually fail when you run xtest and the bug is fixed? @coke what do you think?

[coke]

Our current code testing is compile time checks only (wherever possible) - I think having a standard way to refer to the bug status and then check the bug status (rather than running code) is the way to go. We could just say any bug covered this way has to have a github issue, making it easier to do the check.

[JJ]

Hum. I don't see an easy way to do this. Plus we are doing this before documenting any bugs. I would say that when the need arises, just insert a =comment in the documentation saying : please remember to check this and that out. If we add tests for state of bugs anyway, please make that a separate issue (and maybe include the NOTSPECCED and NYI issues). But really, I don't think that's going to be needed at all.

[coke]

up to 27 severe bugs at https://github.com/rakudo/rakudo/issues?page=2&q=is%3Aissue+is%3Aopen+label%3Asevere

[coke]

if someone can please identify at least one rakudo bug that we should document, I'll write the test and then we'll have a way forward for any new ones.

[patrickbkr]

I think rakudo/rakudo#2380 would be easy to test and is specific.

[coke]

@patrickbkr can you suggest where in the docs we should warn about that bug and maybe some text for it? Then I can focus on checking the bug status in an xt/ test.

[patrickbkr]

@coke I think the bug I mentioned four years ago is not a good example anymore, because of these four PRs which I plan to get merged soon-ish. I'll have to think a bit if I can come up with a better example issue.

[coke]

It looks like several links exist already in =comment blocks... but it also looks like =comment blocks are stripped in our process by precompilation and not available when we are examining $=pod.

Once we can see the =comment block, we can examine the L<> to see if the referenced issue is open or closed. If closed, we fail the test for that link, with a note to remove the comment or exposed link.

We could also move the links to issues outside of a =comment block (but this requires some editor-ing)