Lots of broken links, invalid HTML and so on (checklink)

[AlexDaniel]

I've used checklink utility on doc.perl6.org. Here is the output. I believe that all of these issues have to be fixed.

Some of the errors are repeated many times, that's why the output is so large.

Current status

Number of lines in the output	Date
126591	2016-05-31
128555	2016-06-03
132005	2016-06-09
128771	2016-06-11
126707	2016-06-12
130491	2016-06-16
134073	2016-06-23
131489	2016-06-25
134542	2016-07-06
35561	2016-07-13
35005	2016-08-01
33926	2016-08-02
20540	2016-08-03
17432	2016-08-04
17002	2016-08-05
8253	2016-08-06
8615	2016-08-10
7374	2016-11-12
8157	2016-12-20
208521	2017-06-30
207664	2017-07-01
211899	2017-11-04
213547	2018-01-05
184904	2018-03-03
55393	2018-05-05
54232	2018-05-06
57626	2018-11-04
39726	2020-07-26

[zoffixznet]

Are you sure you ran the program correctly? In the first item, http://doc.perl6.org/type/X::TypeCheck::Splice, the output says

x:/js/main.js   
  Line: 703
  Code: 501 Protocol scheme 'x' is not supported
 To do: Could not check this link: method not implemented or scheme not
    supported.

Yet, no scheme x: is in use on that line: <script type="text/javascript" src="/js/main.js"></script>

I checked a couple of other errors, and they similarly are false positives.

[zoffixznet]

There're a bazillion of markup errors though: https://validator.w3.org/nu/?doc=http%3A%2F%2Fdoc.perl6.org%2Ftype%2FX%3A%3ATypeCheck%3A%3ASplice

[dogbert17]

I’ll take a look at the 404’s, that should keep me occupied for a while. I suspect some of them will not be easily fixed, e.g. #155

From: Zoffix Znet [mailto:notifications@github.com]
Sent: den 31 maj 2016 17:52
To: perl6/doc
Subject: Re: [perl6/doc] Lots of broken links, invalid HTML and so on (checklink) (#561)

There's a bazillion markup errors though: https://validator.w3.org/nu/?doc=http%3A%2F%2Fdoc.perl6.org%2Ftype%2FX%3A%3ATypeCheck%3A%3ASplice

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub #561 (comment) , or mute the thread https://github.com/notifications/unsubscribe/ARklMk4v7pJ-1cu0eVQPU-YiAMupz_lqks5qHFktgaJpZM4IqGXO . https://github.com/notifications/beacon/ARklMlCJiQn1CkeROy2G3MMRyR8OAugMks5qHFktgaJpZM4IqGXO.gif

[AlexDaniel]

I checked a couple of other errors, and they similarly are false positives.

Saying that these are false positives is very optimistic. Sometimes the error message is LTA, other times the tool got confused because of markup errors. I'm not sure what's the issue in this particular case, but if you think that it is working incorrectly (that is, if it gets confused with correct HTML) then please file a bug report here.

[dogbert17]

I have fixed quite a few broken links. Unless I have missed some, which I probably have, the ones that remain point to the following, undocumented, entities:
NaN
HOW
mro

[AlexDaniel]

I've added a graph so that we can track the progress. However, until someone fixes html problems we are not going to get any meaningful data there.

[zoffixznet]

If it helps anyone, I tried to debug the double-links on headers and gotten only as far as the three %routines-by-type{*}.list in this section stuff links into headers and then when pod is generated the headers get wrapped into links by Pod::To::Html. Since complete content is included anyway, the %routines-by-type{*}.list needn't to have links in the content.

passes the torch

[AlexDaniel]

Actually, I think that most problems are now either fixed or have a corresponding issue filed. We need #584 to be fixed.

[coke]

@AlexDaniel - looks like #584 might be fixed; can you re-run this scan?

[AlexDaniel]

@coke Just did.

[Altai-man]

Mostly there are issues caused by "broken" method names.

The whole thing is like this:

1. We attach a type name to the method name to create unique link on a page like this:
   #(Metamodel::AttributeContainer)_method_rw
2. But when we make a link to method/routine/sub, it becomes just method_rw - so the link is invalid.

Without a type name there will be duplication of ids with routine/method/sub pages.

The priority task is to find a way to avoid duplication and to avoid breaking fragments. About 3 months ago, there was a talk with gfldex++ about how one can use number suffixes to avoid duplication in Pod::To::Html, but this part is hard.

There also a few easier issues such as links to items removed from Glossary, etc. They can be just removed/fixed and it will be okay, I think.

[Altai-man]

Some further investigation notes:

• There are a lot of links to .Str method. It should point to Str method page with Mu-related description, but instead it points to https://docs.perl6.org/routine/Str#class_Mu - Str .val method. Should be investigated and fixed. Related issue: /type/Str contains doc for method Str.val #683

[AlexDaniel]

Still looks pretty bad.

[AlexDaniel]

Updated the graph again.

[AlexDaniel]

FWIW this is the command I'm using:

checklink -b -D 25 -q doc.perl6.org | tee "$(date '+%F')"

[JJ]

What about listing broken links by file and go at them, one by one? Some links might be errors, some of them might actually have disappeared due to bitrot...

[AlexDaniel]

@JJ well, you can just search the output for “404 Not Found” error and you'll find all broken links. Everything else is probably broken html. The goal here is to get the output to 0 lines (we were almost there in 2016).

P.S. Updated the graph and the output again.

[JJ]

@abraxxa thanks a lot for the report, but there are so many, that it's difficult to know where to start...

[JJ]

Most 404 are now done, but now heading titles have drifted from their links, like in #2146 . Whenever you're working on a page, please check outgoing anchors for correctness.

[JJ]

@AlexDaniel can you run this again?

[AlexDaniel]

@JJ thanks for pinging, updated.

[JJ]

So it got worse...

[JJ]

Maybe we should check this again...

[AlexDaniel]

@JJ updated. But there's not much of use updating it unless somebody wants to actively work on fixing the issues.

[AlexDaniel]

A lot of broken links seem to be caused by a different directory structure:

Things like:

https://github.com/perl6/doc/blob/master/doc/Type/X::Proc::Async::AlreadyStarted.pod6
-> https://github.com/Raku/doc/blob/master/doc/Type/X::Proc::Async::AlreadyStarted.pod6
Line: 224
Code: 301 -> 404 Not Found

It doesn't mind the redirect (although we should avoid it anyway), but the link should have Type/X/Proc/Async/AlreadyStarted.pod6 (without ::).

[antoniogamiz]

Where have you found that link?

[AlexDaniel]

@antoniogamiz you can see it in the output yourself.

In this case it's the 🖉 link (Edit this page) from https://docs.raku.org/type/X::Proc::Async::AlreadyStarted.

[JJ]

Thanks, Aleks. In many cases, broken links are the result of errors in HTML that might or might not have been fixed; I was curious about that. And, of course, I don't think there's been any check after the Big Change (TM). I'm also going to start a round of bugfixes, so I just wanted to know the state of this. Again, thanks, I appreciate it.

[antoniogamiz]

In reality, output is a little bit smaller, because if you take a look at:

Processing	https://docs.raku.org/type.html


List of broken links and other issues:
https://webchat.freenode.net/?channels=	
  Line: 50
  Code: 200 OK
 To do: Some of the links to this resource point to broken URI fragments
	(such as index.html#fragment).
The following fragments need to be fixed:
	raku                          	Line: 50

https://github.com/Raku/doc/blob/master/CONTRIBUTING.md	
  Line: 78
  Code: 200 OK
 To do: Some of the links to this resource point to broken URI fragments
	(such as index.html#fragment).
The following fragments need to be fixed:
	reporting-bugs                	Line: 78

You that is reporting

• https://github.com/Raku/doc/blob/master/CONTRIBUTING.md#reporting-bugs
• https://webchat.freenode.net/?channels=#raku

But these links work perfectly. I do not know what those appear as broken fragments.

[JJ]

Because there are HTML errors and these are converted for some reason into missing links.

[antoniogamiz]

Hum, that's a weird behavior. Anyway, I'm using this command

 checklink -s -b -r --exclude "(webchat|CONTRIBUTING|perl6\.html)" docs.raku.org

And errors in edit links will be solved in the next Documentable release.

[AlexDaniel]

Yeah, I'm not sure about these particular ones. The link to webchat can be changed to have %23raku instead, it will both work and be a bit more correct I think.

[coke]

Will track in Raku/doc-website#72 on new site.