engine: return HTTP 404 when Page404 component is rendered

[algomaster99]

Extracted from #690 (comment)

Example: the engine renders a huge number "404", however it already responded with HTTP 200:

URL: https://dvc.org/doc/command-reference/bad

It would be ideal if the engine could check for the URL path validity first before responding by rendering the general layout (navigation, etc) with HTTP 200, and render it instead with HTTP 404 (and Page404 ).

This can aid in checking links by running script in #690. Once 404 is returned, we will be able to distinguish between existing pages and Not found ones.

[jorgeorpinel]

And apparently one of these bad URLs from the website, it responds 304 Not Modified, not sure why.

Anyway, I wonder if @iAdramelk has any tips as to how to address these questions?

[shcheklein]

please, take a look at what URLs exactly does it return 304 (200 if you click it once again, I think - it's just a matter of caching on the browser side), 404. The point here is that it's engine requesting specific *.md files, not a regular dvc.org/doc/something.

[algomaster99]

@shcheklein Is it possible if we can render Page404 on a different URL? This way we will be able to pass writeHead(404) in server.js. I am not sure about this method though.

[shcheklein]

@algomaster99 could you elaborate please, not sure I understand the suggestion.

[algomaster99]

@shcheklein can you tell me in which file are the components mapped to their URL. I don't have much experience in next framework. But with simple react, I'm familiar. Usually the URLs are defined like this <Route path='/' component={App} />. I am looking for something corresponding to this here.
Basically, how does Next map each url to a component?

[shcheklein]

@algomaster99 the exact .md file to fetch is a convention that is based on the page URL + sidebar.json. Check also helper.js file. It's somewhat related to Next.js, but mostly our own logic that maps URLs to source files.

[algomaster99]

@shcheklein okay so, / is served my the server. Then rest after is mapped by helper.js, right?

[shcheklein]

@algomaster99 no, I think dvc.org/doc is mapped (by Next.js, implicitly) to the doc.js that is responsible for everything else after that. Unless I'm missing something.

[algomaster99]

@shcheklein so I went through the next documentation. I found two possible techniques to solve the problem.

• https://nextjs.org/docs#custom-error-handling
  I tried this but I am not sure on what URL does this component get triggered.

• https://nextjs.org/docs#custom-routes-using-props-from-url
  We can create a URL /doc/404 where we simply render the 404 component.

What do you suggest?

[shcheklein]

so, even if change URL to /doc/404 when we render this wget won't detect this. The only way to detect with wget is either add SSR support (@iAdramelk should have more context on this), or use Selenium to write tests (which might be a good idea anyway, since will be able to test more things, not just broken URLs).