add theme's robots.txt to tera with name "robots.txt"

[mwcz]

IMPORTANT: Please do not create a Pull Request adding a new feature without discussing it first.

The place to discuss new features is the forum: https://zola.discourse.group/
If you want to add a new feature, please open a thread there first in the feature requests section.

This fixes #1721 by registering the theme's robots.txt template under the name "robots.txt" whereas it was previously being registered with the filepath, and later ignored during rendering.

Note: there are no tests for this PR at the moment. I'm looking into how and where to add some, but it might take me a bit.

Sanity check:

• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

Code changes

• Are you doing the PR on the next branch?

If the change is a new feature or adding to/changing an existing one:

• Have you created/updated the relevant documentation page(s)? (docs are already correct)

[Keats]

Thanks!

Note: there are no tests for this PR at the moment. I'm looking into how and where to add some, but it might take me a bit.

Probably an integration test: add a robots.txt file in test_site/themes/sample/ and then add an assert in components/site/tests/site.rs

[mwcz]

I tried this out but even if I add a test_site/themes/sample/robots.txt, the test site already has a templates/robots.txt which takes precedence. I've been trying to figure out a way to test this scenario without affecting the existing tests for templates/robots.txt and so far nothing has worked very well. Here's what I've toyed with:

1. make a slimmed-down copy of test_site to test this scenario → no, too much duplication
2. in the test, use build_site_with_setup and remove templates/robots.txt somehow before the site is rendered, allowing test_site/themes/sample/robots.txt to shine through. It seems like this could work but I haven't found an API yet that would allow that, after poking around Site, Library, and Tera references available from build_site_with_setup.

Do either of these sound good, or is there some other cleaner approach I'm missing?

[Keats]

Hmm, easier test setup is definitely something that needs work. Can you just add a TODO above the code saying it needs a test and a link to that PR? We'll add a test after some refactoring

[mwcz]

TODO added.

[Keats]

Just to confirm before merging, you did test manually that it worked correct?

[mwcz]

Yeah, I tested it on the blog @t3link posted in #1721.

https://asciinema.org/a/aSi4IQxolor2IYUkMxeLiflbi

[Keats]

Thanks!