Stop generating sitemap.xml.gz (#6561)

[reebalazs]

We generate sitemap-index.xml which is also put correctly into robots.txt. However we still provide the old sitemap.xml.gz.

But, while this serves no purpose in addition to the index, it can cause problems if Google somehow parses it (for example it's submitted to the Google Search console). For this reason, we stop providing sitemap.xml.gz.

• I signed and returned the Plone Contributor Agreement, and received and accepted an invitation to join a team in the Plone GitHub organization.
• I verified there aren't other open pull requests for the same change.
• I followed the guidelines in Contributing to Volto.
• I succesfully ran code linting checks on my changes locally.
• I succesfully ran unit tests on my changes locally.
• I succesfully ran acceptance tests on my changes locally.
• If needed, I added new tests for my changes.
• If needed, I added documentation for my changes, either in the Storybook or narrative documentation.
• I included a change log entry in my commits.

---

Closes #6561

[netlify]

✅ Deploy Preview for plone-components canceled.

Name	Link
🔨 Latest commit	3ccc9b3
🔍 Latest deploy log	https://app.netlify.com/sites/plone-components/deploys/6776921d9e55540008e6aa35

[sneridagh]

@plone/volto-team anybody else feedback?

[ichim-david]

@sneridagh I asked at Edw and at least as a preliminary reply was that it doesn't affect us as we check also for sitemap-index.xml. So at least for us it seems fine. Waiting to see what the other companies have to say about this but in the meanwhile I wanted to write to comment to know that we've discussed about this and we are ok with these changes

[erral]

I would add a redirect from sitemap.xml.gz to sitemap-index.xml.gz to gracefully handle the change.

[reebalazs]

I would add a redirect from sitemap.xml.gz to sitemap-index.xml.gz to gracefully handle the change.

I think that a redirect, while technically possible, is problematic because sitemap-index.xml is not gzipped and the original one is. So we would serve a file ending with .gz but actually not gzipped. Whether or not google eats this or not, is not possible to find out from the documentation. We could figure out experimentally... but then what if they change things.

So if we don't redirect, then Google either finds the sitemap from robots.txt. or it has to be changed once in the console, but it's explicit and there is no magic behind that might break things in the future.

(The reason for it being not gzipped is because of following the google documentation. It seemed simpler to do the same then to start experimenting what can work and what not.)

[erral]

Sorry I have written it incorrectly, I meant a redirect between sitemap.xml.gz and sitemap-index.xml

[sneridagh]

@erral Maybe we are not talking about the same, but if we create a redirect from sitemap.xml.gz to sitemap-index.xml it can bring some other problems, starting with the fact that it's not a .gz file.

@reebalazs I can understand that people that have their consoles pointing to the old, can have problems and will require action, so it's not an easy one (and then, it would require to add it in a breaking change).

The main problem here is that we are providing pointers to both... and could lead to duplicated indexes. Not an easy one. We'll have to test if a issuing a 301 would work for the console and make sure we issue the correct mimetype too.

[erral]

Then, if we are removing an existing URL, I think it should be considered a breaking change.

[reebalazs]

@erral Maybe we are not talking about the same, but if we create a redirect from sitemap.xml.gz to sitemap-index.xml it can bring some other problems, starting with the fact that it's not a .gz file.

@reebalazs I can understand that people that have their consoles pointing to the old, can have problems and will require action, so it's not an easy one (and then, it would require to add it in a breaking change).

The main problem here is that we are providing pointers to both... and could lead to duplicated indexes. Not an easy one. We'll have to test if a issuing a 301 would work for the console and make sure we issue the correct mimetype too.

The problems are

1. the duplication
2. the old one is only working until a site has grown as big that Google will reject the sitemap.

[reebalazs]

Then, if we are removing an existing URL, I think it should be considered a breaking change.

@erral:

Yes it's a breaking change, but we need this change if we want to fix the situation. The alternative is not doing anything. In this case the old index will continue to work, until the site grows large enough and it breaks. Or even worse, there will be a duplicate that might spoil the SEO. By providing the indexed version by default, we can avoid this.

Also, by "breaking" it means that Google won't be able to access the old index any more. So one should either go to Google search console and add the new index manually, OR Google can pick up the new index from robots.txt, which it SHOULD be smart enough to do, but I'm not 100% sure if that's the case.

If someone could confirm that Google picks up the new index from robots.txt. then there is no breaking at all. If this is not the case then "breaking" means that manual intervention is needed in the search console. I believe if we put a loud enough message in the changelog to "CHECK YOUR SEO AND UPDATE IF NECESSARY", then this should be enough.

[erral]

If we have hit a maximum number of URLs that Google can consider in a single sitemap.xml.gz we can set a limit in the URLs generated in that file, and document that in the Upgrade guide.

This way, although still being a breaking change, the url will keep working, and those small sites that don't reach the limit can still keep working.

We can set the limit to the limit you have set to build each of the files referenced in the sitemap-index.xml

[tisto]

@erral setting a limit is very dangerous, IMO, since the admins of a site won't get notified if they hit the limit. I agree that this is a breaking change. However, I would fix this once and for all with a breaking change release and just support the index sitemap. Anything else will lead to confusion and make the situation worse than it is already (we lost our entire SEO rankings for plone.de thanks to this problem; I don't want to imagine what would happen if that happened in a client project).