Stop generating sitemap.xml.gz (#6561) #6562
Conversation
✅ Deploy Preview for plone-components canceled.
|
|
@sneridagh I asked at Edw and at least as a preliminary reply was that it doesn't affect us as we check also for sitemap-index.xml. So at least for us it seems fine. Waiting to see what the other companies have to say about this but in the meanwhile I wanted to write to comment to know that we've discussed about this and we are ok with these changes |
|
I would add a redirect from |
I think that a redirect, while technically possible, is problematic because So if we don't redirect, then Google either finds the sitemap from robots.txt. or it has to be changed once in the console, but it's explicit and there is no magic behind that might break things in the future. (The reason for it being not gzipped is because of following the google documentation. It seemed simpler to do the same then to start experimenting what can work and what not.) |
|
Sorry I have written it incorrectly, I meant a redirect between |
|
@erral Maybe we are not talking about the same, but if we create a redirect from @reebalazs I can understand that people that have their consoles pointing to the old, can have problems and will require action, so it's not an easy one (and then, it would require to add it in a breaking change). The main problem here is that we are providing pointers to both... and could lead to duplicated indexes. Not an easy one. We'll have to test if a issuing a 301 would work for the console and make sure we issue the correct mimetype too. |
|
Then, if we are removing an existing URL, I think it should be considered a breaking change. |
The problems are
|
Yes it's a breaking change, but we need this change if we want to fix the situation. The alternative is not doing anything. In this case the old index will continue to work, until the site grows large enough and it breaks. Or even worse, there will be a duplicate that might spoil the SEO. By providing the indexed version by default, we can avoid this. Also, by "breaking" it means that Google won't be able to access the old index any more. So one should either go to Google search console and add the new index manually, OR Google can pick up the new index from robots.txt, which it SHOULD be smart enough to do, but I'm not 100% sure if that's the case. If someone could confirm that Google picks up the new index from robots.txt. then there is no breaking at all. If this is not the case then "breaking" means that manual intervention is needed in the search console. I believe if we put a loud enough message in the changelog to "CHECK YOUR SEO AND UPDATE IF NECESSARY", then this should be enough. |
|
If we have hit a maximum number of URLs that Google can consider in a single This way, although still being a breaking change, the url will keep working, and those small sites that don't reach the limit can still keep working. We can set the limit to the limit you have set to build each of the files referenced in the |
|
@erral setting a limit is very dangerous, IMO, since the admins of a site won't get notified if they hit the limit. I agree that this is a breaking change. However, I would fix this once and for all with a breaking change release and just support the index sitemap. Anything else will lead to confusion and make the situation worse than it is already (we lost our entire SEO rankings for plone.de thanks to this problem; I don't want to imagine what would happen if that happened in a client project). |
|
Don't get me wrong, I have also faced picky SEO things, and I understand how hard and important is. I understand that if we have hit the limit we need to provide a way to fix it, and we have done so using the index thing. I only say that just removing the sitemap.xml.gz can be a "hard" thing, so to say. That's why I proposed 2 possible solutions (redirect and limit the amount of the URLs). If none of them is valid, or if the solution is worst than the problem, fair enough. I just wanted to raise the concern of removing a URL. |
Ok so if we want to do the redirect, we have to test it out. There is no other way.
If the answer to these is true then yes, we can do the redirect. If no, then no it does not work. EDIT actually we can eliminate the redirect and simply serve the index file also with the old name. (Even simpler to do from the middleware.) Then only the second question remains, is it a problem for Google that the extension is .gz but it's actually a plain xml? If Google is opportunistic about this, then this might just work. (If not, we'd need to gzip it, which is also not too hard.) Does anyone have the capacity to test this out with the Google console, see if a file is actually picked up? I'm flooded right now so I can't make any promises. |
|
I will test the redirect thing between |
|
We have to document the issues |
Where should the docs go? I'd say at least a mention in the Upgrade Guide. Is any of this configurable or changeable? If so, then it's probably a how to guide, else it's probably explanation. |
|
After talking with the guy that handles Search Console in our company, the solution may be easier than we thought: The This way, we don't need to change anything, we just change the contents of the My colleague says it doesn't matter to serve the file compress or uncompressed, he even suggests to compress every sitemap file, even the index one. |
That should be pretty easy then. I also suspected that Google does not care if the file is compressed or uncompressed. Still for future compatibility I suggest:
That way every new site setup would see |
|
We have tested this approach (serve the index file gzipped in the sitemap.xml.gz file) in 2 sites of ours and Search Console reports that the files are OK and the reported number of URLs is correct. |
reebalazs
force-pushed
the
ree-remove-old-sitemap
branch
from
3ccc9b3 to
6dc9010
Compare
@erral I've added this, can you please test it out and approve? Thank you! |
reebalazs
force-pushed
the
ree-remove-old-sitemap
branch
from
6dc9010 to
c1e3d8c
Compare
|
I will try to deploy these changes in a Volto 18 site. |
I've left out some part and force pushed later, can you please double check that you have the final? It has |
|
Mmm, strange. The headers mark that the content is sent gzipped, which it happens, but the transferred file is not a gzipped file, but a text/plain: And then: I would expect that not only the transfer is gzipped, but the file itself also, right? This is the example with the current https://demo.plone.org/sitemap.xml.gz And the contents: |
reebalazs
changed the title
|
@erral, sorry, my bad. I'll ping you when I've updated it. |
reebalazs
force-pushed
the
ree-remove-old-sitemap
branch
from
c1e3d8c to
4212296
Compare
|
@erral please check it out now! |
reebalazs
changed the title
We generate sitemap-index.xml which is also put correctly into robots.txt. However we still provide the old sitemap.xml.gz. But, while this serves no purpose in addition to the index, it can cause problems if Google somehow parses it (for example it's submitted to the Google Search console). For this reason, we stop providing sitemap.xml.gz. - also serve the batched sitemap under the old name sitemap.xml.gz - update comment in robots.txt
reebalazs
force-pushed
the
ree-remove-old-sitemap
branch
from
4212296 to
17491ce
Compare
We generate sitemap-index.xml which is also put correctly into robots.txt. However we still provide the old sitemap.xml.gz.
But, while this serves no purpose in addition to the index, it can cause problems if Google somehow parses it (for example it's submitted to the Google Search console). For this reason, we stop providing sitemap.xml.gz.
Closes #6561