Media files in url_cache/ and url_cache_thumbnails/ should not be stored in storage providers

[squahtx]

Description

Spotted by @richvdh:

Currently media files in url_cache/ and url_cache_thumbnails/ get replicated to storage providers.

These files are supposed to be transient, and so should not be stored in any storage providers (eg. S3)

Version information

• Homeserver: matrix.org

[clokep]

Or I wonder if they're meant to be deleted when we purge old previews?

synapse/synapse/rest/media/v1/preview_url_resource.py

Lines 482 to 484 in b932590

	async def _expire_url_cache_data(self) -> None:
	"""Clean up expired url cache content, media and thumbnails."""
	# TODO: Delete from backup media store

[squahtx]

Looks like this is a duplicate of #5411, except that issue is about local_thumbnails/ and remote_thumbnail/ as well.
Let's keep this issue for the url_cache part of #5411 and let the original issue be about the thumbnails.

[squahtx]

Or I wonder if they're meant to be deleted when we purge old previews?

Interesting find!

I'd argue that since url cache data is so short lived, it makes sense to not hand it over to any storage providers:

• Incurring S3/cloud costs for short lived data seems wasteful (echoing the complaint in Synapse uploads thumbnail data to S3 #5411)
• The impact of omitting it from storage providers acting as backups is limited to 24 hours anyway

... or we could add another per-storage provider config option, defaulting to do-not-store?

[clokep]

... or we could add another per-storage provider config option, defaulting to do-not-store?

I think limiting config is desirable, so best not to do this unless there's a huge benefit!