Add custom extension to robots.txt cache

[p37307]

Because of the domain names extension, my OS is associating filetypes to the cached robots files in artifacts.

For instance, it associates domains ending in .au as audio files, .one as MS Office OneNote files, .cat as Windows security cat files, .com as MS_DOS applications, etc.

It may not be a big deal unless my search indexer tries to read its data, expecting to get data associated with those filetypes and mucks up.

Could you add a unique extension when it saves the robots file that is universally recognized as a text file.

[nanos]

Oops. I do realise that this is probably not the smartest anyway. I’m sure bad stuff will happen if someone has utf-8 glyphs in their domain name. I think I’ll just do a hash of the domain name and use that…