Web-content management tool

[krizhanovsky]

Web-content is stored in TDB with tdbq interface. So a simple program is required to load arbitrary file to TDB using particular URI path, Content-Type, Last-Modified and ETag values generation. The script must be able to compress and load content in different compressed formats (so support of Vary/#508 is required). Also it should be able to extract web-content from TDB by particular URL to a file.

The same tool should be able to add Web content by URI/host/etc, inspect it (i.e. represent in nicer view than tdbq does it in it's raw format), delete or update, so it depends on #279 (TDB: full select, insertion and deletion of binary data from tdbq). In particular multi-oobject cache purging regular expressions must be supported. This is much flexible than current PURGE request scheme, since it can purge cache entries by content type, modification time, domain (vhost), wildcards etc. Probably TDB index must be updated to use Particia trie for efficient pattern matching over the cache tree.

Since the tool must be able to operate with TfwCacheEntry it has sense to develop it in C++ to be able to include kernel source file. Also it should be assumed to be able to load very large existing file system hierarchies of existing Web sites into Tempesta, so performance matters and that's one more point to do this in low level language instead of Perl/Python scripting. Probably, it should be separated from tdbq tool.

In future the tool must have RESTful inteface, so current PURGE scheme (#501) has no sense and should be removed. The tool also can be extended by Web page optimization stuff like minifying CSS, HTML or JS. This work can be run in separate daemon.

TBD: This issue doubles #471 and #66 as it provides third access method to web data. Probably we should just provide io_uring interface to request mmaped data from TDB and in this way it might also relate to #77.

Automatic Platform Optimization

CloudFlare provides APO which seems just an automatic population of changed content into the cache, i.e. instead of purging content on update, they immediately update the web cache. Very similar set of features is provided by the LiteSpeed WordPress plugin. This task will be easily done with the web-content tool. We can do not only web cache population, but also optimizations such as content minification.

Basically, the APO can be easily implemented using wget -r <domain_name>, probably with generated headers according to the web server settings for Vary values. However, in more advanced (heavy loaded) case wget requires to transfer response bodies and process them on the wget side. This is not wished and the Tempesta tool should send some header or introduce a specific method (this requires the Tempesta FW update) to make Tempesta FW just cache the responses and return only status code, without unnecessary headers and body.

Moved to a separate issue #1516

[krizhanovsky]

The task must also consider instant cache migration from other Web accelearators in case if Tempesta FW is introduced into an infrastructure just to mitigate a DDoS attack.