Abstract storage backend so that custom tables aren't required

[westonruter]

In terms of where/how log entries are stored, there is a wp_stream_log_handler and there is also a wp_stream_no_tables filter (see #147). However, these filters don't touch on changing how log entries are read back out of the system.

In the case of sites running on WordPress VIP or for sites located on hosts which disallow custom tables, it would be great if the DB storage could be abstracted. Two storage backends could be included, one for using the existing custom tables, and then another (fallback) one which uses posts, postmeta, and other standard tables.

[frankiejarrett]

This is an excellent idea, just need someone to champion it.

[shadyvb]

I think it should be delayed till #65 is cleared so table structure is more mature with the new fields added for site_id and blog_id. I'll probably take care of it then.

[westonruter]

We may want to abstract this to the point where the the data is interacted with via a REST interface. This would ensure that the API is suitable for remote data storage.

[westonruter]

Of course, abstracting the storage backend also entails abstracting the querying interface. You gotta be able to get the data back out!

[shadyvb]

Lets just say that lots of stuff is gonna change 😆

[bordoni]

Would be awesome to be able to send these logs to something like ElasticSearch to be used with something like Kibana

[shadyvb]

@bordoni That's the plan! 😄 /five!

[gedex]

If logs are going to be sent to external service, it would be good to defer all logs into temporary stack and then submit all deferred logs in one request to that external service. During page load lifecycle there might be dozen of action-callbacks being logged, I think it's good for performance to avoid dozen of external requests too.

[shadyvb]

@gedex Sure, we'll need to tweak the logger to allow sending stuff in patches, via cron, or via async requests.