Enable web site log analysis

[dfandrich]

Right now, the statistics that Fastly gives on web site accesses is extremely broad. It would be useful at times to have more details on which URLs are being hit most, and which ones are causing errors. Logs on the origin server don't provide many details because of the caching. This would be very useful in cases like #344 for example, and it might have spotted #272 for us, as well as just giving general estimates on site usage.

Fastly has just announced their own analysis system that is in beta: https://www.fastly.com/blog/introducing-log-manager-and-insights-now-in-beta It might be worth asking them if we could try it out. With curl's traffic levels, even running it for an hour would likely provide some useful insights.

Alternately, running a simple access log analysis tool like https://www.awstats.org/ on the origin server would be better than nothing.

If this is done, privacy of the reports needs to be considered, which probably means most reports can't be publicly visible.

[vszakats]

Assuming it doesn't involve scripts / cookies / tracking pixels on the client-side — IOW it's based on server logs —, this seems useful to get a picture of what's consumed from the website.

[bagder]

I honestly can't see how origin logs would have helped for these problems.

It would be better with some sort of scan/verification of the content.

[vszakats]

If it could be counting e.g. curl-for-win downloads on any timescale, without any other parameter, it'd already be useful input. I assume fastly is filtering bogus requests from such logs, but even with those included, it's an indicator of scale. For now it's not possible to tell if that number is 1000, 100000, 10 million or 100.