Cloud.gov app log improvements

[FuhuXia]

We are collecting cloud.gov logs on New Relic, implemented by this ticket. Improvements can be done to make it more useful.

1. Collect complete logs from all apps.
   Web access logs are missing. We'd like to see access logs containing info such as visitor's real IP, response code, referral url...
2. Be able to sort logs by spaces, apps, pods...
3. Break log messages into query-able fields.
   Should be able to set filters on process type(web or harvest task), error code, client IP...

[hkdctol]

Moving to icebox for now, until FCS move.

[FuhuXia]

https://docs.newrelic.com/docs/logs/ui-data/parsing/
It allow us to break the unstructured log message into query-able data, make it possible to do facet search and better charts and alerts.

[jbrown-xentity]

Used Grok to build in a new system to filter out and be able to parse the cloud.gov info in front of the logs. Current grok looks like:

%{TIMESTAMP_ISO8601:log_time} %{GREEDYDATA:summary} \- (\[)?(gauge@%{NOTSPACE:gauge} )?(name="%{NOTSPACE:name}" )?(value="%{NOTSPACE:value}" )?(unit="%{NOTSPACE:unit}")?(\])?\[(tags@%{NOTSPACE:tags} )?(app_id="%{NOTSPACE:app_id}" )?(app_name="%{NOTSPACE:app_name}" )?(deployment="%{NOTSPACE:deployment} )?(index="%{NOTSPACE:index}" )?(instance_id="%{NUMBER:instance_num}" )?(ip="%{IP:ip}" )?(job="%{NOTSPACE:job}" )?(organization_id="%{NOTSPACE:org_id}" )?(organization_name="%{NOTSPACE:org_name}" )?(origin="%{NOTSPACE:origin}" )?(process_id="%{NOTSPACE:process_id}" )?(process_instance_id="%{NOTSPACE:process_instance_id}" )?(process_type="%{NOTSPACE:process_type}" )?(source_id="%{NOTSPACE:source_id}" )?(source_type="%{NOTSPACE:source_type}" )?(space_id="%{NOTSPACE:space_id}" )?(space_name="%{NOTSPACE:space_name}")?\]%{GREEDYDATA:message}