Add support for structured logs #624

TrevinTeacutter · 2018-08-13T22:02:18Z

It would be nice to have access logs that are structured as we use splunk for our logging aggregation and extracting key/value pairs is much nicer in splunk with structured logs than it is with regex. Could this be some sort of flag to enable this or did I just miss that this option is already enabled somewhere?

Blocked

contour serve should take its configuration from a configuration file #1130

davecheney · 2018-08-14T06:38:23Z

Thank you for raising this issue. The access logs are generated directly from envoy. At the moment we have flags to control their output destination, but their format is currently defaulting to whatever envoy uses by default -- something akin to apache2 log.

https://github.com/heptio/contour/blob/master/internal/contour/listener.go#L338

Looking at

https://www.envoyproxy.io/docs/envoy/v1.6.0/configuration/access_log#config-access-log-format

It looks like its possible to redefine the output of the logging format.

I'm wary of exposing the full configuration syntax of this to end users, it's too close to showing how the sausage is made. With that said, when you say structured logging, do you mean JSON?

TrevinTeacutter · 2018-08-14T15:05:52Z

Yes, I mean JSON. We recently turned JSON access logs on for Traefik in our old DC/OS clusters and they have been useful as well as capable of including/redacting/excluding headers of requests coming through the ingress making correlation really easy, stuff that normally doesn't get included in apache2 logs for brevity's sake.

yinsenyan · 2018-08-28T07:27:54Z

Maybe support define log format by configmap is a good way , just like nginx-ingress-controller use log-format-upstream parameter, also istio use logentry is a good practice.

davecheney · 2018-10-08T04:57:47Z

We're not going to get to this for the 0.7 release. Removing milestone.

azuretek · 2019-04-24T21:08:06Z

Has there been any agreement on this? Right now this is making Contour a less usable replacement for Nginx, having Envoy metrics available is great but getting full access log data would go a long way to making this a viable option.

davecheney · 2019-04-24T21:11:27Z

No there has been no agreement because, apart from a request for json formatted logs, no details have been provided.

…

On 25 Apr 2019, at 07:08, Abi X Renhart ***@***.***> wrote: Has there been any agreement on this? Right now this is making Contour a less usable replacement for Nginx, having Envoy metrics available is great but getting full access log data would go a long way to making this a viable option. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

azuretek · 2019-04-24T21:34:45Z

For our use case it would be nice if we could add an annotation or even just load a ConfigMap that will add a structure like so to every listener filter chain.

access_log: [
  {
    name: "envoy.file_access_log",
    config: {
      json_format: {
        downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%",
        path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
        authority: "%REQ(:AUTHORITY)%",
        protocol: "%PROTOCOL%",
        upstream_service_time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
        upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%",
        duration: "%DURATION%",
        downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%",
        user_agent: "%REQ(USER-AGENT)%",
        response_code: "%RESPONSE_CODE%",
        response_flags: "%RESPONSE_FLAGS%",
        start_time: "%START_TIME%",
        method: "%START_TIME%",
        request_id: "%REQ(X-REQUEST-ID)%",
        upstream_host: "%UPSTREAM_HOST%",
        x_forwarded_for: "%REQ(X-FORWARDED-FOR)%",
        requested_server_name: "%REQUESTED_SERVER_NAME%",
        bytes_received: "%BYTES_RECEIVED%",
        bytes_sent: "%BYTES_SENT%",
        upstream_cluster: "%UPSTREAM_CLUSTER%"
      },
      path: "/dev/stdout"
    },
  },
]

It looks like we can probably add it into this function with minimal effort https://github.com/heptio/contour/blob/master/internal/envoy/listener.go#L189

The configuration is defined here (and doesn't support nesting) for the json_format dictionary:
https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#format-dictionaries

Support for the format key should probably also be included in case users don't need json formatted output, though I don't know what would happen if both keys are defined.

I've validated this works for the static resources, but for LDS it looks like Contour is the only way to configure it.

davecheney · 2019-04-24T21:48:01Z

Our general preference is rather than make things configurable, we choose a reasonable default for all users. Is there a standard for json logs in this format? If so, we might just make that the default for everyone.

…

On 25 Apr 2019, at 07:34, Abi X Renhart ***@***.***> wrote: For our use case it would be nice if we could add an annotation or even just load a ConfigMap that will add a structure like so to every listener filter chain. access_log: [ { name: "envoy.file_access_log", config: { json_format: { downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%", path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%", authority: "%REQ(:AUTHORITY)%", protocol: "%PROTOCOL%", upstream_service_time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%", upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%", duration: "%DURATION%", downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%", user_agent: "%REQ(USER-AGENT)%", response_code: "%RESPONSE_CODE%", response_flags: "%RESPONSE_FLAGS%", start_time: "%START_TIME%", method: "%START_TIME%", request_id: "%REQ(X-REQUEST-ID)%", upstream_host: "%UPSTREAM_HOST%", x_forwarded_for: "%REQ(X-FORWARDED-FOR)%", requested_server_name: "%REQUESTED_SERVER_NAME%", bytes_received: "%BYTES_RECEIVED%", bytes_sent: "%BYTES_SENT%", upstream_cluster: "%UPSTREAM_CLUSTER%" }, path: "/dev/stdout" }, }, ] It looks like we can probably add it into this function with minimal effort https://github.com/heptio/contour/blob/master/internal/envoy/listener.go#L189 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

azuretek · 2019-04-24T21:59:44Z

I agree with that approach, there doesn't seem to be a standard for JSON access logs but this structure appears to have everything that Envoy reports (other than custom headers)

json_format: {
  @timestamp: "%START_TIME%",
  downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%",
  path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
  authority: "%REQ(:AUTHORITY)%",
  protocol: "%PROTOCOL%",
  upstream_service_time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
  upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%",
  duration: "%DURATION%",
  downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%",
  user_agent: "%REQ(USER-AGENT)%",
  response_code: "%RESPONSE_CODE%",
  response_flags: "%RESPONSE_FLAGS%",
  method: "%REQ(:METHOD)%",
  request_id: "%REQ(X-REQUEST-ID)%",
  upstream_host: "%UPSTREAM_HOST%",
  x_forwarded_for: "%REQ(X-FORWARDED-FOR)%",
  requested_server_name: "%REQUESTED_SERVER_NAME%",
  bytes_received: "%BYTES_RECEIVED%",
  bytes_sent: "%BYTES_SENT%",
  upstream_cluster: "%UPSTREAM_CLUSTER%"
}

azuretek · 2019-04-29T16:59:31Z

So I went ahead and did a build against the 0.11.0 code base with the changes we discussed.

#1053

The dynamic listener logs now look like this:

{"upstream_cluster":"default/hello-world/8080/da39a3ee5e","downstream_remote_address":"1.2.3.4:49286","path":"/?abc=123","authority":"hello.test.foo.bar","protocol":"HTTP/2","upstream_service_time":"1","@timestamp":"2019-04-25T06:31:07.162Z","upstream_local_address":"-","duration":"1","downstream_local_address":"10.220.3.107:443","response_code":"200","user_agent":"curl/7.54.0","response_flags":"-","method":"GET","request_id":"b4ecadf1-8fc0-4e87-89d8-a3aa851196d4","upstream_host":"10.141.181.15:8080","x_forwarded_for":"1.2.3.4","requested_server_name":"hello.test.foo.bar","bytes_received":"0","uber_trace_id":"-","bytes_sent":"17"}

I also added the same format to the static listener configs, this is the log for the health check endpoiont:

{"response_code":"200","user_agent":"kube-probe/1.11","response_flags":"-","method":"GET","request_id":"aadded21-6b3c-4fa4-bc3a-de62b6b4c2ec","upstream_host":"-","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","uber_trace_id":"-","bytes_sent":"0","upstream_cluster":"-","downstream_remote_address":"10.220.40.142:43216","path":"/healthz","authority":"10.141.207.73:8002","protocol":"HTTP/1.1","upstream_service_time":"-","@timestamp":"2019-04-25T06:31:09.869Z","upstream_local_address":"-","duration":"0","downstream_local_address":"10.141.207.73:8002"}

Let me know what needs to be done to get this approved. I was thinking it might make sense to add a flag to enable or disable JSON access logs, but as you said the preferred approach is to create sane defaults which does seem like a good approach.

bgagnon · 2019-05-02T01:34:47Z

@azuretek is there a reason or restriction for outputting numerical values as strings in the JSON objects?

Bytes and duration are things one would want to do aggregations and calculate metrics on (assuming the logs end up in Elasticsearch or similar database). For the response code, it's convenient for range queries (<100, 200-299, 300-399, 400-499,>500).

It would be nice to avoid an ETL for such simple JSON type coercions.

EDIT: just found the relevant part of the Envoy docs:

Format dictionaries have the following restrictions:

The dictionary must map strings to strings (specifically, strings to command operators). Nesting is not currently supported.

That's too bad :/

azuretek · 2019-05-02T17:39:05Z

@bgagnon definitely ran into that in my PoC, it's annoying but ultimately it's able to be handled by adjusting the mappings on the elastic search side. To make it report correctly we'd have to make changes to envoy (which should probably be done).

davecheney · 2019-05-04T00:23:02Z

Abi, Can you please detail the changes you require. If you can give complete examples of the expected json output that would be best. Thanks Dave

…

On 3 May 2019, at 01:39, Abi X Renhart ***@***.***> wrote: @bgagnon definitely ran into that in my PoC, it's annoying but ultimately it's able to be handled by adjusting the mappings on the elastic search side. To make it report correctly we'd have to make changes to envoy (which should probably be done ultimately). — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

azuretek · 2019-05-04T03:22:48Z

@davecheney I need the default access log format to be structured, preferably JSON to more easily ingest into storage backends like Elasticsearch. This is a requirement for us to move away from using NGINX as our ingress.

Currently access logs use the default format string, as defined in the envoy documentation: https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#default-format-string

[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%"
%RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION%
%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%"
"%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"\n

The desired format is JSON specifically with these keys and values. As defined in the envoy documentation https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#format-dictionaries

{
  "@timestamp": "%START_TIME%",
  "authority": "%REQ(:AUTHORITY)%",
  "bytes_received": "%BYTES_RECEIVED%",
  "bytes_sent": "%BYTES_SENT%",
  "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
  "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
  "duration": "%DURATION%",
  "method": "%REQ(:METHOD)%",
  "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
  "protocol": "%PROTOCOL%",
  "request_id": "%REQ(X-REQUEST-ID)%",
  "requested_server_name": "%REQUESTED_SERVER_NAME%",
  "response_code": "%RESPONSE_CODE%",
  "response_flags": "%RESPONSE_FLAGS%",
  "uber_trace_id": "%REQ(UBER-TRACE-ID)%",
  "upstream_cluster": "%UPSTREAM_CLUSTER%",
  "upstream_host": "%UPSTREAM_HOST%",
  "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
  "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
  "user_agent": "%REQ(USER-AGENT)%",
  "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%"
},

This is an example of how the access log should look

{
  "upstream_cluster": "default/hello-world/8080/da39a3ee5e",
  "downstream_remote_address": "1.2.3.4:49286",
  "path": "/?abc=123",
  "authority": "hello.test.foo.bar",
  "protocol": "HTTP/2",
  "upstream_service_time": "1",
  "@timestamp": "2019-04-25T06:31:07.162Z",
  "upstream_local_address": "-",
  "duration": "1",
  "downstream_local_address": "10.220.3.107:443",
  "response_code": "200",
  "user_agent": "curl/7.54.0",
  "response_flags": "-",
  "method": "GET",
  "request_id": "b4ecadf1-8fc0-4e87-89d8-a3aa851196d4",
  "upstream_host": "10.141.181.15:8080",
  "x_forwarded_for": "1.2.3.4",
  "requested_server_name": "hello.test.foo.bar",
  "bytes_received": "0",
  "uber_trace_id": "-",
  "bytes_sent": "17"
}

davecheney · 2019-05-04T04:36:17Z

Thank you for your reply. My goal is to change the log output format once, to something structured, without the option to change it. To do this I need your assistance to know if there is an established standard for json structured http logs to replace the current ncsa style log format.

…

On 4 May 2019, at 13:22, Abi X Renhart ***@***.***> wrote: @davecheney I need the default access log format to be structured, preferably JSON to more easily ingest into storage backends like Elasticsearch. This is a requirement for us to move away from using NGINX as our ingress. Currently access logs use the default format string, as defined in the envoy documentation: https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#default-format-string [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"\n The desired format is JSON specifically with these keys and values. As defined in the envoy documentation https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#format-dictionaries { ***@***.***": "%START_TIME%", "authority": "%REQ(:AUTHORITY)%", "bytes_received": "%BYTES_RECEIVED%", "bytes_sent": "%BYTES_SENT%", "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%", "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%", "duration": "%DURATION%", "method": "%REQ(:METHOD)%", "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%", "protocol": "%PROTOCOL%", "request_id": "%REQ(X-REQUEST-ID)%", "requested_server_name": "%REQUESTED_SERVER_NAME%", "response_code": "%RESPONSE_CODE%", "response_flags": "%RESPONSE_FLAGS%", "uber_trace_id": "%REQ(UBER-TRACE-ID)%", "upstream_cluster": "%UPSTREAM_CLUSTER%", "upstream_host": "%UPSTREAM_HOST%", "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%", "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%", "user_agent": "%REQ(USER-AGENT)%", "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%" }, — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

davecheney · 2019-05-06T05:59:51Z

Thank you to everyone who has contributed to this issue over the last few weeks.

I want to make it clear to all readers that I am not opposed to adding this feature. In fact I want to see it adopted as default.

What I do not want to see happen is JSON or NCSA become selectable especially if there are options to make each of those formats configurable. If this is the outcome we will stick with the default NCSA format.

Please do not send any PRs until there is agreement on which format will be used.

davecheney · 2019-07-19T12:47:47Z

#1130 landed in 0.14. Moving to 0.15 to implement.

davecheney · 2019-08-21T23:10:44Z

Sorry, we ran out of time in 0.15 for this. @youngnick has a plan for how to implement this in beta1.

Updates projectcontour#624 Signed-off-by: Nick Young <ynick@vmware.com>

youngnick · 2019-09-11T23:37:05Z

I've published #1485 with our current design ideas around this feature. I'm hoping to get this reviewed and agreed soon with the aim to land this in beta1.

Updates projectcontour#624 Signed-off-by: Nick Young <ynick@vmware.com>

youngnick · 2019-09-13T04:20:27Z

After internal review, I've merged #1485 with the team's preferred design for JSON logging. Please review either #1485 or envoy-json-logging.md for the details. Comments are welcome here.

Updates projectcontour#624 Signed-off-by: Nick Young <ynick@vmware.com>

Updates #624 Signed-off-by: Nick Young <ynick@vmware.com>

Fixes projectcontour#624. Also update a small bug in config file definition that can lead to parsing weirdness. Signed-off-by: Nick Young <ynick@vmware.com>

Fixes #624. Also update a small bug in config file definition that can lead to parsing weirdness. Signed-off-by: Nick Young <ynick@vmware.com>

davecheney added this to the 0.7.0 milestone Aug 14, 2018

davecheney removed the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Aug 14, 2018

davecheney removed this from the 0.7.0 milestone Oct 8, 2018

turbotankist mentioned this issue Oct 23, 2018

accesslog_format added #773

Closed

davecheney added blocked/needs-info Categorizes the issue or PR as blocked because there is insufficient information to advance it. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Apr 9, 2019

azuretek mentioned this issue May 2, 2019

JSON access logs #1053

Closed

davecheney added blocked/needs-design Categorizes the issue or PR as blocked because it needs a design document. and removed blocked/needs-info Categorizes the issue or PR as blocked because there is insufficient information to advance it. labels May 6, 2019

bgagnon mentioned this issue Jun 28, 2019

Tracing Configuration #399

Closed

derrickburns mentioned this issue Jul 11, 2019

Access logs solo-io/gloo#814

Closed

davecheney removed the blocked/needs-design Categorizes the issue or PR as blocked because it needs a design document. label Jul 19, 2019

davecheney self-assigned this Jul 19, 2019

davecheney modified the milestones: 0.14.0, 0.15.0 Jul 19, 2019

davecheney removed their assignment Aug 21, 2019

davecheney modified the milestones: 0.15.0, 1.0.0-beta.1 Aug 21, 2019

youngnick self-assigned this Sep 11, 2019

youngnick pushed a commit to youngnick/contour that referenced this issue Sep 11, 2019

design: Add JSON logging design

6dfcfab

Updates projectcontour#624 Signed-off-by: Nick Young <ynick@vmware.com>

youngnick mentioned this issue Sep 11, 2019

design: JSON logging #1485

Merged

youngnick pushed a commit to youngnick/contour that referenced this issue Sep 11, 2019

design: Add JSON logging design

4feb661

Updates projectcontour#624 Signed-off-by: Nick Young <ynick@vmware.com>

youngnick pushed a commit to youngnick/contour that referenced this issue Sep 13, 2019

design: Add JSON logging design

e59eb23

Updates projectcontour#624 Signed-off-by: Nick Young <ynick@vmware.com>

youngnick pushed a commit to youngnick/contour that referenced this issue Sep 16, 2019

Add support for JSON logging.

3281a14

Updates projectcontour#624 Signed-off-by: Nick Young <ynick@vmware.com>

youngnick pushed a commit to youngnick/contour that referenced this issue Sep 16, 2019

Add support for JSON logging.

831c618

Updates projectcontour#624 Signed-off-by: Nick Young <ynick@vmware.com>

youngnick mentioned this issue Sep 16, 2019

Add support for JSON logging. #1511

Merged

youngnick pushed a commit to youngnick/contour that referenced this issue Sep 16, 2019

Add support for JSON logging.

2004d67

Updates projectcontour#624 Signed-off-by: Nick Young <ynick@vmware.com>

youngnick pushed a commit to youngnick/contour that referenced this issue Sep 16, 2019

Add support for JSON logging.

5f76b25

Updates projectcontour#624 Signed-off-by: Nick Young <ynick@vmware.com>

youngnick pushed a commit that referenced this issue Sep 17, 2019

Add support for JSON logging.

63a34e5

Updates #624 Signed-off-by: Nick Young <ynick@vmware.com>

youngnick pushed a commit to youngnick/contour that referenced this issue Sep 17, 2019

Add documentation for structured logs.

cef2a5b

Fixes projectcontour#624. Also update a small bug in config file definition that can lead to parsing weirdness. Signed-off-by: Nick Young <ynick@vmware.com>

youngnick mentioned this issue Sep 17, 2019

Add documentation for structured logs. #1517

Merged

youngnick pushed a commit to youngnick/contour that referenced this issue Sep 17, 2019

Add documentation for structured logs.

c4f163d

Fixes projectcontour#624. Also update a small bug in config file definition that can lead to parsing weirdness. Signed-off-by: Nick Young <ynick@vmware.com>

youngnick closed this as completed in #1517 Sep 17, 2019

youngnick pushed a commit that referenced this issue Sep 17, 2019

Add documentation for structured logs.

b55f572

Fixes #624. Also update a small bug in config file definition that can lead to parsing weirdness. Signed-off-by: Nick Young <ynick@vmware.com>

bgagnon mentioned this issue Oct 11, 2019

Support Access Log Service (ALS) #1691

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for structured logs #624

Add support for structured logs #624

TrevinTeacutter commented Aug 13, 2018 •

edited by davecheney

Loading

davecheney commented Aug 14, 2018

TrevinTeacutter commented Aug 14, 2018

yinsenyan commented Aug 28, 2018

davecheney commented Oct 8, 2018

azuretek commented Apr 24, 2019

davecheney commented Apr 24, 2019 via email

azuretek commented Apr 24, 2019 •

edited

Loading

davecheney commented Apr 24, 2019 via email

azuretek commented Apr 24, 2019 •

edited

Loading

azuretek commented Apr 29, 2019 •

edited

Loading

bgagnon commented May 2, 2019 •

edited

Loading

azuretek commented May 2, 2019 •

edited

Loading

davecheney commented May 4, 2019 via email

azuretek commented May 4, 2019 •

edited

Loading

davecheney commented May 4, 2019 via email

davecheney commented May 6, 2019

davecheney commented Jul 19, 2019

davecheney commented Aug 21, 2019

youngnick commented Sep 11, 2019

youngnick commented Sep 13, 2019

Add support for structured logs #624

Add support for structured logs #624

Comments

TrevinTeacutter commented Aug 13, 2018 • edited by davecheney Loading

davecheney commented Aug 14, 2018

TrevinTeacutter commented Aug 14, 2018

yinsenyan commented Aug 28, 2018

davecheney commented Oct 8, 2018

azuretek commented Apr 24, 2019

davecheney commented Apr 24, 2019 via email

azuretek commented Apr 24, 2019 • edited Loading

davecheney commented Apr 24, 2019 via email

azuretek commented Apr 24, 2019 • edited Loading

azuretek commented Apr 29, 2019 • edited Loading

bgagnon commented May 2, 2019 • edited Loading

azuretek commented May 2, 2019 • edited Loading

davecheney commented May 4, 2019 via email

azuretek commented May 4, 2019 • edited Loading

davecheney commented May 4, 2019 via email

davecheney commented May 6, 2019

davecheney commented Jul 19, 2019

davecheney commented Aug 21, 2019

youngnick commented Sep 11, 2019

youngnick commented Sep 13, 2019

TrevinTeacutter commented Aug 13, 2018 •

edited by davecheney

Loading

azuretek commented Apr 24, 2019 •

edited

Loading

azuretek commented Apr 24, 2019 •

edited

Loading

azuretek commented Apr 29, 2019 •

edited

Loading

bgagnon commented May 2, 2019 •

edited

Loading

azuretek commented May 2, 2019 •

edited

Loading

azuretek commented May 4, 2019 •

edited

Loading