Skip to content

Guide to Debugging Production API with Logs

Casey Flynn edited this page Sep 23, 2015 · 16 revisions

Guide to tracing API worker operations in Loggly

Table of Contents

Tracing Guides

Trace Builds

Relevant workers:

  • create-image-builder-container
  • on-image-builder-container-create
  • on-image-builder-container-die

Start w/ contextVersionId

Set query time range to include time of build. This could be several days if deploy uses a deduped build that was built several days ago.

Use grid view and add json.msg and json.module and json.tx.txTimestamp as display columns. Sort by txTimestamp.
Query

(json.module:"lib/workers/create-image-builder-container.js" OR json.module:"lib/workers/on-image-builder-container-create.js" OR json.module:"lib/workers/on-image-builder-container-die.js") AND "{{contextVersionId}}"

Example of successful build, passing through all workers. image00

Trace Deployments

Relevant workers:

  • create-instance-container
  • on-instance-container-create
  • start-instance-container
  • on-instance-container-start

Deployments can be initiated from several places. Github hooks, build completion events, POST /instances/ with built build. Each of these scenarios creates an internal request to the deploy route. The deploy route enqueues a “create-instance-container” task to be performed by a worker process. Knowing which workers are started & completed can aid in locating a fault in the system. Faults typically include missing images, image transfer failures, docker timeouts, etc. “deploymentUuid” is a unique key assigned to all logs across a single run of these four workers. Find the deploymentUuid of the deployment you’re interested in tracing. Use datetime ranges and following query. json.module:”lib/workers/create-instance-container.js” AND json.environment:”{{production or production-beta}}” AND json.msg:”CreateInstanceContainerWorker.prototype.handle”

Clone this wiki locally