Guide to Development CLI Log Tools

Table of Contents generated with DocToc

Required:
Output Log Levels:
Examples:
ALL OF THE ABOVE CAN BE COMBINED WITH json to filter specific log keys

Note, all below examples are logs produced from running tests. To get the same easy-to-read logs from

production use a command like ssh alpha-navi tail -f /var/log/navi.log in place of npm run bdd-whitelist ex: ssh alpha-navi tail -f /var/log/navi.log | bunyan -o json --strict | json -ga module msg

##Required:

npm install bunyan -g
npm install json -g

bunyan-cli docs
json docs

##Output Log Levels: These ENV vars can specify levels to log to stdout and/or a file. If LOG_LEVEL_FILE is provided, logs will be appended to ./api.log

LOG_LEVEL_STDOUT=none|trace|debug|info|warn|error  
LOG_LEVEL_FILE=none|trace|debug|info|warn|error

NOTE: 1 test run of bdd-whitelist at trace level will result in a 300M+ file if using LOG_LEVEL_FILE

##Examples:

###Output logs from a single application request route (WIP example) Any route that has the following middleware at the beginning:

app.post('/workers/container-create',
  requestTrace('POST_WORKERS_CONTAINER_CREATE'))

Can be traced with ENV vars and bunyan conditional filters:

LOG_LEVEL_STDOUT=trace TID_POST_WORKERS_CONTAINER_CREATE=apples npm run bdd-whitelist | bunyan -c 'this.tid == "apples"'

###Run tests with no logs streamed to stdout, and all logs streamed to file:
LOG_LEVEL_STDOUT=none LOG_LEVEL_FILE=trace npm run bdd-whitelist
tail -f api.log | bunyan -o json --strict | json -d ' -- | --' -ga module msg

Give gif ~20 seconds to display relevant example:

###No filtering or pruning, standard bunyan output: LOG_LEVEL_STDOUT={trace|info|warn|error|fatal} npm run bdd-whitelist | bunyan

###Filtering on 1 file, no pruning, standard bunyan output: LOG_LEVEL_STDOUT={trace|info|warn|error|fatal} npm run bdd-whitelist | bunyan -c "this.module === "lib/workers/start-instance-container.js"

###All logs from two specific files at a given log level: LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -c "~['lib/workers/base-worker.js', 'lib/workers/on-instance-container-start.js'].indexOf(this.module)"

###All logs from modules in a folder at a given log level: LOG_LEVEL_STDOUT={trace|info|warn|error|fatal} npm run bdd-whitelist | bunyan -c "this.module.indexOf('lib/workers') === 0"

###Specific log in a specific file: LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -c "this.module === 'lib/workers/start-instance-container.js' this.msg.indexOf('handle') !== 0'"

###Random 10% of all logs (Totally useless but why not) LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -c "Math.random() < 0.1"

###Also useful: short output mode LOG_LEVEL_STDOUT={trace|info|warn|error|fatal} npm run bdd-whitelist | bunyan -c "this.module.indexOf('lib/workers') === 0"

##ALL OF THE ABOVE CAN BE COMBINED WITH json to filter specific log keys

###All logs from modules in a folder at a given log level, only displaying module and msg keys: LOG_LEVEL_STDOUT={trace|info|warn|error|fatal} npm run bdd-whitelist | bunyan -c "this.module.indexOf('lib/workers') === 0" -o json --strict | json -ga module msg

###Example 1

All logs with level trace and higher. Only display module and msg fields.

#####LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -o json --strict | json -ga module msg

/w/api git:SAN-2271-WORKERS-10-stop-instance-container ❯❯❯ LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -o json --strict | json -ga module msg
lib/models/apis/docker.js cannot load certificates for docker!!
lib/models/rabbitmq/index.js RabbitMQ constructor
lib/models/rabbitmq/index.js RabbitMQ constructor
Warning: connect-redis expects a number for the "db" option
cannot load certificates for docker!! ENOENT, no such file or directory '/opt/ssl/docker/ca.pem'
lib/server.js Server constructor
app.js start
lib/server.js Server.prototype.start
lib/models/rabbitmq/index.js RabbitMQ.prototype.connect
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers create-image-builder-container
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers create-instance-container
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers delete-instance-container
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers on-dock-removed
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers on-image-builder-container-create
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers on-image-builder-container-die
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers on-instance-container-create
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers on-instance-container-die
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers on-instance-container-start
lib/models/rabbitmq/index.js RabbitMQ.prototype.loadWorkers start-instance-container
app.js API started
lib/workers/on-instance-container-start.js OnInstanceContainerStartWorker module.exports.worker
lib/workers/on-instance-container-start.js OnInstanceContainerStartWorker constructor
lib/workers/base-worker.js BaseWorker constructor

###Example 2

All logs from lib/workers/on-instance-container-start.js with level trace or higher. Only display module and msg fields.

#####LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -c "this.module.indexOf('lib/workers/on-instance-container-start.js') === 0" -o json --strict | json -ga module msg

/w/api git:SAN-2271-WORKERS-10-stop-instance-container ❯❯❯ LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -c "this.module === 'lib/workers/on-instance-container-start.js'" -o json --strict | json -ga module msg
Warning: connect-redis expects a number for the "db" option
cannot load certificates for docker!! ENOENT, no such file or directory '/opt/ssl/docker/ca.pem'
express deprecated res.send(status): Use res.sendStatus(status) instead lib/middlewares/domains.js:40:18
express deprecated res.send(status, body): Use res.status(status).send(body) instead lib/middlewares/domains.js:40:18
express deprecated res.json(status, obj): Use res.status(status).json(obj) instead node_modules/dat-middleware/lib/res.js:11:19
lib/workers/on-instance-container-start.js OnInstanceContainerStartWorker module.exports.worker
lib/workers/on-instance-container-start.js OnInstanceContainerStartWorker constructor
lib/workers/on-instance-container-start.js OnInstanceContainerStartWorker.prototype.handle
lib/workers/on-instance-container-start.js OnInstanceContainerStartWorker.prototype._attachContainerToNetwork
lib/workers/on-instance-container-start.js OnInstanceContainerStartWorker attachContainerToNetwork async.series success
lib/workers/on-instance-container-start.js OnInstanceContainerStartWorker.prototype._updateInstance
lib/workers/on-instance-container-start.js _updateInstance: modifyContainerInspect final success

###Example 3

Custom tabular display delimiter

#####LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -o json --strict | json -d ' --|-- ' -ga module msg

/w/api git:readme-link-logging-guide ❯❯❯ LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -o json --strict | json -d ' --|-- ' -ga module msg
lib/models/apis/docker.js --|-- cannot load certificates for docker!!
lib/models/rabbitmq/index.js --|-- RabbitMQ constructor
lib/models/rabbitmq/index.js --|-- RabbitMQ constructor
Warning: connect-redis expects a number for the "db" option
cannot load certificates for docker!! ENOENT, no such file or directory '/opt/ssl/docker/ca.pem'
lib/server.js --|-- Server constructor
app.js --|-- start
lib/server.js --|-- Server.prototype.start
lib/models/rabbitmq/index.js --|-- RabbitMQ.prototype.connect
lib/models/rabbitmq/index.js --|-- RabbitMQ.prototype.loadWorkers
lib/models/rabbitmq/index.js --|-- RabbitMQ.prototype.loadWorkers create-image-builder-container
lib/models/rabbitmq/index.js --|-- RabbitMQ.prototype.loadWorkers create-instance-container
lib/models/rabbitmq/index.js --|-- RabbitMQ.prototype.loadWorkers delete-instance-container
lib/models/rabbitmq/index.js --|-- RabbitMQ.prototype.loadWorkers on-dock-removed
lib/models/rabbitmq/index.js --|-- RabbitMQ.prototype.loadWorkers on-image-builder-container-create
lib/models/rabbitmq/index.js --|-- RabbitMQ.prototype.loadWorkers on-image-builder-container-die
lib/models/rabbitmq/index.js --|-- RabbitMQ.prototype.loadWorkers on-instance-container-create
lib/models/rabbitmq/index.js --|-- RabbitMQ.prototype.loadWorkers on-instance-container-die

###Example 4

Execute code on bunyan output json w/ json -e option

#####LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -o json --strict | json -d ' |-- ' -e "this.module = this.module.split('/')[this.module.split('/').length-1]+'\n'" -ga module msg

/w/api git:readme-link-logging-guide ❯❯❯ LOG_LEVEL_STDOUT=trace npm run bdd-whitelist | bunyan -o json --strict | json -d '  |-- ' -e "this.module = this.module.split('/')[this.module.split('/').length-1]+'\n'" -ga module msg
docker.js
  |-- cannot load certificates for docker!!
index.js
  |-- RabbitMQ constructor
index.js
  |-- RabbitMQ constructor
Warning: connect-redis expects a number for the "db" option
cannot load certificates for docker!! ENOENT, no such file or directory '/opt/ssl/docker/ca.pem'
server.js
  |-- Server constructor
app.js
  |-- start
server.js
  |-- Server.prototype.start
index.js
  |-- RabbitMQ.prototype.connect
index.js
  |-- RabbitMQ.prototype.loadWorkers
index.js
  |-- RabbitMQ.prototype.loadWorkers create-image-builder-container
index.js
  |-- RabbitMQ.prototype.loadWorkers create-instance-container
index.js
  |-- RabbitMQ.prototype.loadWorkers delete-instance-container
index.js
  |-- RabbitMQ.prototype.loadWorkers on-dock-removed
index.js
  |-- RabbitMQ.prototype.loadWorkers on-image-builder-container-create
index.js
  |-- RabbitMQ.prototype.loadWorkers on-image-builder-container-die
index.js
  |-- RabbitMQ.prototype.loadWorkers on-instance-container-create
index.js
  |-- RabbitMQ.prototype.loadWorkers on-instance-container-die
index.js
  |-- RabbitMQ.prototype.loadWorkers on-instance-container-start
index.js
  |-- RabbitMQ.prototype.loadWorkers start-instance-container
app.js
  |-- API started

Guide to Development CLI Log Tools

Note, all below examples are logs produced from running tests. To get the same easy-to-read logs from

Give gif ~20 seconds to display relevant example:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally