Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Any way to return "msg" (in prod) as JSON? #132

Closed
armenr opened this issue Nov 13, 2019 · 6 comments
Closed

Any way to return "msg" (in prod) as JSON? #132

armenr opened this issue Nov 13, 2019 · 6 comments

Comments

@armenr
Copy link

armenr commented Nov 13, 2019

Hey @niftylettuce!

We're using DataDog, and it's friendly to "natural JSON" a la pino.

I've read the documentation and understand that whatever is returned @ the msg property has to be a string, and that it complies with the apache-friendly log RFCs, because you're a man of taste and experience.

BUT, is there any way to get JSON back? Or maybe have the choice between apache-friendly and JSON?

Thanks for all your work, man!
_Armen

@niftylettuce
Copy link
Contributor

Can you give me an example / code snippet of what you're trying to do and the expected vs. actual output? I would definitely accept a PR for sure man or fix this myself for you. Just need to know specifically what you mean

@armenr
Copy link
Author

armenr commented Nov 13, 2019

Thanks dude!

Example prod log entry sent to DataDog:

{
   "level":30,
   "time":1573656121209,
   "pid":12011,
   "hostname":"ip-10-0-12-50",
   "msg":"0.0.0.0 - - 13/Nov/2019:14:42:01 +0000 \"GET /static/favicon.ico HTTP/1.1\" 200 - {\"id\":\"5dcc163992279d2eeb055564\",\"timestamp\":\"2019-11-13T14:42:01.000Z\",\"request\":{\"method\":\"GET\",\"query\":{},\"headers\":{\"accept\":\"image/webp,image/apng,image/*,*/*;q=0.8\",\"accept-encoding\":\"gzip, deflate\",\"accept-language\":\"en-US,en;q=0.9\",\"cache-control\":\"no-cache\",\"connection\":\"keep-alive\",\"cookie\":\"dtCookie=v_4_srv_1_sn_83AA82EA4B50C9EE45C851779D892DC8_perc_100000_ol_0_mul_1\",\"host\":\"1.1.1.1:3000\",\"pragma\":\"no-cache\",\"referer\":\"http://1.1.1.1:3000/\",\"user-agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36\",\"x-dynatrace-application\":\"1%3B1%3Bea7c4b59f27d43eb\",\"x-dynatrace-origin-url\":\"http://1.1.1.1:3000/static/favicon.ico\",\"username\":\"NoAuth\"},\"cookies\":{\"dtCookie\":\"v_4_srv_1_sn_83AA82EA4B50C9EE45C851779D892DC8_perc_100000_ol_0_mul_1\"},\"url\":\"/static/favicon.ico\",\"timestamp\":\"2019-11-13T14:42:01.205Z\",\"id\":\"5d5ac094-9cfb-4789-abb6-cfc8178f85ad\",\"http_version\":\"1.1\"},\"user\":{\"ip_address\":\"0.0.0.0\"},\"response\":{\"headers\":{\"x-powered-by\":\"Good Vibes 1.0\",\"access-control-allow-origin\":\"*\",\"surrogate-control\":\"no-store\",\"cache-control\":\"no-store, no-cache, must-revalidate, proxy-revalidate\",\"pragma\":\"no-cache\",\"expires\":\"0\",\"x-content-type-options\":\"nosniff\",\"x-frame-options\":\"SAMEORIGIN\",\"x-xss-protection\":\"1; mode=block\",\"x-request-id\":\"5d5ac094-9cfb-4789-abb6-cfc8178f85ad\",\"x-request-username\":\"NoAuth\",\"accept-ranges\":\"bytes\",\"last-modified\":\"Mon, 11 Nov 2019 21:07:42 GMT\",\"etag\":\"W/\\\"1a47-16e5c4b3130\\\"\",\"content-type\":\"image/x-icon\",\"x-response-time\":\"1.256ms\",\"vary\":\"Accept-Encoding\",\"content-encoding\":\"gzip\",\"date\":\"Wed, 13 Nov 2019 14:42:01 GMT\",\"connection\":\"keep-alive\",\"transfer-encoding\":\"chunked\"},\"http_version\":\"1.1\",\"status_code\":200,\"reason_phrase\":\"OK\",\"timestamp\":\"2019-11-13T14:42:01.000Z\",\"duration\":1.256},\"duration\":0.534569,\"app\":{\"node\":\"v12.13.0\",\"environment\":\"production\",\"hostname\":\"ip-10-0-12-50\",\"pid\":12011}}",
   "v":1
}

Pino + CabinJS thangz:


const pinoProd = pino({
  customLevels: {
    log: 30,
  },
});

const pinoDev = pino({
  prettyPrint: {
    colorize: true,
  },
});

const cabin = new Cabin({
  capture: false,
  axe: {
    logger: NODE_ENV === 'production' ? pinoProd : pinoDev,
  },
});

Desired behavior:

In the log output, you'll notice that everything at the "msg" value is stringified JSON.

I'm wondering if there's a way to return all of this - in prod - as pure and total JSON, since tools like DataDog and GrayLog are friendly to JSON, and would automatically detect and extract the key/value pairs as labels/fields.

I know it's absolutely ANTITHETICAL to the concept of the apache log standards, but...I'm shamelessly asking :)

@niftylettuce
Copy link
Contributor

Have you tried specifying a custom message function? Not sure if that would fix it for you - let me know otherwise.

https://github.com/cabinjs/cabin#options
https://github.com/cabinjs/cabin/blob/master/src/message.js

@armenr
Copy link
Author

armenr commented Nov 13, 2019

I've read through that - not to be a nag or a baby - but I'm just trying to figure out (maybe with an example) what a custom function would look like?

I'm actually more confused now, since the code I see is this:

function apacheCommonLogFormat(options) {
  const { req, res, ctx } = options;

  const creds = auth(req);

  const startTime = getStartTime(req);

  return `${ctx ? ctx.ip : req.ip} - ${creds ? creds.name : '-'} ${clfDate(
    startTime
  )} "${req.method} ${req.url} HTTP/${req.httpVersionMajor}.${
    req.httpVersionMinor
  }" ${res.statusCode} ${res.getHeader('content-length') || '-'}`;
}

but it looks like I'm getting WAY more than that in my logs to DataDog...sorry for sounding illiterate.

@niftylettuce
Copy link
Contributor

niftylettuce commented Nov 13, 2019

That code returns a string as the message. If you want an object, you could do something like:

function message(options) {
  const { req, res, ctx } = options;
  const creds = auth(req);
  const startTime = getStartTime(req);

  return {
    ip: ctx ? ctx.ip : req.ip,
    creds: creds ? creds.name : '-',
    clf_date: clfDate(startTime),
    method: req.method,
    url: req.url,
    http_version: req.httpVersionMajor + '.' + req.httpVersionMinor,
    status_code: res.statusCode,
    content_length: res.getHeader('content-length') || '-'
  };
}

// ...

const cabin = new Cabin({
  capture: false,
  axe: {
    logger: NODE_ENV === 'production' ? pinoProd : pinoDev,
  },
  message // <--
});

@niftylettuce
Copy link
Contributor

Pino itself might be stringifying the message, I am unsure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants