Supported/recommended libraries in Node.js

[tsimbalar]

Hi,

After years of .NET, I'm now working in a JS/TypeScript environment, with mostly node services running on the backend. I'd like to spread the structured-logging love among my colleagues, and am going to do a small presentation about the concepts. I'm hoping to show off Seq in that context as well ;)

I know that Serilog is definitely the best option in the .NET world, but I was wondering what would be the recommended combination of tools and libraries to use in order to have the best experience in the full-stack Javascript world.

Our current stack contains :

• some web apps developed with Angular
• some mobile apps developed with Cordova / Ionic
• some backend services developed in Node (+Mongo Db) , hosted through docker

From the Seq docs, I see examples about using bunyan, but it looks like the project is sort of dying : trentm/node-bunyan#586 :-/

pino and winston seem to be the winners in the node.js world, but none of them seem to support the notion of Message Template / Message Type that I have grown to love in Serilog.

I'm considering using one of them.

I've read that the recommended way to do log-shipping in Node is to do it out of process, for instance by piping the STDOUT to another process that will take care of forwarding it. I see that seqcli supports this sort of scenario. Is that recommended in a Production environment, or is this more targetted for scripting or developer use ?

Anyway, any recommendation you would have, or feedback from customers who have used Seq in such a set up would be very welcome :)

Thanks and keep on rocking ! ♦️

[nblumhardt]

Hey @tsimbalar! I'm hoping others can chime in with their own experiences, but to throw one more option out there - structured-log is the evolution of a project originally called "serilog.js", and it has message templates + a Seq sink; on the minus side, it's not widely-used... but since it's a fairly small codebase, and given your experience with Serilog, I don't doubt you could get this working to your satisfaction (the project could probably use your input :-))

Pino JSON -> STDOUT -> seqcli ingest --json seems like a decent option; you might need a jq step in there to transform the timestamp, though I'd be interested to see what we could add to seqcli to make this work naturally.

Winston has a couple of GELF/UDP output options, which could be used easily with Seq.Inputs.Gelf.

HTH!

[andymac4182]

We use structured-log writing JSON to STDOUT then picked up and sent to Seq in lambda with node which has been working well for about 12 months. I would use the seqcli ingest option personally.

[nblumhardt]

Another option I forgot about - the Bunyan support is just a thin wrapper around a framework-independent library that we ship called seq-logging. The goal of this package is to make it (trivially) easy to wire up other loggers to Seq; does that unblock any options @tsimbalar ?

[nblumhardt]

(If there's a particular library you think is best, let me know - would be open to adopting a new preferred/supported option, if there's a good one.)

[tsimbalar]

Hi ,
sorry, I have been wanting to reply since forever, but I've been quite busy.

I have nothing concrete yet, but here is what I am planning to try out, with each step adding on top of each other ...

I'm going to take a look at pino as a logging library. It seems to be quite active and quite ok, and does support some sort of structured logging (it basically outputs newline-delimited JSON).

It advocates generating JSON-formatted logs to the std output and piping it to other tools for "log-shipping" and/or formatting. I think it makes a lot of sense for Node which does not have threads to do background-processing.

Step 1 - publish logs to Seq

• use pino in my node web app (logger.info({prop1:'value1'}, 'This is my message'))
• output logs to std output in its row JSON format
• configure docker to use the gelf driver and forward to Seq (as detailed in here)

At that point, I understand I should have all my logs and enrichment showing up in Seq (but no Message Templates yet).

My app should be running as node myapp.js in prod, and as node myapp.js | pino-pretty in development.

Step 2 - publish to Seq in Compact Log Event Format

• start using message templates in the code base (logger.info({prop1:'value1', user: {id: userId, name: userName}}, 'This is a message to say that {prop1} is the current value for {@user}'))
• write a small Pino transport (pino-clef ?) that will slightly transform the JSON to add CLEF special properties (turn the default msg field into @mt, add @x to contain the error.stack if any, convert pino's default level to a proper @l)

Hopefully, unless I'm missing something, I'll get Seq working quite well in that set-up.

I understand that I don't need to provide @i (event id - can be inferred from @mt ?), @r (renderings), @m (can be inferred from @mt ?), @t (provided by the Seq/GELF adapter) and it should work just fine. Please correct me if I'm wrong :)

At that point, I should have something that works quite seamlessly for production-like environments (node myapp.js | pino-clef).

In a development scenario though, I guess I would still be seeing some rather inhumane raw JSON outputs, or can pipe it to pino-pretty, the "out of the box" prettifier for development .... but then I'll be seeing Message Templates ... which will lead us to ...

Step 3 - format CLEF log events to a nicer user-friendly output when developing
I'd like to see a nicer user-friendly output for my logs, ideally mimicking Serilog.Sinks.Console in .NET, with nice highlighting of the message template tokens ...

To do this, I would need create and use a new process in the pipeline that would :

• take the output of pino-clef
• turn that into nice colorized output, by talking the @mt and the attached properties and display something nice.

This could be called something like pino-pretty-clef.

In order to do that I was expecting to take some inspiration/dependency from :

• https://github.com/structured-log/structured-log which seems to already have an implementation of rendering a string from a message template + properties
• https://github.com/pinojs/pino-pretty for the piping / colorization

That seems like bit of work, but nothing completely crazy, and reaching Step 2 would already be nice enough.

@nblumhardt @KodrAus I don't suppose you have any deep knowledge about pino, but do you think that the plan more or less makes sense, and that I am not making any invalid assumptions about how Seq ingests CLEF events and how seq-GELF processes then ? Also maybe you know of existing tools that I could reuse in the OSS ecosystem ...

Thanks !

[nblumhardt]

👍 sounds good to me, @tsimbalar

Keep us posted, keen to stay in the loop and help anywhere things don't quite line up :-)

[tsimbalar]

FYI, here is my current progress, in the shape of a gist :
https://gist.github.com/tsimbalar/f09358048d3d50ffb17bc4191cf2e7fd

[nblumhardt]

@tsimbalar would be great to hear how this all turned out; did you end up using pino-clef etc?

[tsimbalar]

Hi Nick,

sorry I haven't followed up much. At this point, I haven't gone further than the gists I have posted.

It turns out that we already have a Kibana running somewhere in our infrastructure, so this might be what we we'll go towards ... I'm still pretty convinced that having a Seq instance running locally is really the best way to be sure the logs we produce are nice and clean, and are not totally useless once running in production.

We should be doing another round of experiments in the following month, I'll try and keep you posted ;)

[nblumhardt]

Cool, thanks for the update Thibaud! Good luck with it all 👍

[nblumhardt]

Just looping back to this; we're now maintaining https://github.com/datalust/pino-seq, which should provide another widely-available option. I'll close this ticket, but any experience reports are welcome :-)