Idea: App and module specific log levels

[Cashmaney]

Summary

Debugging and working with the various stacks of a chain is quite complex - you get a ton of information thrown at you. A lot of the times different stakeholders care about different parts of the chain. A node runner probably cares about which block their node is on, but unless they're debugging they probably don't want to see stuff like the bank module minting tokens, ibc noise, or random peers connecting or disconnecting. A developer debugging their module will care only about output from their module. Getting more logging information from one part of the system currently isn't possible without eating the spam from all the other modules as well as tendermint logs. We encountered this difficulty while debugging a chain halt a few days ago.

I think it makes much more sense to decouple the loggers for the SDK and Tendermint, and enable the option to configure different log levels for specific modules.

Proposal

I threw together a PoC implementation of the above, which we might end up using for Secret Network. I want to put this here as well to hear feedback on general idea, and how it all connected together. I'm sure there are many considerations that I might not be thinking of. For instance, I couldn't see a way to configure module log levels without (albeit minor) changes to the modules themselves, so they have to be aware of this functionality.

Building on this, I'd also like to add the options to log to a file instead of just stdout/err and configuring log levels from the config. Happy to include any other suggestions here as well.

Here are the PRs I made to get it working:

On the SDK side:
Separate app and sdk log levels: scrtlabs#199
Module specific log levels: scrtlabs#200

Plugging it together on the chain:
scrtlabs/SecretNetwork#1180

[tac0turtle]

We added filter logging back to the sdk here #13236. We can explore decoupling from tendermint as well if we want but in the next release users can watch only p2p:info,state:info while developers could look at specific modules.

[alexanderbez]

Filtered logging was re-introduced for exactly this purpose and reason. This should meet the needs of the issues you've describes. Node operators should be able to filter based on the contexts they're interested in.

If you feel that filtered logging will not solve this for whatever reason, please note them. Otherwise we can safely close this issue. Also, please feel free to play around with filtered logging and see if it meets your needs too.

[Cashmaney]

Oh cool, I didn't know this was a thing. Filtering at the logger level would def make integrating it easier. I'll look into playing around with it 👍