Enrich logging

[YujiOshima]

Currently, the logs of Katib components are so poor.
We need enrich logs in the debug, verbose, and info level.
Maybe we should introduce glog pkg.

[gaocegege]

We prefer logrus in the community, I think. Since it supports structured logging.

[jlewi]

@YujiOshima can we make this issue more specific this will make it easier to get help from the community?

Here are some questions for you:

1. Which microservices within Katib should we start logging using logrus
2. What metadata should be included in log entries?
   • In logrus log entries are json records with key-value pairs for metadata
   • So we can attach metadata that is useful for filtering the logs; eg study id
3. Are there metrics that should be exported e.g. via promtheus to allow better monitoring of Katib?

[YujiOshima]

Logging Component:

• vizier-core
• SuggestionService
  • Random
  • Grid
  • HyperBand
• EarlyStoppping
  • Median

BayseOpt Suggestion is implemented with python. I don't know which log liberally is better for python.

Metadata:
StudyID, TrialID, WorkerID or System
System means the error is not related any study, trial or worker. It is occurred around the component itself.

Integration with Prometheus sounds nice. It enable monitoring across languages.

@gaocegege @jlewi BTW, this library https://github.com/uber-go/zap looks nice since it is faster than logrus. WDYT?

[gaocegege]

I love zap, while we are using logrus now. I think we could use zap in katib if you like as a feasibility research.

[YujiOshima]

@gaocegege Thanks! Lets's try zap for Katib. As zap is also structured with key-value, we can migrate logrus easily when we need.

[gaocegege]

Yeah, I opened an issue here: kubeflow/training-operator#534

I closed it because I think the performance of the logging is not the bottleneck. If the migration is painless, I am glad to use zap.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.