Make logging.yml read by default

[noklam]

Quick summary:

• ✅There are at least one obvious user problem - which is logging get chosen as a tool but has no effect, and the consensus is that we should fix it. - How to enable logging automatically #3446
  • One possible solution is hard coding the path to conf/logging.yml with option to override with KEDRO_LOGGINGI_CONFIG, very similar to be3d28d
• It's unclear whether we should do more for logging, such as adding a new -v flag
• It's mentioned logging in development and logging in production are very different, and users probably have different needs:
  • It was mentioned we may want to de-couple rich as a plugin, pip-uninstallable to deactivate it or at least provide an easy enough way to opt out from it.
• We may consider improving the messaging of logging and people should seek for production grade logging.
• It was mentioned even though there are users need, Kedro can still say NO as we had for many other things.

Antony shared the old user research here in case you miss it at the end: #2281 (comment)

Originally posted by @noklam in #3755 (comment)

Context

Since 0.19, we introduced a few changes:

• Decouple logging.yml from ConfigLoader  #2637
• Add KEDRO_LOGGING_CONFIG env variable to specific where to read logging.yml #2535
• A new kedro project creation flow, i.e. kedro new --tools (with logging)

This create a bad UX because:

• New users expect logging.yml will be used by default
• Existing users migrate from 0.18 -> 0.19 things start working unexpectedly

The scope of ticket is to implement a fix to make sure the default file is read. KEDRO_LOGGING_CONFIG will take priority when it is provided. (See be3d28d for inspiration)

Solution

be3d28d is a good starting point. At the minimal case, the happy path of

• kedro new --tools=log
• kedro run should auotmatically read logging.yml

There are things to consider:

• Should we consider accepting logging* or searching for subfolder?
• We need to consider corner case where logging.yml is at CONF_SOURCE/logging.yml, by default CONF_SOURCE is conf but user has the option to change it. (Not handled in the old PR)
  • You may find that you cannot access CONF_SOURCE because it is not read yet when logging is initialised, does it takes a lot of changes to make it possible? Is there workaround?

[SajidAlamQB]

Completed in #3831