cli wait logs "clearing all items from cache" every 2 seconds

[mashehu]

Description of the bug

This also breaks the loading spinner animation.

Steps to reproduce

use either nf-core launch or nf-core schema build

Expected behaviour

This shouldn't be logged at all

System

Nextflow Installation

• >=1.13.3

Additional context

Probably due to a change of the default logging level of the requests_cache package. Downgrading it to 0.5.2 removes the problem. This should have been fixed since 0.6.1 (nf-core/smrnaseq#74 (comment)), but I still got the error using 0.6.2. Adding log.setLevel(logging.DEBUG) in utils.py, schema.py or launch.py, didn't resolve this problem.

[JWCook]

That message is only shown if there is an explicit call to CachedSession.cache.clear(). So there are two separate things to consider:

1. Configuring which log messages you want to show on the console, and
2. Clearing cached responses

Logging config

There is probably a call to logging.basicConfig() somewhere, which will configure the root logger (and every other logger that isn't configured elsewhere) with a StreamHandler, and that's likely not what you want. If you don't want to see any logging messages from other libraries, the best way to do that is to configure only the logger for this library (logging.getLogger('nf_core')). Alternatively, you can set the logging level for other individual libraries, for example:

logging.getLogger('requests_cache').setLevel('ERROR')

Cache expiration

As for whatever is calling clear() repeatedly, some better options are:

1. Use cache expiration
2. Use a tempfile for the SQLite db (example here)
3. Use a non-persistent cache (CachedSession(backend='memory'))

Depending on how long you want cached items to persist. If you or someone else can describe the caching behavior you want, I can provide some examples.

[ewels]

Thanks for the help @JWCook !

Alternatively, you can set the logging level for other individual libraries, for example:

logging.getLogger('requests_cache').setLevel('ERROR')

Yes this is what we have done in other instances and what I was planning to do here. I quite like having the logs from other libraries in the verbose log as it helps for debugging sometimes.

The cache expiration thing is trickier - for some (most?) API calls we want the cache to be pretty long as it speeds up execution significantly. However there are some calls which should always fetch fresh data. I suspect that we are clearing the entire cache in these cases when we should instead be instead using with requests_cache.disabled().

Checking the code now I suspect that there is actually only one instance in the code where this is set up incorrectly, here:

tools/nf_core/utils.py

Lines 327 to 328 in d0a6649

	# Clear requests_cache so that we get the updated statuses
	requests_cache.clear()

The one in the sync code is a bit of a special case and I think it is probably safer to wipe the entire cache there.