Augur log data oversaturation

[cdolfi]

It can be hard at times to debug because the logs can not be kept for long as 1000s of jobs generate 100s of lines of stack trace, all at once after a potential useful error log (like index corruption messages) has passed. There might be a design problem/change that would help the useful logs be able to stick around longer. @GregSutcliffe tagged to provide more context and lmk if I should edit this description

[Ulincsys]

Hi Cali,

The primary log stream receives the combined output of every logging interface within Augur, so it does go very quickly on an active instance.

However, in the logs/ directory (or elsewhere, if configured otherwise) each logging unit should have its own log file in the form of logging_unit.info and logging_unit.error for separate stdout/stderr logs, or logging_unit.out for a combined log stream.

If you are running in a docker container, then you'll need to mount logs/ as a volume (which I believe that @GregSutcliffe has mentioned doing previously), or you can access the container via an interactive shell with

docker exec -it <container_name> bash

[sgoggins]

I think setting the LOGLEVEL to ERROR in the augur_operations.config table will also reduce logging to contain something more manageable and expected in a production instance.

[cdolfi]

@GregSutcliffe Do you know what our logging level is?