MissingParameterException when generating execution summary

[mcarlsen]

Generating the execution summary may generate a MissingParameterException if a requires attempts to access a Config parameter which was specified only on the command line.

Example:

# trouble.py
import luigi

class conf(luigi.Config):
    opt = luigi.Parameter()

class Root(luigi.WrapperTask):
    def requires(self):
        if conf().opt == 'A':
            return A()

class A(luigi.Task):
    def complete(self):
        return False

    def run(self):
        raise Exception()

$ luigi --module trouble Root --conf-opt=A
===== Luigi Execution Summary =====

Scheduled 2 tasks of which:
* 1 failed:
    - 1 A()
* 1 were left pending, among these:
    * 1 had failed dependencies:
        - 1 Root()

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

Traceback (most recent call last):
  File "/opt/etl-virtualenv/bin/luigi", line 11, in <module>
    sys.exit(luigi_run())
  File "/opt/etl-virtualenv/lib/python3.6/site-packages/luigi/cmdline.py", line 11, in luigi_run
    run_with_retcodes(argv)
  File "/opt/etl-virtualenv/lib/python3.6/site-packages/luigi/retcodes.py", line 83, in run_with_retcodes
    task_sets = luigi.execution_summary._summary_dict(worker)
  File "/opt/etl-virtualenv/lib/python3.6/site-packages/luigi/execution_summary.py", line 337, in _summary_dict
    _populate_unknown_statuses(set_tasks)
  File "/opt/etl-virtualenv/lib/python3.6/site-packages/luigi/execution_summary.py", line 75, in _populate_unknown_statuses
    _depth_first_search(set_tasks, task, visited)
  File "/opt/etl-virtualenv/lib/python3.6/site-packages/luigi/execution_summary.py", line 88, in _depth_first_search
    for task in current_task._requires():
  File "/opt/etl-virtualenv/lib/python3.6/site-packages/luigi/task.py", line 607, in _requires
    return flatten(self.requires())  # base impl
  File "/private/tmp/trouble.py", line 8, in requires
    if conf().opt == 'A':
  File "/opt/etl-virtualenv/lib/python3.6/site-packages/luigi/task_register.py", line 88, in __call__
    param_values = cls.get_param_values(params, args, kwargs)
  File "/opt/etl-virtualenv/lib/python3.6/site-packages/luigi/task.py", line 412, in get_param_values
    raise parameter.MissingParameterException("%s: requires the '%s' parameter to be set" % (exc_desc, param_name))
luigi.parameter.MissingParameterException: conf[args=(), kwargs={}]: requires the 'opt' parameter to be set

This seems to be because the execution summary is generated outside any CmdlineParser context manager in run_with_retcodes. So this should be fairly easy to avoid by extending the entire run_with_retcodes to be within the CmdlineParser cm already there for the retcode() config - or if that could cause side effects I am unaware of then a separate context just for the call to _summary_dict.

I can attempt a PR for either approach.

May be related to #1964

[Tarrasch]

Nice catch. I'm confused by the example run you have pasted. It looks like the execution summary is still printed?

I guess it makes sense to use the same CmdlineParser context managerand put the execution summary generation in there. The damage of global variables it seems.

[mcarlsen]

Yes, the summary is still printed. BTW, the exit code is 1, not the code for task_failed or even unhandled_exception.

I tried to extend the CmdlineParser context manager to the entire run_with_retcodes, but got multiple test failures in retcodes_test.py and cmdline_test.py having to do with expected return codes. So at least some code (or maybe just some test code?) expects the CmdlineParser to be unavailable during some parts of run_with_retcodes.

The PR simply puts the call to _summary_dict inside a CmdlineParser context manager.

[Tarrasch]

Aha, now I know what code path you are talking about! Now it makes sense.