Agent job refactor

[boukeas]

Description

This PR is a refactoring of the main Testflinger agent loop, i.e. the TestflingerAgent.process_jobs method which controls how an agent picks up a job and controls the execution of its individual phases.

A lot of functionality has been abstracted away from the TestflingerAgent.process_jobs method into the TestflingerJob class and the newly introduced JobPhase class and its derived classes.

Note to reviewers

The structural changes are significant and focusing narrowly on the diff in order to review this sort of refactoring would be disorientating and make little sense. Please read and review TestflingerAgent.process_jobs and the job.py module in their entirety.

Resolved issues

Documentation

Web service API changes

Tests

[plars]

This is one spot that's been begging for a refactor for a while now. I really like your idea and where you are taking it, thanks for taking this on!
I know it's still draft and some obvious things. I was actually able to get it running through some phases in some local testing. I just set up some fake commands for it to run and had it attach to a server I was running in docker locally, then submitted a job to that server for it to pick up. I did find one corner case because it was a new agent and didn't have the directory structure in place for (results, logs, run...). There's an os.makedirs() you removed that I think we'll either need to keep, or handle somewhere to ensure that run dir exists before we try to drop the job file into it.
Looking good so far!

[plars]

Not really done testing this, but wanted to get started on it... I did see this so far. Because it never forces creation of this dir, if it's missing, then it'll result in problems later:

[24-10-08 19:01:02]   ERROR: (cmd.py:32)| cannot access local variable 'job' where it is not associated with a value
Traceback (most recent call last):
  File "/home/plars/work/cert/testflinger/agent/testflinger_agent/cmd.py", line 27, in main
    start_agent()
  File "/home/plars/work/cert/testflinger/agent/testflinger_agent/__init__.py", line 139, in start_agent
    agent.process_jobs()
  File "/home/plars/work/cert/testflinger/agent/testflinger_agent/agent.py", line 159, in process_jobs
    if job.go(phase):
       ^^^
UnboundLocalError: cannot access local variable 'job' where it is not associated with a value

[boukeas]

Because it never forces creation of this dir, if it's missing

The rundir is created here, when the job object is __init__ed.

That error is raised in the finally clause because the job object was never properly created so there should be an indication in the agent log about what the actual error was. I just don't think it's the rundir not being created.