In process Client API Executor Part 1

[chesterxgchen]

Description

The Client API currently only support running the training code in sub-process. This works great for cases where 3rd party needs to run in a command line.
For many data scientist users, where running training directly in-process is also desired as it will be easier for debugging in IDE debugger. Also, the in-process training code will be a lot easier both in configuration and setup avoid the communication issues.

The proposal

This PR proposed an additional alternative in-memory implementation of the executor and Client API.

• The Client API stays the same, nothing changed. (The configuration is different)
• To call user training code directly, we requires uses provide a function method that can be called directly, such as run(), or train() or sub_main(**kwargs). User can specify the function name.
  user will need to specify the path as 'module.function_name' to indicate the call function.
  For example

 fn_path = "cifar10_fl.sub_main"

which indicate the function name is "sub_main" located in cifar10_fl.py module

User Experience

The user experience is the same as previous sub-process case. The code change from ML/DL to FL stays identical.
The configuration changes to indicate the Memory Executor

 executor {
        path = "nvflare.app_opt.pt.client_api_mem_executor.PTClientAPIMemExecutor"
        args {
                    task_fn_path = "cifar10_fl.sub_main
                    task_fn_args { 
                         batch_size = 6
                         dataset_path = "/tmp/nvflare/data/cifar10"
                         num_workers = 2
                   }
              }
              params_transfer_type = "DIFF"
              train_with_evaluation = true
              result_pull_interval = 0.5
              log_pull_interval = 0.1
        }

All the pipe and responding component configurations are not needed.

The implementation

• main approach:
  a new InProcessClientAPIExecutor is implemented, We leverage the new DataBus and EventManager. The main thread of the executor will subscribe to both LOCAL_RESULT and LOG_DATA topics to receive callbacks when results are available.
  The client API will also register a Global_RESULT topic callback to receive global model.

we use one thread is running the Client's training code,
The executor main thread is sending global model once the model shareable is received and waiting for the local model update.

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Quick tests passed locally by running ./runtest.sh.
• In-line docstrings updated.
• Documentation updated.

[yanchengnv]

This PR is too big for this stage of 2.4. We could consider it for 2.4.1.

[chesterxgchen]

/build

[chesterxgchen]

/build

[chesterxgchen]

To reduce the PR size, I split the PR into three parts

1. message bus ( changes are still in this PR), but its duplicate of[ messagebus] (DataBus #2285)
   The message bus PR is independent of client API PR. once it merged, this PR can rebase, the PR file size will be smaller (28 -8 = 20 files, which includes 8 test related files. The real code changes besides tests are 12 files)
2. The main client API changes Part 1, this PR.
3. The Job Templates and related example changes will be Part 2 of Client API

[SYangster]

Cleanup, add additional support for main function with cli args.

Will add and enhance examples in follow up PR.

[SYangster]

/build

[YuanTingHsieh]

Mostly good, we just need to make changes following recent PRs in main regarding api.py

[YuanTingHsieh]

Mostly good, minor comments

[YuanTingHsieh]

Thanks, LGTM, can think about these logging levels later

[SYangster]

/build