add FedBN Implementation on NVFlare research folder - a local batch normalization federated learning method #2524

MinghuiChen43 · 2024-04-20T09:52:04Z

Fixes # .

Description

Implementation of FedBN (https://arxiv.org/pdf/2102.07623.pdf) - local batch normalization with NVFlare

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Quick tests passed locally by running ./runtest.sh.
In-line docstrings updated.
Documentation updated.

chesterxgchen · 2024-04-23T15:32:33Z

can you add description so we know what this PR is about ?

research/fedbn/custom/wf_controller.py

research/fedbn/custom/figs/plot_requirements.txt

research/fedbn/custom/figs/plot_tensorboard_events.py

research/fedbn/custom/fl_bn_model_utils.py

research/fedbn/custom/pt/learners/__init__.py

research/fedbn/cifar10_sim_fedbn/run_cifar10_sim_fedbn.sh

research/fedbn/cifar10_sim_fedbn/run_simulator.sh

research/fedbn/cifar10_sim_fedbn/set_alpha.py

research/fedbn/custom/app_constant.py

research/fedbn/custom/base_fedbn.py

research/fedbn/custom/fedbn.py

research/fedbn/custom/pt/learners/cifar10_learner.py

research/fedbn/custom/pt/learners/cifar10_model_learner.py

research/fedbn/custom/pt/learners/cifar10_scaffold_learner.py

research/fedbn/custom/pt/learners/cifar10_scaffold_model_learner.py

chesterxgchen

This PR request quite some of changes.
It seems to be the author simply copied some of the examples and put all the files into this PR, even most of them are not even used or needed.

I think it might be better to reference to ML2FL or step-by-step/cifar10 examples for the Client side training using Client API and the server side use BaseFedAvg as base.

Also leverage nvflare job CLI to update job config rather write python files.

delete un-used files, and avoid copy the files in FLARE src files.

Here is a PR using Client API on the client side, and BaseFedAvg API on the server side you can probably reference :

#2529

ZiyueXu77 · 2024-05-09T13:09:36Z

/build

ZiyueXu77 · 2024-05-09T13:52:55Z

Hi @MinghuiChen43 could you try signing your commits (setting up some keys etc. in your local machine)? We have this requirement for our repo. See instruction below:

The base branch requires all commits to be signed. Learn more about signing commits.

Thanks!

* clean up to allow creation of nvflare light * move defs to cellnet

MinghuiChen43 · 2024-05-10T11:44:19Z

Hi @MinghuiChen43 could you try signing your commits (setting up some keys etc. in your local machine)? We have this requirement for our repo. See instruction below:

The base branch requires all commits to be signed. Learn more about signing commits.

Thanks!

I have signed my recent commits. Do I need to have all my commits verified? It might be difficult to go back and sign the previous commits.

ZiyueXu77 · 2024-05-10T13:08:27Z

/build

Hi @MinghuiChen43 could you try signing your commits (setting up some keys etc. in your local machine)? We have this requirement for our repo. See instruction below:
The base branch requires all commits to be signed. Learn more about signing commits.
Thanks!

I have signed my recent commits. Do I need to have all my commits verified? It might be difficult to go back and sign the previous commits.

Yes we do need to have all signed, I think we can squash the commits in that case, @YuanTingHsieh any suggestions on this?

MinghuiChen43 · 2024-05-10T16:43:02Z

/build

Hi @MinghuiChen43 could you try signing your commits (setting up some keys etc. in your local machine)? We have this requirement for our repo. See instruction below:
The base branch requires all commits to be signed. Learn more about signing commits.
Thanks!

I have signed my recent commits. Do I need to have all my commits verified? It might be difficult to go back and sign the previous commits.

Yes we do need to have all signed, I think we can squash the commits in that case, @YuanTingHsieh any suggestions on this?

I used the command git rebase --exec 'git commit --amend --no-edit -n -S' -i xxx to sign old commits, as suggested in an answer I found (https://superuser.com/questions/397149/can-you-gpg-sign-old-commits). Every commit seemed to be verified successfully, but I failed to verify the initial commit. I'm wondering if there is anything inappropriate about my approach and how I can fix the issue with verifying the initial commit.

ZiyueXu77 · 2024-05-10T16:52:28Z

@chesterxgchen , we have some issue with signing requirements, ccould you merge it?

ZiyueXu77 · 2024-05-10T18:15:21Z

/build

YuanTingHsieh · 2024-05-10T20:06:10Z

@MinghuiChen43 @ZiyueXu77 in this case, we can only force merge, or rebase the whole commit into one and sign that specific one.

But no worries, I just merge it, thanks for the effort!

…ormalization federated learning method (NVIDIA#2524) * add research/fedbn * delete redudant controller and correct figs requirements * update plot_requirements * rewrite fedbn * update jobs * remove workspace * update README * simplify job simulator_run to take only one workspace parameter. (NVIDIA#2528) * Add missing client api test jobs (NVIDIA#2535) * Fixed the simulator server workspace root dir (NVIDIA#2533) * Fixed the simulator server root dir error. * Added unit test for SimulatorRunner start_server_app. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> * Improve InProcessClientAPIExecutor (NVIDIA#2536) * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * make result check and result pull use the same configurable variable * rename exec_task_fn_wrapper to task_script_runner.py * fix typo * FIX MLFLow and Tensorboard Output to be consistent with new Workspace root changes (NVIDIA#2537) * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1. Remove the default code to use configuration 2. fix some broken notebook * rollback changes * Fix decorator issue (NVIDIA#2542) * update create and run job script * FLModel summary (NVIDIA#2544) * add FLModel Summary * format * remove jobs folder * expose aggregate_fn to users for overwriting (NVIDIA#2539) * handle cases where the script with relative path in Script Runner (NVIDIA#2543) * handle cases where the script with relative path * handle cases where the script with relative path * add more unit test cases and change the file search logics * code format * add more unit test cases and change the file search logics * Lr newton raphson (NVIDIA#2529) * Implement federated logistic regression with second-order newton raphson. Update file headers. Update README. Update README. Fix README. Refine README. Update README. Added more logging for the job status changing. (NVIDIA#2480) * Added more logging for the job status changing. * Fixed a logging call error. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Fix update client status (NVIDIA#2508) * check workflow id before updating client status * change order of checks Add user guide on how to deploy to EKS (NVIDIA#2510) * Add user guide on how to deploy to EKS * Address comments Improve dead client handling (NVIDIA#2506) * dev * test dead client cmd * added more info for dead client tracing * remove unused imports * fix unit test * fix test case * address PR comments --------- Co-authored-by: Sean Yang <seany314@gmail.com> Enhance WFController (NVIDIA#2505) * set flmodel variables in basefedavg * make round info optional, fix inproc api bug temporarily disable preflight tests (NVIDIA#2521) Upgrade dependencies (NVIDIA#2516) Use full path for PSI components (NVIDIA#2437) (NVIDIA#2517) Multiple bug fixes from 2.4 (NVIDIA#2518) * [2.4] Support client custom code in simulator (NVIDIA#2447) * Support client custom code in simulator * Fix client custom code * Remove cancel_futures args (NVIDIA#2457) * Fix sub_worker_process shutdown (NVIDIA#2458) * Set GRPC_ENABLE_FORK_SUPPORT to False (NVIDIA#2474) Pythonic job creation (NVIDIA#2483) * WIP: constructed the FedJob. * WIP: server_app josn export. * generate the job app config. * fully functional pythonic job creation. * Added simulator_run for pythonic API. * reformat. * Added filters support for pythonic job creation. * handled the direct import case in fed_job. * refactor. * Added the resource_spec set function for FedJob. * refactored. * Moved the ClientApp and ServerApp into fed_app.py. * Refactored: removed the _FilterDef class. * refactored. * Rename job config classes (NVIDIA#3) * rename config related classes * add client api example * fix metric streaming * add to() routine * Enable obj in the constructor as paramenter. * Added support for the launcher script. * refactored. * reformat. * Update the comment. * re-arrange the package location. * Added add_ext_script() for BaseAppConfig. * codestyle fix. * Removed the client-api-pt example. * removed no used import. * fixed the in_time_accumulate_weighted_aggregator_test.py * Added Enum parameter support. * Added docstring. * Added ability to handle parameters from base class. * Move the parameter data format conversion to the START_RUN event for InProcessClientAPIExecutor. * Added params_exchange_format for PTInProcessClientAPIExecutor. * codestyle fix. * Fixed a custom code folder structure issue. * work for sub-folder custom files. * backed to handle parameters from base classes. * Support folder structure job config. * Added support for flat folder from '.XXX' import. * codestyle fix. * refactored and add docstring. * Address some of the PR reviews. --------- Co-authored-by: Holger Roth <6304754+holgerroth@users.noreply.github.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Enhancements from 2.4 (NVIDIA#2519) * Starts heartbeat after task is pull and before task execution (NVIDIA#2415) * Starts pipe handler heartbeat send/check after task is pull before task execution (NVIDIA#2442) * [2.4] Improve cell pipe timeout handling (NVIDIA#2441) * improve cell pipe timeout handling * improved end and abort handling * improve timeout handling --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * [2.4] Enhance launcher executor (NVIDIA#2433) * Update LauncherExecutor logs and execution setup timeout * Change name * [2.4] Fire and forget for pipe handler control messages (NVIDIA#2413) * Fire and forget for pipe handler control messages * Add default timeout value * fix wait-for-reply (NVIDIA#2478) * Fix pipe handler timeout in task exchanger and launcher executor (NVIDIA#2495) * Fix metric relay pipe handler timeout (NVIDIA#2496) * Rely on launcher check_run_status to pause/resume hb (NVIDIA#2502) Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> --------- Co-authored-by: Yan Cheng <58191769+yanchengnv@users.noreply.github.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Update ci cd from 2.4 (NVIDIA#2520) * Update github actions (NVIDIA#2450) * Fix premerge (NVIDIA#2467) * Fix issues on hello-world TF2 notebook * Fix tf integration test (NVIDIA#2504) * Add client api integration tests --------- Co-authored-by: Isaac Yang <isaacy@nvidia.com> Co-authored-by: Sean Yang <seany314@gmail.com> use controller name for stats (NVIDIA#2522) Simulator workspace re-design (NVIDIA#2492) * Redesign simulator workspace structure. * working, needs clean. * Changed the simulator workspacce structure to be consistent with POC. * Moved the logfile init to start_server_app(). * optimzed. * adjust the stats pool location. * Addressed the PR views. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Simulator end run for all clients (NVIDIA#2514) * Provide an option to run END_RUN for all clients. * Added end_run_all option for simulator to run END_RUN event for all clients. * Fixed a add_argument type, added help message. * Changed to use add_argument(() compatible with python 3.8. * reformat. * rewrite the _end_run_clients() and add docstring for easier understanding. * reformat. * adjusting the locking in the _end_run_clients. * Fixed a potential None pointer error. * renamed the clients_finished_end_run variable. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Sean Yang <seany314@gmail.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Secure XGBoost Integration (NVIDIA#2512) * Updated FOBS readme to add DatumManager, added agrpcs as secure scheme * Refactoring * Refactored the secure version to histogram_based_v2 * Replaced Paillier with a mock encryptor * Added license header * Put mock back * Added metrics_writer back and fixed GRPC error reply simplify job simulator_run to take only one workspace parameter. (NVIDIA#2528) Fix README. Fix file links in README. Fix file links in README. Add comparison between centralized and federated training code. Add missing client api test jobs (NVIDIA#2535) Fixed the simulator server workspace root dir (NVIDIA#2533) * Fixed the simulator server root dir error. * Added unit test for SimulatorRunner start_server_app. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Improve InProcessClientAPIExecutor (NVIDIA#2536) * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * make result check and result pull use the same configurable variable * rename exec_task_fn_wrapper to task_script_runner.py * fix typo Update README for launching python script. Modify tensorboard logdir. Link to environment setup instructions. expose aggregate_fn to users for overwriting (NVIDIA#2539) FIX MLFLow and Tensorboard Output to be consistent with new Workspace root changes (NVIDIA#2537) * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1. Remove the default code to use configuration 2. fix some broken notebook * rollback changes Fix decorator issue (NVIDIA#2542) Remove line number in code link. FLModel summary (NVIDIA#2544) * add FLModel Summary * format formatting Update KM example, add 2-stage solution without HE (NVIDIA#2541) * add KM without HE, update everything * fix license header * fix license header - update year to 2024 * fix format --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> * update license --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Holger Roth <hroth@nvidia.com> * Add information about dig (bind9-dnsutils) in the document * format update * Update KM example, add 2-stage solution without HE (NVIDIA#2541) * add KM without HE, update everything * fix license header * fix license header - update year to 2024 * fix format --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> * Update monai readme to remove logging.conf (NVIDIA#2552) * MONAI mednist example (NVIDIA#2532) * add monai notebook * add training script * update example * update notebook * use job template * call init later * swith back * add gitignore * update notebooks * add readmes * send received model to GPU * use monai tb stats handler * formatting * Improve AWS cloud launch script * Add in process client api tests (NVIDIA#2549) * Add in process client api tests * Fix headers * Fix comments * Add client controller executor (NVIDIA#2530) * add client controller executor * address comments * enhance abort, set peer props * remove asserts --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * Add option in dashboard cli for AWS vpc and subnet * add note on README visualization * update README * update readme * update readme * update readme * [2.5] Clean up to allow creation of nvflare light (NVIDIA#2573) * clean up to allow creation of nvflare light * move defs to cellnet * Enable patch and build for nvflight (NVIDIA#2574) * verified commit --------- Co-authored-by: Yuhong Wen <yuhongw@nvidia.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Sean Yang <seany314@gmail.com> Co-authored-by: Zhijin <zhijinl@nvidia.com> Co-authored-by: Holger Roth <hroth@nvidia.com> Co-authored-by: Isaac Yang <isaacy@nvidia.com> Co-authored-by: Ziyue Xu <ziyue.xu@gmail.com> Co-authored-by: Ziyue Xu <71786575+ZiyueXu77@users.noreply.github.com> Co-authored-by: Holger Roth <6304754+holgerroth@users.noreply.github.com> Co-authored-by: Yan Cheng <58191769+yanchengnv@users.noreply.github.com>

* Updated FOBS readme to add DatumManager, added agrpcs as secure scheme * Implemented horizontal calls in nvflare plugin * Added support for horizontal secure XGBoost * Fixed a few horizontal issues * Added reliable message * Added ReliableMessage parameters * Added log for debugging empty rcv_buf * Added finally block to finish duplicate seq * Removed debug statements * format change * Add in process client api tests (#2549) * Add in process client api tests * Fix headers * Fix comments * Add client controller executor (#2530) * add client controller executor * address comments * enhance abort, set peer props * remove asserts --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * Add option in dashboard cli for AWS vpc and subnet * [2.5] Clean up to allow creation of nvflare light (#2573) * clean up to allow creation of nvflare light * move defs to cellnet * Enable patch and build for nvflight (#2574) * add FedBN Implementation on NVFlare research folder - a local batch normalization federated learning method (#2524) * add research/fedbn * delete redudant controller and correct figs requirements * update plot_requirements * rewrite fedbn * update jobs * remove workspace * update README * simplify job simulator_run to take only one workspace parameter. (#2528) * Add missing client api test jobs (#2535) * Fixed the simulator server workspace root dir (#2533) * Fixed the simulator server root dir error. * Added unit test for SimulatorRunner start_server_app. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> * Improve InProcessClientAPIExecutor (#2536) * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * make result check and result pull use the same configurable variable * rename exec_task_fn_wrapper to task_script_runner.py * fix typo * FIX MLFLow and Tensorboard Output to be consistent with new Workspace root changes (#2537) * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1. Remove the default code to use configuration 2. fix some broken notebook * rollback changes * Fix decorator issue (#2542) * update create and run job script * FLModel summary (#2544) * add FLModel Summary * format * remove jobs folder * expose aggregate_fn to users for overwriting (#2539) * handle cases where the script with relative path in Script Runner (#2543) * handle cases where the script with relative path * handle cases where the script with relative path * add more unit test cases and change the file search logics * code format * add more unit test cases and change the file search logics * Lr newton raphson (#2529) * Implement federated logistic regression with second-order newton raphson. Update file headers. Update README. Update README. Fix README. Refine README. Update README. Added more logging for the job status changing. (#2480) * Added more logging for the job status changing. * Fixed a logging call error. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Fix update client status (#2508) * check workflow id before updating client status * change order of checks Add user guide on how to deploy to EKS (#2510) * Add user guide on how to deploy to EKS * Address comments Improve dead client handling (#2506) * dev * test dead client cmd * added more info for dead client tracing * remove unused imports * fix unit test * fix test case * address PR comments --------- Co-authored-by: Sean Yang <seany314@gmail.com> Enhance WFController (#2505) * set flmodel variables in basefedavg * make round info optional, fix inproc api bug temporarily disable preflight tests (#2521) Upgrade dependencies (#2516) Use full path for PSI components (#2437) (#2517) Multiple bug fixes from 2.4 (#2518) * [2.4] Support client custom code in simulator (#2447) * Support client custom code in simulator * Fix client custom code * Remove cancel_futures args (#2457) * Fix sub_worker_process shutdown (#2458) * Set GRPC_ENABLE_FORK_SUPPORT to False (#2474) Pythonic job creation (#2483) * WIP: constructed the FedJob. * WIP: server_app josn export. * generate the job app config. * fully functional pythonic job creation. * Added simulator_run for pythonic API. * reformat. * Added filters support for pythonic job creation. * handled the direct import case in fed_job. * refactor. * Added the resource_spec set function for FedJob. * refactored. * Moved the ClientApp and ServerApp into fed_app.py. * Refactored: removed the _FilterDef class. * refactored. * Rename job config classes (#3) * rename config related classes * add client api example * fix metric streaming * add to() routine * Enable obj in the constructor as paramenter. * Added support for the launcher script. * refactored. * reformat. * Update the comment. * re-arrange the package location. * Added add_ext_script() for BaseAppConfig. * codestyle fix. * Removed the client-api-pt example. * removed no used import. * fixed the in_time_accumulate_weighted_aggregator_test.py * Added Enum parameter support. * Added docstring. * Added ability to handle parameters from base class. * Move the parameter data format conversion to the START_RUN event for InProcessClientAPIExecutor. * Added params_exchange_format for PTInProcessClientAPIExecutor. * codestyle fix. * Fixed a custom code folder structure issue. * work for sub-folder custom files. * backed to handle parameters from base classes. * Support folder structure job config. * Added support for flat folder from '.XXX' import. * codestyle fix. * refactored and add docstring. * Address some of the PR reviews. --------- Co-authored-by: Holger Roth <6304754+holgerroth@users.noreply.github.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Enhancements from 2.4 (#2519) * Starts heartbeat after task is pull and before task execution (#2415) * Starts pipe handler heartbeat send/check after task is pull before task execution (#2442) * [2.4] Improve cell pipe timeout handling (#2441) * improve cell pipe timeout handling * improved end and abort handling * improve timeout handling --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * [2.4] Enhance launcher executor (#2433) * Update LauncherExecutor logs and execution setup timeout * Change name * [2.4] Fire and forget for pipe handler control messages (#2413) * Fire and forget for pipe handler control messages * Add default timeout value * fix wait-for-reply (#2478) * Fix pipe handler timeout in task exchanger and launcher executor (#2495) * Fix metric relay pipe handler timeout (#2496) * Rely on launcher check_run_status to pause/resume hb (#2502) Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> --------- Co-authored-by: Yan Cheng <58191769+yanchengnv@users.noreply.github.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Update ci cd from 2.4 (#2520) * Update github actions (#2450) * Fix premerge (#2467) * Fix issues on hello-world TF2 notebook * Fix tf integration test (#2504) * Add client api integration tests --------- Co-authored-by: Isaac Yang <isaacy@nvidia.com> Co-authored-by: Sean Yang <seany314@gmail.com> use controller name for stats (#2522) Simulator workspace re-design (#2492) * Redesign simulator workspace structure. * working, needs clean. * Changed the simulator workspacce structure to be consistent with POC. * Moved the logfile init to start_server_app(). * optimzed. * adjust the stats pool location. * Addressed the PR views. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Simulator end run for all clients (#2514) * Provide an option to run END_RUN for all clients. * Added end_run_all option for simulator to run END_RUN event for all clients. * Fixed a add_argument type, added help message. * Changed to use add_argument(() compatible with python 3.8. * reformat. * rewrite the _end_run_clients() and add docstring for easier understanding. * reformat. * adjusting the locking in the _end_run_clients. * Fixed a potential None pointer error. * renamed the clients_finished_end_run variable. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Sean Yang <seany314@gmail.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Secure XGBoost Integration (#2512) * Updated FOBS readme to add DatumManager, added agrpcs as secure scheme * Refactoring * Refactored the secure version to histogram_based_v2 * Replaced Paillier with a mock encryptor * Added license header * Put mock back * Added metrics_writer back and fixed GRPC error reply simplify job simulator_run to take only one workspace parameter. (#2528) Fix README. Fix file links in README. Fix file links in README. Add comparison between centralized and federated training code. Add missing client api test jobs (#2535) Fixed the simulator server workspace root dir (#2533) * Fixed the simulator server root dir error. * Added unit test for SimulatorRunner start_server_app. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Improve InProcessClientAPIExecutor (#2536) * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * make result check and result pull use the same configurable variable * rename exec_task_fn_wrapper to task_script_runner.py * fix typo Update README for launching python script. Modify tensorboard logdir. Link to environment setup instructions. expose aggregate_fn to users for overwriting (#2539) FIX MLFLow and Tensorboard Output to be consistent with new Workspace root changes (#2537) * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1. Remove the default code to use configuration 2. fix some broken notebook * rollback changes Fix decorator issue (#2542) Remove line number in code link. FLModel summary (#2544) * add FLModel Summary * format formatting Update KM example, add 2-stage solution without HE (#2541) * add KM without HE, update everything * fix license header * fix license header - update year to 2024 * fix format --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> * update license --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Holger Roth <hroth@nvidia.com> * Add information about dig (bind9-dnsutils) in the document * format update * Update KM example, add 2-stage solution without HE (#2541) * add KM without HE, update everything * fix license header * fix license header - update year to 2024 * fix format --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> * Update monai readme to remove logging.conf (#2552) * MONAI mednist example (#2532) * add monai notebook * add training script * update example * update notebook * use job template * call init later * swith back * add gitignore * update notebooks * add readmes * send received model to GPU * use monai tb stats handler * formatting * Improve AWS cloud launch script * Add in process client api tests (#2549) * Add in process client api tests * Fix headers * Fix comments * Add client controller executor (#2530) * add client controller executor * address comments * enhance abort, set peer props * remove asserts --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * Add option in dashboard cli for AWS vpc and subnet * add note on README visualization * update README * update readme * update readme * update readme * [2.5] Clean up to allow creation of nvflare light (#2573) * clean up to allow creation of nvflare light * move defs to cellnet * Enable patch and build for nvflight (#2574) * verified commit --------- Co-authored-by: Yuhong Wen <yuhongw@nvidia.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Sean Yang <seany314@gmail.com> Co-authored-by: Zhijin <zhijinl@nvidia.com> Co-authored-by: Holger Roth <hroth@nvidia.com> Co-authored-by: Isaac Yang <isaacy@nvidia.com> Co-authored-by: Ziyue Xu <ziyue.xu@gmail.com> Co-authored-by: Ziyue Xu <71786575+ZiyueXu77@users.noreply.github.com> Co-authored-by: Holger Roth <6304754+holgerroth@users.noreply.github.com> Co-authored-by: Yan Cheng <58191769+yanchengnv@users.noreply.github.com> * fix MLFLOW example (#2575) Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * BugFix: InProcessClientAPIExecutor's TaskScriptRunner (#2558) * 1) find script full path to indicate which site script to avoid loading run script 2) make sure the task script failed will cause the client to return failure status which will trigger job stop rather wait forever 3) add different unit tests * sort key in unit test * add logic to improve error message * style format * add more tests and logics * code format * code format * fix steps error * fix global steps * rollback some changes and split it into another PR * rollback some changes and split it into another PR --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * update client_api.png (#2577) * Fix the simulator worker sys path (#2561) * Fixed the simulator worker sys path. * fixed the get_new_sys_path() logic, added in unit test. * fixed isort. * Changed the _get_new_sys_path() implementation. --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * ReliableMessage register is changed to register aux message. Added support for Mac with vertical --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Co-authored-by: Sean Yang <seany314@gmail.com> Co-authored-by: Isaac Yang <isaacy@nvidia.com> Co-authored-by: Yan Cheng <58191769+yanchengnv@users.noreply.github.com> Co-authored-by: Minghui Chen <50226876+MinghuiChen43@users.noreply.github.com> Co-authored-by: Yuhong Wen <yuhongw@nvidia.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Zhijin <zhijinl@nvidia.com> Co-authored-by: Holger Roth <hroth@nvidia.com> Co-authored-by: Ziyue Xu <ziyue.xu@gmail.com> Co-authored-by: Ziyue Xu <71786575+ZiyueXu77@users.noreply.github.com> Co-authored-by: Holger Roth <6304754+holgerroth@users.noreply.github.com>

* Updated FOBS readme to add DatumManager, added agrpcs as secure scheme * Implemented horizontal calls in nvflare plugin * Added support for horizontal secure XGBoost * Fixed a few horizontal issues * Added reliable message * Added ReliableMessage parameters * Added log for debugging empty rcv_buf * Added finally block to finish duplicate seq * Removed debug statements * format change * Add in process client api tests (NVIDIA#2549) * Add in process client api tests * Fix headers * Fix comments * Add client controller executor (NVIDIA#2530) * add client controller executor * address comments * enhance abort, set peer props * remove asserts --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * Add option in dashboard cli for AWS vpc and subnet * [2.5] Clean up to allow creation of nvflare light (NVIDIA#2573) * clean up to allow creation of nvflare light * move defs to cellnet * Enable patch and build for nvflight (NVIDIA#2574) * add FedBN Implementation on NVFlare research folder - a local batch normalization federated learning method (NVIDIA#2524) * add research/fedbn * delete redudant controller and correct figs requirements * update plot_requirements * rewrite fedbn * update jobs * remove workspace * update README * simplify job simulator_run to take only one workspace parameter. (NVIDIA#2528) * Add missing client api test jobs (NVIDIA#2535) * Fixed the simulator server workspace root dir (NVIDIA#2533) * Fixed the simulator server root dir error. * Added unit test for SimulatorRunner start_server_app. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> * Improve InProcessClientAPIExecutor (NVIDIA#2536) * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * make result check and result pull use the same configurable variable * rename exec_task_fn_wrapper to task_script_runner.py * fix typo * FIX MLFLow and Tensorboard Output to be consistent with new Workspace root changes (NVIDIA#2537) * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1. Remove the default code to use configuration 2. fix some broken notebook * rollback changes * Fix decorator issue (NVIDIA#2542) * update create and run job script * FLModel summary (NVIDIA#2544) * add FLModel Summary * format * remove jobs folder * expose aggregate_fn to users for overwriting (NVIDIA#2539) * handle cases where the script with relative path in Script Runner (NVIDIA#2543) * handle cases where the script with relative path * handle cases where the script with relative path * add more unit test cases and change the file search logics * code format * add more unit test cases and change the file search logics * Lr newton raphson (NVIDIA#2529) * Implement federated logistic regression with second-order newton raphson. Update file headers. Update README. Update README. Fix README. Refine README. Update README. Added more logging for the job status changing. (NVIDIA#2480) * Added more logging for the job status changing. * Fixed a logging call error. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Fix update client status (NVIDIA#2508) * check workflow id before updating client status * change order of checks Add user guide on how to deploy to EKS (NVIDIA#2510) * Add user guide on how to deploy to EKS * Address comments Improve dead client handling (NVIDIA#2506) * dev * test dead client cmd * added more info for dead client tracing * remove unused imports * fix unit test * fix test case * address PR comments --------- Co-authored-by: Sean Yang <seany314@gmail.com> Enhance WFController (NVIDIA#2505) * set flmodel variables in basefedavg * make round info optional, fix inproc api bug temporarily disable preflight tests (NVIDIA#2521) Upgrade dependencies (NVIDIA#2516) Use full path for PSI components (NVIDIA#2437) (NVIDIA#2517) Multiple bug fixes from 2.4 (NVIDIA#2518) * [2.4] Support client custom code in simulator (NVIDIA#2447) * Support client custom code in simulator * Fix client custom code * Remove cancel_futures args (NVIDIA#2457) * Fix sub_worker_process shutdown (NVIDIA#2458) * Set GRPC_ENABLE_FORK_SUPPORT to False (NVIDIA#2474) Pythonic job creation (NVIDIA#2483) * WIP: constructed the FedJob. * WIP: server_app josn export. * generate the job app config. * fully functional pythonic job creation. * Added simulator_run for pythonic API. * reformat. * Added filters support for pythonic job creation. * handled the direct import case in fed_job. * refactor. * Added the resource_spec set function for FedJob. * refactored. * Moved the ClientApp and ServerApp into fed_app.py. * Refactored: removed the _FilterDef class. * refactored. * Rename job config classes (NVIDIA#3) * rename config related classes * add client api example * fix metric streaming * add to() routine * Enable obj in the constructor as paramenter. * Added support for the launcher script. * refactored. * reformat. * Update the comment. * re-arrange the package location. * Added add_ext_script() for BaseAppConfig. * codestyle fix. * Removed the client-api-pt example. * removed no used import. * fixed the in_time_accumulate_weighted_aggregator_test.py * Added Enum parameter support. * Added docstring. * Added ability to handle parameters from base class. * Move the parameter data format conversion to the START_RUN event for InProcessClientAPIExecutor. * Added params_exchange_format for PTInProcessClientAPIExecutor. * codestyle fix. * Fixed a custom code folder structure issue. * work for sub-folder custom files. * backed to handle parameters from base classes. * Support folder structure job config. * Added support for flat folder from '.XXX' import. * codestyle fix. * refactored and add docstring. * Address some of the PR reviews. --------- Co-authored-by: Holger Roth <6304754+holgerroth@users.noreply.github.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Enhancements from 2.4 (NVIDIA#2519) * Starts heartbeat after task is pull and before task execution (NVIDIA#2415) * Starts pipe handler heartbeat send/check after task is pull before task execution (NVIDIA#2442) * [2.4] Improve cell pipe timeout handling (NVIDIA#2441) * improve cell pipe timeout handling * improved end and abort handling * improve timeout handling --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * [2.4] Enhance launcher executor (NVIDIA#2433) * Update LauncherExecutor logs and execution setup timeout * Change name * [2.4] Fire and forget for pipe handler control messages (NVIDIA#2413) * Fire and forget for pipe handler control messages * Add default timeout value * fix wait-for-reply (NVIDIA#2478) * Fix pipe handler timeout in task exchanger and launcher executor (NVIDIA#2495) * Fix metric relay pipe handler timeout (NVIDIA#2496) * Rely on launcher check_run_status to pause/resume hb (NVIDIA#2502) Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> --------- Co-authored-by: Yan Cheng <58191769+yanchengnv@users.noreply.github.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Update ci cd from 2.4 (NVIDIA#2520) * Update github actions (NVIDIA#2450) * Fix premerge (NVIDIA#2467) * Fix issues on hello-world TF2 notebook * Fix tf integration test (NVIDIA#2504) * Add client api integration tests --------- Co-authored-by: Isaac Yang <isaacy@nvidia.com> Co-authored-by: Sean Yang <seany314@gmail.com> use controller name for stats (NVIDIA#2522) Simulator workspace re-design (NVIDIA#2492) * Redesign simulator workspace structure. * working, needs clean. * Changed the simulator workspacce structure to be consistent with POC. * Moved the logfile init to start_server_app(). * optimzed. * adjust the stats pool location. * Addressed the PR views. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Simulator end run for all clients (NVIDIA#2514) * Provide an option to run END_RUN for all clients. * Added end_run_all option for simulator to run END_RUN event for all clients. * Fixed a add_argument type, added help message. * Changed to use add_argument(() compatible with python 3.8. * reformat. * rewrite the _end_run_clients() and add docstring for easier understanding. * reformat. * adjusting the locking in the _end_run_clients. * Fixed a potential None pointer error. * renamed the clients_finished_end_run variable. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Sean Yang <seany314@gmail.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Secure XGBoost Integration (NVIDIA#2512) * Updated FOBS readme to add DatumManager, added agrpcs as secure scheme * Refactoring * Refactored the secure version to histogram_based_v2 * Replaced Paillier with a mock encryptor * Added license header * Put mock back * Added metrics_writer back and fixed GRPC error reply simplify job simulator_run to take only one workspace parameter. (NVIDIA#2528) Fix README. Fix file links in README. Fix file links in README. Add comparison between centralized and federated training code. Add missing client api test jobs (NVIDIA#2535) Fixed the simulator server workspace root dir (NVIDIA#2533) * Fixed the simulator server root dir error. * Added unit test for SimulatorRunner start_server_app. --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Improve InProcessClientAPIExecutor (NVIDIA#2536) * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * 1. rename ExeTaskFnWrapper class to TaskScriptRunner 2. Replace implementation of the inprocess function exection from calling a main() function to user runpy.run_path() which reduce the user requirements to have main() function 3. redirect print() to logger.info() * make result check and result pull use the same configurable variable * rename exec_task_fn_wrapper to task_script_runner.py * fix typo Update README for launching python script. Modify tensorboard logdir. Link to environment setup instructions. expose aggregate_fn to users for overwriting (NVIDIA#2539) FIX MLFLow and Tensorboard Output to be consistent with new Workspace root changes (NVIDIA#2537) * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1) fix mlruns and tb_events dirs due to workspace directory changes 2) for MLFLow, add tracking_rui default to workspace_dir / <job_id>/mlruns instead current default <workspace_dir>/mlruns. This is a) consistent with Tensorboard 2) avoid job output oeverwrite the 1st job * 1. Remove the default code to use configuration 2. fix some broken notebook * rollback changes Fix decorator issue (NVIDIA#2542) Remove line number in code link. FLModel summary (NVIDIA#2544) * add FLModel Summary * format formatting Update KM example, add 2-stage solution without HE (NVIDIA#2541) * add KM without HE, update everything * fix license header * fix license header - update year to 2024 * fix format --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> * update license --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Holger Roth <hroth@nvidia.com> * Add information about dig (bind9-dnsutils) in the document * format update * Update KM example, add 2-stage solution without HE (NVIDIA#2541) * add KM without HE, update everything * fix license header * fix license header - update year to 2024 * fix format --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> * Update monai readme to remove logging.conf (NVIDIA#2552) * MONAI mednist example (NVIDIA#2532) * add monai notebook * add training script * update example * update notebook * use job template * call init later * swith back * add gitignore * update notebooks * add readmes * send received model to GPU * use monai tb stats handler * formatting * Improve AWS cloud launch script * Add in process client api tests (NVIDIA#2549) * Add in process client api tests * Fix headers * Fix comments * Add client controller executor (NVIDIA#2530) * add client controller executor * address comments * enhance abort, set peer props * remove asserts --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * Add option in dashboard cli for AWS vpc and subnet * add note on README visualization * update README * update readme * update readme * update readme * [2.5] Clean up to allow creation of nvflare light (NVIDIA#2573) * clean up to allow creation of nvflare light * move defs to cellnet * Enable patch and build for nvflight (NVIDIA#2574) * verified commit --------- Co-authored-by: Yuhong Wen <yuhongw@nvidia.com> Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Sean Yang <seany314@gmail.com> Co-authored-by: Zhijin <zhijinl@nvidia.com> Co-authored-by: Holger Roth <hroth@nvidia.com> Co-authored-by: Isaac Yang <isaacy@nvidia.com> Co-authored-by: Ziyue Xu <ziyue.xu@gmail.com> Co-authored-by: Ziyue Xu <71786575+ZiyueXu77@users.noreply.github.com> Co-authored-by: Holger Roth <6304754+holgerroth@users.noreply.github.com> Co-authored-by: Yan Cheng <58191769+yanchengnv@users.noreply.github.com> * fix MLFLOW example (NVIDIA#2575) Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * BugFix: InProcessClientAPIExecutor's TaskScriptRunner (NVIDIA#2558) * 1) find script full path to indicate which site script to avoid loading run script 2) make sure the task script failed will cause the client to return failure status which will trigger job stop rather wait forever 3) add different unit tests * sort key in unit test * add logic to improve error message * style format * add more tests and logics * code format * code format * fix steps error * fix global steps * rollback some changes and split it into another PR * rollback some changes and split it into another PR --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * update client_api.png (NVIDIA#2577) * Fix the simulator worker sys path (NVIDIA#2561) * Fixed the simulator worker sys path. * fixed the get_new_sys_path() logic, added in unit test. * fixed isort. * Changed the _get_new_sys_path() implementation. --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> * ReliableMessage register is changed to register aux message. Added support for Mac with vertical --------- Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <yuantingh@nvidia.com> Co-authored-by: Sean Yang <seany314@gmail.com> Co-authored-by: Isaac Yang <isaacy@nvidia.com> Co-authored-by: Yan Cheng <58191769+yanchengnv@users.noreply.github.com> Co-authored-by: Minghui Chen <50226876+MinghuiChen43@users.noreply.github.com> Co-authored-by: Yuhong Wen <yuhongw@nvidia.com> Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Co-authored-by: Zhijin <zhijinl@nvidia.com> Co-authored-by: Holger Roth <hroth@nvidia.com> Co-authored-by: Ziyue Xu <ziyue.xu@gmail.com> Co-authored-by: Ziyue Xu <71786575+ZiyueXu77@users.noreply.github.com> Co-authored-by: Holger Roth <6304754+holgerroth@users.noreply.github.com>

add research/fedbn

a25cf54

ZiyueXu77 requested review from holgerroth, ZiyueXu77, YuanTingHsieh and chesterxgchen April 23, 2024 15:27

chesterxgchen reviewed Apr 23, 2024

View reviewed changes

research/fedbn/custom/wf_controller.py Outdated Show resolved Hide resolved

chesterxgchen reviewed Apr 23, 2024

View reviewed changes

research/fedbn/custom/figs/plot_requirements.txt Outdated Show resolved Hide resolved

delete redudant controller and correct figs requirements

37587c5

MinghuiChen43 changed the title ~~add research/fedbn~~ add research/fedbn to Implementation of FedBN - a local batch normalization federated learning method with NVFlare Apr 24, 2024

MinghuiChen43 changed the title ~~add research/fedbn to Implementation of FedBN - a local batch normalization federated learning method with NVFlare~~ add FedBN Implementation on NVFlare research folder - a local batch normalization federated learning method Apr 24, 2024

update plot_requirements

a682140