HPO: Alibaba DSW+DLC support #4055

weidankong · 2021-08-10T20:58:37Z

HPO: DSW(DataScienceWorkShop) as client and DLC(DeepLearningContainers) as computing resources.

please find config in /nni/examples/trials/mnist-pytorch/config_dlc.yml

ghost · 2021-08-10T20:58:49Z

All CLA requirements met.

SparkSnail · 2021-08-11T12:41:38Z

Please add doc for DLC mode, refer https://github.com/microsoft/nni/blob/master/docs/en_US/TrainingService/AMLMode.rst

SparkSnail · 2021-08-12T11:08:04Z

ts/nni_manager/training_service/reusable/dlc/dlcClient.ts

+        const deferred: Deferred<string> = new Deferred<string>();
+        this.pythonShellClient = new PythonShell('dlcUtil.py', {
+            scriptPath: './config/dlc',
+            pythonPath: 'python',


Detect OS, use python for windows, otherwise use python3.

in dlc mode, only Linux environment is available, no windows case.

SparkSnail · 2021-08-12T11:08:55Z

ts/nni_manager/training_service/reusable/dlc/dlcClient.ts

+    private getScript(): string[] {
+        const script: string[] = [];
+        script.push(
+            `python ./config/dlc/dlcUtil.py --type ${this.type} --image ${this.image} --pod_count ${this.podCount} ` +


Detect OS, use python for windows, otherwise use python3.

in dlc mode, only Linux environment is available, no windows case.

SparkSnail · 2021-08-12T11:10:13Z

ts/nni_manager/training_service/reusable/dlc/dlcClient.ts

+        this.monitorError(this.pythonShellClient, deferred);
+        return deferred.promise;
+
+        return deferred.promise;


SparkSnail · 2021-08-12T11:20:49Z

ts/nni_manager/training_service/reusable/environments/dlcEnvironmentService.ts

+        this.experimentId = info.experimentId;
+        this.experimentRootDir = info.logDir;
+        this.config = flattenConfig(config, 'dlc');
+        component.Container.bind(StorageService).to(MountedStorageService).scope(Scope.Singleton);


should initialize storage service with your own localStorageMountPoint.

SparkSnail · 2021-08-12T11:20:56Z

ts/nni_manager/training_service/reusable/environments/dlcEnvironmentService.ts

+    }
+
+    public get hasStorageService(): boolean {
+        return false;


SparkSnail · 2021-08-12T11:27:41Z

ts/nni_manager/training_service/reusable/environments/dlcEnvironmentService.ts

+            await fs.promises.mkdir(environmentLocalTempFolder, {recursive: true});
+        }
+
+        const dlcFolder: string = this.experimentRootDir.replace(


seems this.experimentRootDir is assigned to info.logDir, why does it contains this.config.localStorageMountPoint? why don't use dlcFolder = this.config.containerStorageMountPoint directly?

updated to use storageService similar to openPai.

SparkSnail · 2021-08-12T11:29:04Z

ts/nni_manager/training_service/reusable/environments/dlcEnvironmentService.ts

+        }
+
+        const prepare = `cd ${dlcEnvironment.runnerWorkingFolder} && cp -r ../../environment-temp/envs/* ../`;
+        const startrun = `sh ../install_nni.sh && python -m nni.tools.trial_tool.trial_runner`;


environment.command already initialized this command https://github.com/microsoft/nni/blob/master/ts/nni_manager/training_service/reusable/trialDispatcher.ts#L660. BTW why use python instead of python3?

in default DLC container image, python is python3.

SparkSnail · 2021-08-12T11:39:16Z

nni/tools/nnictl/config_schema.py

@@ -130,7 +130,7 @@ def validate(self, data):
    Optional('maxTrialDuration'): And(Regex(r'^[1-9][0-9]*[s|m|h|d]$', error='ERROR: maxTrialDuration format is [digit]{s,m,h,d}')),
    Optional('maxTrialNum'): setNumberRange('maxTrialNum', int, 1, 99999),
    'trainingServicePlatform': setChoice(
-        'trainingServicePlatform', 'remote', 'local', 'pai', 'kubeflow', 'frameworkcontroller', 'dlts', 'aml', 'adl', 'hybrid'),
+        'trainingServicePlatform', 'remote', 'local', 'pai', 'kubeflow', 'frameworkcontroller', 'dlts', 'aml', 'adl', 'hybrid', 'dlc'),


If you provide v1 configuration, please add corresponding convert in https://github.com/microsoft/nni/blob/master/nni/experiment/config/convert.py#L16.

remove v1 support.

SparkSnail · 2021-08-12T11:42:28Z

examples/trials/mnist-pytorch/config_dlc.yml

+  region: cn-hangzhou
+  nasDataSourceId: ${your_nas_data_source_id}
+  accessKeyId: ${your_ak_id}
+  accessKeySecret: ${your_ak_key}


provide accessKeySecret directly in yaml file directly is not a recommended way.

SparkSnail · 2021-08-12T11:45:32Z

ts/nni_manager/config/dlc/dlcUtil.py

+        ecs_spec=args.ecs_spec,
+    )
+
+    job_type = 'TFJob' if args.image.find('tensorflow') >= 0 else 'PyTorchJob'


Provide job type as nni job configuration, detect image name is not good way.

fixed, add a new param in config.

SparkSnail · 2021-08-16T03:16:08Z

docs/en_US/reference/experiment_config.rst

@@ -933,3 +1039,5 @@ containerName
 AzureBlob container name.

 type: ``str``
+
+ion


what is this used for?

SparkSnail · 2021-08-16T03:17:57Z

nni/tools/nnictl/config_schema.py

@@ -477,6 +499,7 @@ def validate(self, data):
    'dlts': Schema({**common_schema, **dlts_trial_schema, **dlts_config_schema}),
    'hybrid': Schema({**common_schema, **hybrid_trial_schema, **hybrid_config_schema, **machine_list_schema,
                             **pai_config_schema, **aml_config_schema, **remote_config_schema}),
+    'dlc': Schema({**common_schema, **dlc_trial_schema, **dlc_config_schema}),


This config schema file is for v1 config, if you don't support v1 config, should remove this.

SparkSnail · 2021-08-16T03:19:33Z

ts/nni_manager/training_service/reusable/environments/dlcEnvironmentService.ts

+        }
+
+        environment.command = `cd ${environmentRoot} && ${environment.command}`;
+        environment.command = `${environment.command} 1>${environment.runnerWorkingFolder}/trialrunner_stdout 2>${environment.runnerWorkingFolder}/trialrunner_stderr`;


These two lines can be merged.

SparkSnail · 2021-08-17T05:03:14Z

docs/en_US/TrainingService/DLCMode.rst

+
+Step 2. Create PAI-DSW server following this `link <https://help.aliyun.com/document_detail/163684.html?section-2cw-lsi-es9#title-ji9-re9-88x>`__. Note as the training service will be run on PAI-DLC, it won't cost many resources to run and you may just need a PAI-DSW server with CPU.
+
+Step 3. Open PAI-DLC `here <https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest>`__, select the same region as your PAI-DSW server. Move to ``dataset configuration`` and mount the same NAS disk as the PAI-DSW server does. (Note currently only PAI-DLC public-cluster is supported.)


thanks, updated!

cruiseliu · 2021-08-20T09:32:49Z

ts/nni_manager/training_service/reusable/dlc/dlcClient.ts

+        this.userCommand = userCommand;
+    }
+
+    private getScript(): string[] {


Seems this function is replaced by PythonShell.

cruiseliu · 2021-08-20T09:41:54Z

ts/nni_manager/config/dlc/dlcUtil.py

+import json
+from argparse import ArgumentParser
+# ref: https://help.aliyun.com/document_detail/203290.html?spm=a2c4g.11186623.6.727.6f9b5db6bzJh4x
+from alibabacloud_pai_dlc20201203.client import Client


Who is responsible for installing these dependencies?

For Ali PAI-DSW, it's default installed.
otherwise, follow the "Step 4" DLCMode.rst.

cruiseliu · 2021-08-20T09:49:24Z

ts/nni_manager/training_service/reusable/environments/dlcEnvironmentService.ts

+        return 'dlc';
+    }
+
+    public async refreshEnvironmentsStatus(environments: EnvironmentInformation[]): Promise<void> {


I suppose the returned promise should be resolved when refreshing is done. @SparkSnail
If that's the case, this function should return Promise.all(...).

this is to keep same as refreshEnvironmentsStatus of KubernetesEnvironmentService/AMLEnvironmentService.

SparkSnail requested review from liuzhe-lz, SparkSnail and QuanluZhang August 11, 2021 09:22

weidankong force-pushed the dlc_test branch 3 times, most recently from 8cf25e0 to d57d738 Compare August 11, 2021 19:17

SparkSnail reviewed Aug 12, 2021

View reviewed changes

weidankong added 3 commits August 13, 2021 04:25

dlc: init dlc & sumit dlc & start trial_runner

64a59c4

DLC: support file command channel used in dlcEnvironment

5818644

DLC: remove redundant script dir param

45eb3a5

weidankong force-pushed the dlc_test branch from d57d738 to 45eb3a5 Compare August 12, 2021 20:25

DLC: add document && add job_type in config

b30c1a2

weidankong force-pushed the dlc_test branch from 916a91e to 840b756 Compare August 14, 2021 00:15

SparkSnail reviewed Aug 16, 2021

View reviewed changes

weidankong force-pushed the dlc_test branch from 840b756 to 5279863 Compare August 17, 2021 02:54

SparkSnail reviewed Aug 17, 2021

View reviewed changes

SparkSnail approved these changes Aug 17, 2021

View reviewed changes

weidankong force-pushed the dlc_test branch from 5279863 to 87b2db3 Compare August 17, 2021 05:27

cruiseliu reviewed Aug 20, 2021

View reviewed changes

DLC: use storage service & update doc

891994f

weidankong force-pushed the dlc_test branch from 87b2db3 to 891994f Compare August 20, 2021 19:48

liuzhe-lz approved these changes Aug 21, 2021

View reviewed changes

SparkSnail merged commit e9c21fd into microsoft:master Aug 23, 2021

weidankong deleted the dlc_test branch September 11, 2021 03:30

liuzhe-lz mentioned this pull request Oct 15, 2021

NNI 2021 August~September Iteration Planning #3986

Closed

78 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HPO: Alibaba DSW+DLC support #4055

HPO: Alibaba DSW+DLC support #4055

weidankong commented Aug 10, 2021

ghost commented Aug 10, 2021 •

edited by ghost

Loading

SparkSnail commented Aug 11, 2021

SparkSnail Aug 12, 2021 •

edited

Loading

weidankong Aug 12, 2021

SparkSnail Aug 12, 2021 •

edited

Loading

weidankong Aug 12, 2021

SparkSnail Aug 12, 2021

weidankong Aug 13, 2021

SparkSnail Aug 12, 2021

SparkSnail Aug 12, 2021

SparkSnail Aug 12, 2021

weidankong Aug 14, 2021

SparkSnail Aug 12, 2021 •

edited

Loading

weidankong Aug 12, 2021

SparkSnail Aug 12, 2021

weidankong Aug 14, 2021

SparkSnail Aug 12, 2021

SparkSnail Aug 12, 2021

weidankong Aug 13, 2021

SparkSnail Aug 16, 2021

weidankong Aug 17, 2021

SparkSnail Aug 16, 2021 •

edited

Loading

weidankong Aug 17, 2021

SparkSnail Aug 16, 2021

weidankong Aug 17, 2021

SparkSnail Aug 17, 2021

weidankong Aug 17, 2021

cruiseliu Aug 20, 2021 •

edited

Loading

weidankong Aug 20, 2021

cruiseliu Aug 20, 2021

weidankong Aug 20, 2021

cruiseliu Aug 20, 2021 •

edited

Loading

weidankong Aug 20, 2021


		Step 2. Create PAI-DSW server following this `link <https://help.aliyun.com/document_detail/163684.html?section-2cw-lsi-es9#title-ji9-re9-88x>`__. Note as the training service will be run on PAI-DLC, it won't cost many resources to run and you may just need a PAI-DSW server with CPU.

		Step 3. Open PAI-DLC `here <https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest>`__, select the same region as your PAI-DSW server. Move to ``dataset configuration`` and mount the same NAS disk as the PAI-DSW server does. (Note currently only PAI-DLC public-cluster is supported.)

HPO: Alibaba DSW+DLC support #4055

HPO: Alibaba DSW+DLC support #4055

Conversation

weidankong commented Aug 10, 2021

ghost commented Aug 10, 2021 • edited by ghost Loading

SparkSnail commented Aug 11, 2021

SparkSnail Aug 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkSnail Aug 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkSnail Aug 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkSnail Aug 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cruiseliu Aug 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cruiseliu Aug 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Aug 10, 2021 •

edited by ghost

Loading

SparkSnail Aug 12, 2021 •

edited

Loading

SparkSnail Aug 12, 2021 •

edited

Loading

SparkSnail Aug 12, 2021 •

edited

Loading

SparkSnail Aug 16, 2021 •

edited

Loading

cruiseliu Aug 20, 2021 •

edited

Loading

cruiseliu Aug 20, 2021 •

edited

Loading