18 Jul 17:13

geyang

d01884a

Fix Interaction with ML-Logger during configuration Latest

Latest

Working example in 01_ssh_docker

def train_fn(seed=None):
    from time import sleep
    from ml_logger import logger

    logger.print('this is running')
    logger.print(f"The exp seed is: {seed}", color="green")

    logger.print('This is sleeping...', color="yellow")
    sleep(5)
    logger.print('done!')


if __name__ == '__main__':
    import jaynes
    from ml_logger import logger, instr

    jaynes.config()

    for i in range(5):
        thunk = instr(train_fn)
        jaynes.add(thunk, seed=i * 100)

    jaynes.execute()
    jaynes.listen()

Assets 2

14 Sep 04:42

geyang

v0.7.10

72247e0

v0.7.10

Interactive vs Batch Jobs

When the interactive flag is true, we launch via the srun command. Otherwise, we pipe the bash script into a file, and launch using sbatch.

Launching Multiple Jobs (Interactive)

Just call jaynes.run with multiple copies of the function:

#! ./multi_launch.py
import jaynes
from launch_entry import train_fn

if __name__ == "__main__":
    jaynes.config(verbose=False)

    for i in range(3):
        jaynes.run(train_fn, seed=i * 100)

    jaynes.listen(200)

And the output should be 3 streams of stdout pipe-back combined together, running in parallel.

/Users/ge/opt/anaconda3/envs/plan2vec/bin/python /Users/ge/mit/jaynes-starter-kit/04_slurm_configuration/launch_entry.py
Jaynes pipe-back is now listening...
Running on login-node
Running inside worker
Running on login-node
Running on login-node
Running on login-node
Running inside worker
[seed: 200] See real-time pipe-back from the server:
[seed: 200] step: 0
[seed: 200] step: 1
Running inside worker
[seed: 200] step: 2
Running inside worker
[seed: 200] Finished!
[seed: 300] See real-time pipe-back from the server:
[seed: 300] step: 0
[seed: 300] step: 1
[seed: 300] step: 2
[seed: 100] See real-time pipe-back from the server:
[seed: 100] step: 0
[seed: 100] step: 1
[seed: 100] step: 2
[seed: 300] Finished!
[seed: 100] Finished!
[seed: 0] See real-time pipe-back from the server:
[seed: 0] step: 0
[seed: 0] step: 1
[seed: 0] step: 2
[seed: 0] Finished!

Launching Sequential Jobs with `SBatch`

To submit a sequence of jobs with sbatch,

turn off the interactive mode by setting it to false.
specify n_seq_jobs to be > 1 (default: null).
make sure you set a job name, because otherwise, all of your sbatch calls will be sequentially ordered.

For example, .jaynes.yml may look like:

- !runners.Slurm &slurm
  envs: >-
    LC_CTYPE=en_US.UTF-8 LANG=en_US.UTF-8 LANGUAGE=en_US
  startup: >-
    source /etc/profile.d/modules.sh
    source $HOME/.bashrc
  interactive: false
  n_seq_jobs: 3

Then, just call jaynes.run(train_fn) once:

#! ./seq_jobs_launch.py
import jaynes
from launch_entry import train_fn

if __name__ == "__main__":
    for index in range(10):
        jaynes.config(verbose=False, launch=dict(job_name=f"unique-job-{index}"))
        jaynes.run(train_fn)

This runs sbatch --job-name unique-job-0 -d singleton ... for n_seq_jobs=3 times, which requests sequential jobs.

Assets 2

12 Sep 03:01

geyang

v0.7.7

f74e00b

v0.7.7

Launching On GCP with Jaynes and Docker

This folder contains a working example for launching jobs on the Google Cloud Platform (GCP) with docker containers. At the end of the day, you would have 1. a python script and 2. a simple .jaynes script that allows you to scale your experiment instantly to thousands of instances on the GCP.

Example script:

import jaynes
from your_project import train, Args

for seed in [100, 200, 300]:
    jaynes.config(name=f"demo-instance/seed-{seed}")
    jaynes.run(train, seed=seed)

Note: The example config currently uses an S3 mount for the code upload. We currently do not have support for gce buckets, but that is an easy to implement. To add this support, submit a PR.

Before You Begin

Step 1: Installing `jaynes`

You need to have gcloud and gsutil installed on your computer, as well as jaynes.

pip install jaynes

Step 2: Installing the Cloud SDK (Google)

Then install and configure your gcloud and gsutil command line utilities according the these guides:

install the cloud SDK (gcloud): https://cloud.google.com/sdk/docs/install
Install gsutil: https://cloud.google.com/storage/docs/gsutil_install
Set a default region and zone.

Now after you have finished, you can verify that your cloud SDK is working via:

$ gcloud auth list

which should print out:

Credentialed Accounts
ACTIVE ACCOUNT
     * your-email@gmail.com
       your-other-email@gmail.com

To set the active account, run:

$ gcloud config set account <account>

Machine Learning At Scale with `jaynes` on GCP

The following are supported in jaynes>=v0.7.7 and above. See https://pypi.org/project/jaynes/0.7.7/

Part 1: Creating A GCP Bucket for Your Code and Data

First make sure that you are able to run the gsutil command. Now, create two buckets using the following command:

gsutil mb gs://$USER-jaynes-$ORGANIZATION
gsutil mb gs://$USER-data-$ORGANIZATION

If you mess up, remember even if you delete a bucket, it would take a while for its name to be released, so that you can recreate it using different settings. Just don't panic!

gsutil rb gs://$USER-jaynes-$ORGANIZATION
gsutil rb gs://$USER-data-$ORGANIZATION

Using AWS S3 with GCE instances

The aws cli is not pre-installed on the machine learning GCE VM images. Therefore to download from AWS S3, you need to install the commandline tool as part of the setup step of your .jaynes.runner configuration.

launch: !ENVS
  setup: pip install -q awscli jaynes ml-logger params-proto

To reuse the S3 code mount, you can copy and pasting the S3Mount config from the AWS tutorial into this .jaynes.yml config, to replace the existing mount. Make sure that you follow the AWS tutorial first.

Part 2: Double-Check Your Environment Variables

you need to have these in your ~/.profile.

#~/.profile

# environment variables for Google Compute Engine
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.gce/<your-project>.json
export JYNS_GCP_PROJECT=<your-project-id-1234>
export JYNS_GCP_BUCKET=<your-bucket-name>

Part 3: Docker Image

We include an example docker image in the ./docker/Dockerfile file. You need to install jaynes via RUN pip install jaynes in the docker image, to make the jaynes entry script available.

Part 4: Launch!

Now the launch is as simple as running

python launch_entry.py

Remember, turn on the verbose=True flag, to see the script being generated and details of the request.

Common Errors

error: name already exists: This means that the name you are using already exists as an VM instance. You should use a different instance name.

Config Examples and Values

Here is an example configuration for launching on GCE:

launch: !ENV
    type: gce
    launch_dir: /home/ec2-user/jaynes-mounts
    project_id: "{env.JYNS_GCE_PROJECT}"
    zone: us-east1-b
    image_project: deeplearning-platform-release
    image_family: pytorch-latest-gpu
    instance_type: n1-standard-1
    accelerator_type: 'nvidia-tesla-k80'
    accelerator_count: 1
    preemptible: true
    terminate_after: true

For the instance_type, you can only attach GPUs to general-purpose N1 VMs or accelerator-optimized A2 VMs. GPUs are not supported by other machine families.

general purpose machine types

The cpu count comes in powers of 2:

Machine types	vCPUs1	Memory (GB)
`n1-standard-1`	1	3.75
`n1-standard-2`	2	7.50
`n1-standard-4`	4	15
`n1-standard-8`	8	30
`n1-standard-16`	16	60
`n1-standard-32`	32	120
`n1-standard-64`	64	240
`n1-standard-96`	96	360

A vCPU is implemented as a single hardware Hyper-thread on one of the available CPU platforms.
Persistent disk usage is charged separately from machine type pricing.

For the accelerator_type, you can choose between the following gpus:

value	Details
`nvidia-tesla-t4`	NVIDIA® T4
`nvidia-tesla-t4-vws`	NVIDIA® T4 Virtual Workstation with NVIDIA® GRID®
`nvidia-tesla-p4`	NVIDIA® P4
`nvidia-tesla-p4-vws`	NVIDIA® P4 Virtual Workstation with NVIDIA® GRID®
`nvidia-tesla-p100`	NVIDIA® P100
`nvidia-tesla-p100-vws`	NVIDIA® P100 Virtual Workstation with NVIDIA® GRID®
`nvidia-tesla-v100`	NVIDIA® V100
`nvidia-tesla-k80`	NVIDIA® K80

accelerator optimized A2 types

comes in a 12:1 vCPU/A100 ratio. A2 VMs are only available on the Cascade Lake platform.

Machine types	vCPUs1	Memory (GB)
`a2-highgpu-1g`	12	85
`a2-highgpu-2g`	24	170
`a2-highgpu-4g`	48	340
`a2-highgpu-8g`	96	680
`a2-megagpu-16g`	96	1360

Pricing

NVIDIA GPUs

Model	GPUs	GPU memory	GPU price (USD)	Preemptible GPU price (USD)	1 year commitment price (USD)	3 year commitment price (USD)
NVIDIA® A100	1 GPU	40 GB HBM2	$2.933908 per GPU	$0.8801724 per GPU	$1.84836204 per GPU	$1.0268678 per GPU
NVIDIA® Tesla® T4	1 GPU	16 GB GDDR6	$0.35 per GPU	$0.11 per GPU	$0.220 per GPU	$0.160 per GPU
NVIDIA® Tesla® P4	1 GPU	8 GB GDDR5	$0.60 per GPU	$0.216 per GPU	$0.378 per GPU	$0.270 per GPU
NVIDIA® Tesla® V100	1 GPU	16 GB HBM2	$2.48 per GPU	$0.74 per GPU	$1.562 per GPU	$1.116 per GPU
NVIDIA® Tesla® P100	1 GPU	16 GB HBM2	$1.46 per GPU	$0.43 per GPU	$0.919 per GPU	$0.657 per GPU
NVIDIA® Tesla® K80	1 GPU	12 GB GDDR5	$0.45 per GPU	$0.135 per GPU	$0.283 per GPU	$0.92 per GPU

NVIDIA® GRID® Virtual Workstation GPUs

Model	GPUs	GPU memory	GPU price (USD)	Preemptible GPU price (USD)	1 year commitment price (USD)	3 year commitment price (USD)
NVIDIA® Tesla® T4 Virtual Workstation	1 GPU	16 GB GDDR6	$0.55 per GPU	$0.31 per GPU	$0.42 per GPU	$0.36 per GPU
NVIDIA® Tesla® P4 Virtual Workstation	1 GPU	8 GB GDDR5	$0.80 per GPU	$0.416 per GPU	$0.578 per GPU	$0.47 per GPU
NVIDIA® Tesla® P100 Virtual Workstation	1 GPU	16 GB HBM2	$1.66 per GPU	$0.63 per GPU	$1.119 per GPU	$0.857 per GPU

Assets 2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Interactive vs Batch Jobs

Launching Multiple Jobs (Interactive)

Launching Sequential Jobs with `SBatch`

Launching On GCP with Jaynes and Docker

Before You Begin

Step 1: Installing `jaynes`

Step 2: Installing the Cloud SDK (Google)

Machine Learning At Scale with `jaynes` on GCP

Part 1: Creating A GCP Bucket for Your Code and Data

Using AWS S3 with GCE instances

Part 2: Double-Check Your Environment Variables

Part 3: Docker Image

Part 4: Launch!

Common Errors

Config Examples and Values

general purpose machine types

accelerator optimized A2 types

Pricing

NVIDIA GPUs

NVIDIA® GRID® Virtual Workstation GPUs

Releases: geyang/jaynes-starter-kit

Fix Interaction with ML-Logger during configuration

v0.7.10

Interactive vs Batch Jobs

Launching Multiple Jobs (Interactive)

Launching Sequential Jobs with SBatch

v0.7.7

Launching On GCP with Jaynes and Docker

Before You Begin

Step 1: Installing jaynes

Step 2: Installing the Cloud SDK (Google)

Machine Learning At Scale with jaynes on GCP

Part 1: Creating A GCP Bucket for Your Code and Data

Using AWS S3 with GCE instances

Part 2: Double-Check Your Environment Variables

Part 3: Docker Image

Part 4: Launch!

Common Errors

Config Examples and Values

general purpose machine types

accelerator optimized A2 types

Pricing

NVIDIA GPUs

NVIDIA® GRID® Virtual Workstation GPUs

Launching Sequential Jobs with `SBatch`

Step 1: Installing `jaynes`

Machine Learning At Scale with `jaynes` on GCP