Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.8.75 Docs Release #325

Merged
merged 12 commits into from
Jul 20, 2022
Original file line number Diff line number Diff line change
Expand Up @@ -113,15 +113,14 @@ grid datastore create s3://ryft-public-sample-data/esRedditJson/ --name lightnin
Example:
`grid datastore create S3://ruff-public-sample-data/esRedditJson --no-copy`

In certain cases, your S3 bucket may fit one (or both) of the following criteria:
Use --no-copy when you want to avoid creating a duplicate copy of the data in your cluster account. Using this flag can significantly speed up Datastore creation time by preventing the copy process.

(1) the bucket is continually updating with new data which you want included in a Grid Datastore
(2) the bucket is particularly large (leading to long Datastore creation times)
:::info
We recommend that you use this flag when creating a private S3 Datastore within a BYOC cluster.

In these cases, you can pass the `--no-copy` flag to the `grid datastore create` command. This flag will
prevent Grid from making a copy of the dataset, which significantly speeds up Datastore creation time.
Also consider using this option when the your dataset size is greater than `500 GB`.

:::info
If your Datastore was from a public S3 bucket, having a copy in the cluster can be advantageous if someone deletes or modifys the public bucket. `--no-copy` does not prevent you from deleting the file in S3 as thats managed by the user. If thats done the Datastore will show that the file still exists since the metadata is cached, On access S3 will indicate that the file does not exist anymore.

When using this flag, you cannot remove files from your bucket. If you'd like to add files, please create a new version of the Datastore after you've added files to your bucket.

Expand Down
7 changes: 6 additions & 1 deletion docs/getting-started/typical-workflow-cli-user.md
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ cd grid-tutorials/getting-started
pip install -r requirements.txt
```

now run the following command to train a resnet18 on 2 GPUs
now run the following command to train a resnet18 on 1 GPU

```bash
python flash-image-classifier.py \
Expand Down Expand Up @@ -387,6 +387,11 @@ Make sure you are in the /grid-tutorials/getting-started directory for the tutor

Now kick off the run with grid run

:::note
To convert the below run into a multi-gpu Run please set the `--instance_type` parameter to a machine with multiple gpus
and update the `--gpus` argument to the desired number of GPUs. See [here](../platform/4_machines.md) for a table of machine types and their hardware specs.
:::

```bash
grid run --dependency_file ./requirements.txt \
--name cifar-tut-hpo \
Expand Down
66 changes: 19 additions & 47 deletions docs/platform/3_credentials.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,65 +135,37 @@ When complete, please enter the role ARN:

The `grid credential create` command will output the required trust and permission policies.

a. Leave the command prompt open while visiting the [AWS IAM Role
Console](https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/roles)
a. Naviagte to AWS WebUI -> IAM -> [Role](https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/roles)

b. Click "Create Role"
b. Create Role -> Custom trust policy

![](../../static/images/credentials/role-console.png)

c. Select the "custom trust policy" setting, and copy/paste the trust policy output
from the CLI into the `json` editor:
c. Paste the output of the `trust policy` from the CLI into the `json` editor and click Next

![](../../static/images/credentials/select_trusted_entity.png)

d. When prompted with the "add permissions to role" tab, click "Create Policy" to create a new permission
policy
d. In the `Add permissions` page click Next

e. The role name must begin with `grid-s3-access-` and click on Create role

![](../../static/images/credentials/role_naming.png)

f. You will be redirected to IAM -> Roles. Select your just created

g. Click on Add Permissions -> Create Inline Policy. Paste the output `permission policy` from the CLI output after replacing your bucket name and click on Review Policy

![](../../static/images/credentials/permission_policy_naming.png)

![](../../static/images/credentials/add_permissions_to_role.png)

e. A new tab will open. Select the `json` tab, and
copy/paste the permission policy into the editor. Also replace the text
`<replace-with-bucket-name>` with the name of the private bucket to provide access to
(i.e. `gridai-demo-bucket`).

![](../../static/images/credentials/aws_permission_policy.png)

f. Click "next" until reaching the "permission policy naming" tab. Enter an appropriate
name, for example: `my-permission-policy-for-grid`.

![](../../static/images/credentials/permission_policy_naming.png)

g. Click "create policy".

![](../../static/images/credentials/permission_policy_created.png)

h. Go back to the previous tab for attaching a permission policy to the role

i. Refresh the list of policies by clicking the refresh symbol
h. Enter an appropriate name, for example: my-permission-policy-for-grid

j. Select the `my-permission-policy-for-grid` policy name and click "next".

![](../../static/images/credentials/attach_permission_policy.png)

### 3. Create AWS Role Name

I then navigate through the prompts until I am presented with the "role naming" screen. At
this point, I create a role name **keeping in mind that the role must begin with the
prefix: `grid-s3-access-`**. In this case I name the role
`grid-s3-access-for-my-demo-bucket` and click create.

![](../../static/images/credentials/role_naming.png)


I am then presented with a success screen! The role has been created.
![](../../static/images/credentials/add_permissions_to_role.png)


### 4. Input role ARN in Grid

![](../../static/images/credentials/role_created.png)

I select the role and click on the icon next to the "role ARN" in order to
copy the value to my clipboard:
Within Roles -> Search for the role you created that began with `grid-s3-access-` and click on it
Click on the icon next to the "role ARN" in order to copy the value to my clipboard:

![](../../static/images/credentials/arn_copy.png)

Expand Down