Skip to content

Commit

Permalink
Merge pull request #926 from MaxRis/917
Browse files Browse the repository at this point in the history
gdrive: update docs
  • Loading branch information
shcheklein authored Feb 4, 2020
2 parents ef74b9d + b37c6ab commit 0e831a3
Show file tree
Hide file tree
Showing 6 changed files with 131 additions and 49 deletions.
51 changes: 7 additions & 44 deletions public/static/docs/command-reference/remote/add.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,52 +200,15 @@ $ dvc remote add myremote "azure://"

### Click for Google Drive

Since Google Drive has tight API usage quotas, creation and configuration of
your own `Google Project` is required:

1. Log into the [Google Cloud Platform](https://console.developers.google.com)
account.
2. Create `New Project` or select available one.
3. Click `ENABLE APIS AND SERVICES` and search for `drive` to enable
`Google Drive API` from search results.
4. Navigate to
[All Credentials](https://console.developers.google.com/apis/credentials)
page and click `Create Credentials` to select `OAuth client ID`. It might
ask you to setup a product name on the consent screen.
5. Select `Other` for `Application type` and click `Create` to proceed with
default `Name`.
6. `client id` and `client secret` should be showed to you. Use them for
further DVC's configuration.
Please check out
[Setup a Google Drive DVC Remote](/doc/user-guide/setup-google-drive-remote) for
a full guide on configuring Google Drives for use as DVC remote storage,
including obtaining the necessary credentials, and how to form `gdrive://` URLs.

```dvc
$ dvc remote add myremote gdrive://root/my-dvc-root
$ dvc remote modify myremote gdrive_client_id my_gdrive_client_id
$ dvc remote modify myremote gdrive_client_secret gdrive_client_secret
```

> See `dvc remote modify` for a full list of GDrive parameters.
On first usage of the remote you will be prompted to visit an access token
generation URL via browser. It will ask you to log into the Google account
associated with the Google Drive you want to use as remote. The login process
will guide you through the granting of the required access permissions.

On successful access token generation, the token data will be cached in a
Git-ignored directory (located in `.dvc/tmp/gdrive-user-credentials.json`).

> 💡 Do not share the token data with anyone else to prevent unauthorized access
> to your Google Drive.
**Support for shared drives**

For this, you need to obtain the directory ID and use it as part of URL passed
to DVC. This ID can be found in your web browser address bar when the shared
drive is opened. For example, for the URL
`https://drive.google.com/drive/folders/0AIac4JZqHhKmUk9PDA`, use
`0AIac4JZqHhKmUk9PDA` as ID:

```dvc
$ dvc remote add myremote gdrive://0AIac4JZqHhKmUk9PDA/my-dvc-root
$ dvc remote add -d myremote gdrive://root/path/to/folder
$ dvc remote modify myremote gdrive_client_id <client ID>
$ dvc remote modify myremote gdrive_client_secret <client secret>
```

Note that GDrive remotes are not "trusted" by default. This means that the
Expand Down
15 changes: 10 additions & 5 deletions public/static/docs/command-reference/remote/modify.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,22 +248,27 @@ For more information on configuring Azure Storage connection strings, visit

### Click for Google Drive

Please check out
[Setup a Google Drive DVC Remote](/doc/user-guide/setup-google-drive-remote) for
a full guide on configuring Google Drives for use as DVC remote storage,
including obtaining the necessary credentials, and how to form `gdrive://` URLs.

- `url` - remote location URL.

```dvc
$ dvc remote modify myremote url "gdrive://root/my-dvc-root"
$ dvc remote modify myremote url gdrive://root/path/to/folder
```

- `gdrive_client_id` - Google Project's OAuth 2.0 client id.
- `gdrive_client_id` - Google Project's OAuth 2.0 **client ID**.

```dvc
$ dvc remote modify myremote gdrive_client_id my_gdrive_client_id
$ dvc remote modify myremote gdrive_client_id <client ID>
```

- `gdrive_client_secret` - Google Project's OAuth 2.0 client secret.
- `gdrive_client_secret` - Google Project's OAuth 2.0 **client secret**.

```dvc
$ dvc remote modify myremote gdrive_client_secret gdrive_client_secret
$ dvc remote modify myremote gdrive_client_secret <client secret>
```

</details>
Expand Down
1 change: 1 addition & 0 deletions public/static/docs/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@
}
},
"updating-tracked-files",
"setup-google-drive-remote",
"large-dataset-optimization",
"external-dependencies",
{
Expand Down
113 changes: 113 additions & 0 deletions public/static/docs/user-guide/setup-google-drive-remote.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# Setup a Google Drive DVC Remote

Follow this guide to setup Google Drive as your <abbr>DVC project</abbr>'s
[remote storage](/doc/command-reference/remote).

## Configure a Google Cloud project

DVC uses the [Google Drive API](https://developers.google.com/drive) to connect
to your Google Drive. This requires a Google Cloud _project_ that allows Drive
API connections, and its
[OAuth](https://developers.google.com/identity/protocols/OAuth2) credentials
(**client ID** and **client secret**).

1. Sign into the [Google API Console](https://console.developers.google.com).

> Double check you're using the intended Google account (upper-right corner).
2. Select or
[Create](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project)
a project for DVC remote connections.

3. [Enable the Drive API](https://developers.google.com/drive/api/v2/about-sdk)
from the **APIs & Services** **Dashboard** (left sidebar), click on **+
ENABLE APIS AND SERVICES**. Find and select the "Google Drive API" in the API
Library, and click on the **ENABLE** button.

![](/static/img/gdrive-enable-apis-and-services.png)

4. Go back to **APIs & Services** in the left sidebar, and select **OAuth
consent screen**. Chose a **User Type** and click **CREATE**. On the next
screen, enter an **Application name** e.g. "DVC remote storage", and click
the **Save** (scroll to bottom).

5. From the left sidebar, select **Credentials**, and click the **Create
credentials** dropdown to select **OAuth client ID**. Chose **Other** and
click **Create** to proceed with a default client name.

![](/static/img/gdrive-create-credentials.png)

6. The newly generated **client ID** and **client secret** should be shown to
you now, and you can always come back to **Credentials** to fetch them.

⚠️ It should be safe to share **client ID** and **client secret** among your
team. These credentials are only used to generate the
[authorization](#authorization) URL DVC will later prompt to visit in order to
connect to the Google Drive.

> Note that Google Drive API has usage limits/quotas per credentials in a
> _project_ (which you can review in the
> [OAuth consent screen](https://console.developers.google.com/apis/credentials/consent)).
> Please keep this in mind when sharing them, or you may
> [exceed the limits](https://developers.google.com/drive/api/v2/handle-errors?hl=ro#resolve_a_403_error_usage_limit_exceeded).
## URL format for Google Drive remotes

A GDrive DVC remote URLs is needed for the `dvc remote add` command. It can be
constructed with a _base_, and an **optional** _path_ to a folder i.e.
`gdrive://<base>/path/to/folder`. The base can be one of:

1. `root` - indicates your topmost Google Drive directory.

```dvc
$ dvc remote add -d mygdroot gdrive://root
$ dvc remote add mygdfolder gdrive://root/path/to/folder
```

2. Google Drive _Folder ID_

To obtain the unique _Folder ID_ in question, navigate into that folder in your
web browser, and find it in the address bar. For example, for
`https://drive.google.com/drive/folders/0AIac4JZqHhKmUk9PDA`:

```dvc
$ dvc remote add mygdfolder gdrive://0AIac4JZqHhKmUk9PDA
$ dvc remote add mygdsubfolder \
gdrive://0AIac4JZqHhKmUk9PDA/sub/path
```

[Shared drives](https://developers.google.com/drive/api/v2/about-shareddrives)
should be referenced by _Folder ID_ in the same way.

3. `appDataFolder` - special
[application-specific data](https://developers.google.com/drive/api/v2/appdata)
folder only accessible by your application and hidden from the user.

```dvc
$ dvc remote add mygdappata gdrive://appDataFolder
```

## Configure the GDrive remote

Use the `dvc remote modify` command to set the credentials for each `gdrive://`
remote, for example:

```dvc
$ dvc remote add -d mygdfolder gdrive://root/path/to/folder
$ dvc remote modify mygdfolder gdrive_client_id <client ID>
$ dvc remote modify mygdfolder gdrive_client_secret <client secret>
```

## Authorization

On the first usage of a GDrive [DVC remote](/doc/command-reference/remote),
you'll be prompted to visit a web page to generate an access token, which will
require you to sign into the corresponding Google account. The
[sign-in process](https://developers.google.com/drive/api/v2/about-auth) will
guide you through granting the necessary access permissions. On success, the
token data will be cached in a Git-ignored directory located in
`.dvc/tmp/gdrive-user-credentials.json`.

⚠️ In order to prevent unauthorized access to your Google Drive, **do not share
access token data with others**. Each team member should generate their own
tokens.
Binary file added public/static/img/gdrive-create-credentials.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 0e831a3

Please sign in to comment.