Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial #3745

Merged
merged 24 commits into from
Oct 5, 2021

Conversation

TOsmanov
Copy link
Contributor

@TOsmanov TOsmanov commented Sep 29, 2021

Motivation and context

Depended #3724

Issues:

  • To generate a manifest file, you need to download all the files to your pc that will be inconvenient for datasets of large volume. As well, adding data to the dataset will require updating the manifest file.
    It may be worth adding support for S3 inventory in CVAT. In the S3 inventory contains paths to files with extension, and you can also add an additional field ETag which can be an MD5 digest of the object data.
  • After the file manifest is created, the name parameter has \\ Instead of /. For fix this, I used .replace in core.py file, perhaps there is a better solution.

Preview:
https://tosmanov.github.io/cvat/docs/manual/basics/attach-cloud-storage/

How has this been tested?

Checklist

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below)
# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

@TOsmanov TOsmanov changed the title Documentation update, added info about the cloud storage UI Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial Sep 29, 2021
@TOsmanov TOsmanov changed the title Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial [WIP]Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial Sep 29, 2021
### Upload dataset

Initially you need to create a manifest file for your image dataset. Information on how to do that is available
on the [Simple command line to prepare dataset manifest file](/docs/manual/advanced/dataset_manifest) page.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TOsmanov , let's repeat here basic steps to upload a simple dataset. Need exact instructions. Let's take a video file (take a link from serverless tutorial), prepare a manifest and upload them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nmanovic , as far as I understand cloud storage into UI does not support the use of video file, do you want me to describe the division of the video into frame sequence?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Marishka17 , could you please comment on that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TOsmanov , OK. Let's use a dataset with images in the case.

@TOsmanov TOsmanov changed the title [WIP]Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial Sep 30, 2021
Copy link
Contributor

@Marishka17 Marishka17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

[script for prepare manifest file](https://github.com/openvinotoolkit/cvat/tree/develop/utils/dataset_manifest).
Perform the installation using the manual [aws-shell](https://github.com/awslabs/aws-shell),
I used `aws-cli 1.20.49` `Python 3.7.9` `Windows 10`.
You can configure credentials by running.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can configure credentials by running.
You can configure credentials by running `aws configure`.

![](/images/aws-s3_tutorial_7.jpg)

### Prepare manifest file
Now you need to prepare manifest file. I used [AWS cli](https://aws.amazon.com/cli/) and
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Now you need to prepare manifest file. I used [AWS cli](https://aws.amazon.com/cli/) and
Now you need to prepare a manifest file. I used [AWS cli](https://aws.amazon.com/cli/) and

### Create user and configure permissions

To access bucket you will need to create a user, to do this, go [IAM](https://console.aws.amazon.com/iamv2/home#/users)
and click `add users`. You need to choose AWS access type have access key ID and secret access key.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
and click `add users`. You need to choose AWS access type have access key ID and secret access key.
and click `add users`. You need to choose AWS access type, have an access key ID and secret access key.

You'll be taken to the bucket creation page. Here you have to specify the name of the bucket, region,
optionally you can copy the settings of another bucket by clicking on the `choose bucket` button.
Checkbox block all public access can be enabled as we will use `access key ID` and `secret access key` to gain access.
In the following sections you can leave the default settings and click `create bucket`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In the following sections you can leave the default settings and click `create bucket`.
In the following sections, you can leave the default settings and click `create bucket`.

@Marishka17
Copy link
Contributor

@TOsmanov , Could you please also prepare the same tutorial for Azure in another PR?

@TOsmanov
Copy link
Contributor Author

TOsmanov commented Oct 5, 2021

@TOsmanov , Could you please also prepare the same tutorial for Azure in another PR?

@Marishka17 Okay, I'll add

Copy link
Contributor

@nmanovic nmanovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job!

@nmanovic nmanovic merged commit ced1bc8 into cvat-ai:develop Oct 5, 2021
@TOsmanov TOsmanov mentioned this pull request Oct 8, 2021
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants