Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extension of cloud storage server part #3386

Merged
merged 27 commits into from
Aug 31, 2021

Conversation

Marishka17
Copy link
Contributor

@Marishka17 Marishka17 commented Jul 3, 2021

Motivation and context

This PR contains:

  • changed the way of checking existing cloud storage and remote files: added checking cloud storage && file status (available, not found, forbidden)
  • modified the cloud storage creating
  • modified getting content from cloud storage: returned only content from the specified manifest file
  • fixed and modified cloud storage updating
  • fixed cloud storage filtration
  • fixed the credentials mapping
  • moved specific attributes parsing into utils
  • added file manifest model
  • added creating task manifest file from selected cloud storage manifest
  • added support of cloud storage preview
  • removed support TEMP_KEY_SECRET_KEY_TOKEN_SET for AWS S3
  • extended dataset manifest functionality

How has this been tested?

Checklist

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below)
# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

@Marishka17 Marishka17 requested a review from nmanovic as a code owner July 3, 2021 15:38
@Marishka17 Marishka17 mentioned this pull request Jul 5, 2021
22 tasks
@bsekachev bsekachev mentioned this pull request Aug 12, 2021
8 tasks
specific_attributes = serializer.validated_data.get('specific_attributes', '')

check_cloud_storage_existing(provider_type, credentials_type, session_token, account_name,
key, secret_key, resource, specific_attributes)
owner = self.request.data.get('owner')
if owner:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we consider owner to be a user who does not send the request? In theory I can create a storage and make owner a user who I do not know accidentally. This user might get access to private data.

cvat/apps/engine/views.py Outdated Show resolved Hide resolved
cvat/apps/engine/views.py Outdated Show resolved Hide resolved
cvat/apps/engine/views.py Outdated Show resolved Hide resolved
cvat/apps/engine/views.py Show resolved Hide resolved
cvat/apps/engine/views.py Outdated Show resolved Hide resolved
cvat/apps/engine/models.py Outdated Show resolved Hide resolved
@Marishka17 Marishka17 force-pushed the mk/expansion_server_cloud_storage branch from 60f348a to 070dbcf Compare August 13, 2021 11:20
@Marishka17 Marishka17 changed the title [WIP] expansion of cloud storage server part expansion of cloud storage server part Aug 13, 2021
cvat/apps/engine/cache.py Show resolved Hide resolved
cvat/apps/engine/views.py Show resolved Hide resolved
@@ -223,6 +223,9 @@ def load(self):
self._index = json.load(index_file,
object_hook=lambda d: {int(k): v for k, v in d.items()})

def remove(self):
os.remove(self._path)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't self.path be null or something like that after deleting?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, it shouldn't

@Marishka17 Marishka17 changed the title expansion of cloud storage server part [WIP] expansion of cloud storage server part Aug 25, 2021
@Marishka17
Copy link
Contributor Author

@azhavoro , @bsekachev What do you think about this?

@Marishka17 Marishka17 changed the title [WIP] expansion of cloud storage server part Extension of cloud storage server part Aug 27, 2021
@Marishka17 Marishka17 force-pushed the mk/expansion_server_cloud_storage branch from 3eb6a81 to c1f68a7 Compare August 27, 2021 15:17
@@ -6,7 +6,7 @@
class Migration(migrations.Migration):

dependencies = [
('engine', '0040_cloud_storage'),
('engine', '0041_auto_20210813_0853'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this is the right way, maybe reordering the migration is here. Please make sure this works if migration "0040" has already been applied.

@Marishka17
Copy link
Contributor Author

@azhavoro, Do you have any other comments?

@azhavoro
Copy link
Contributor

@Marishka17 LGTM

@bsekachev bsekachev merged commit fa15f78 into develop Aug 31, 2021
@bsekachev bsekachev deleted the mk/expansion_server_cloud_storage branch September 2, 2021 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants