Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Storage][Blob] Added support for Object Replication #11525

Merged
merged 5 commits into from
Jun 5, 2020
Merged

[Storage][Blob] Added support for Object Replication #11525

merged 5 commits into from
Jun 5, 2020

Conversation

zezha-msft
Copy link
Contributor

@zezha-msft zezha-msft commented May 19, 2020 •
edited
Loading

Description of the context replicated from .NET PR:

Object Replication for Download Response and Get Properties Response.

Object replication service allows user set a policy on source account to back up some blobs to another account. When retrieving the properties details when doing a Get Properties Call or Download Call, on a blob that has a Object Replication Policy applied to it or was created as a result of a replication policy, it will return the header prefixed with "x-ms-or". If no Object replication policy was applied, this header is not returned.

When creating a source/destination pair, a replication policy gets created (policy id). Within each policy there may be multiple replication rules (e.g. rule 1= src/container/.pdf to dst/container2/; rule2 = src/container1/.jpg to dst/container3;). Multiple object replication policies ids can be applied to a source account, and for each policy id, multiple rule ids can be applied to each policy id.

For source blobs (blobs that have policy ids and rule ids applied to them), the header will be formatted as "x-ms-or-<policy_id>_<rule_id>: {Complete, Failed}". The value of this header is the status of the replication. For destination blobs (blobs created as result of a replication policy), the header will returned will be formatted as "x-ms-or-policy-id: <policy_id>". The value of this header is the policy id.

This PR enables users to retrieve the policy id on a destination blob. It also enables users to retrieve a dictionary that contains the details of the Object Replication applied to the source blob. The dictionary is formatted as so Dictionary< policy_id, Dictionary < rule_id, status of replication(Complete,Failed) >(e.g. { policy_id1, { {rule_id1, Complete}, {rule_id2, Complete} }; {policy_id2, { {rule_id3, Complete} };..)

TODO: Eventually make tests apply policies to storage accounts through using the management plane sdk. Which is why the tests are recorded only for now.
To be added in the future: Object Replication headers returned on List Blobs call

@annatisch annatisch added the Storage Storage Service (Queues, Blobs, Files) label May 22, 2020
annatisch
annatisch reviewed May 29, 2020
View reviewed changes
Copy link
Member

@annatisch annatisch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good - a couple of tiny things :)

sdk/storage/azure-storage-blob/azure/storage/blob/_deserialize.py Outdated Show resolved Hide resolved
sdk/storage/azure-storage-blob/azure/storage/blob/_deserialize.py Outdated Show resolved Hide resolved
sdk/storage/azure-storage-blob/azure/storage/blob/_deserialize.py Outdated Show resolved Hide resolved
sdk/storage/azure-storage-blob/azure/storage/blob/_models.py Outdated Show resolved Hide resolved
sdk/storage/azure-storage-blob/tests/test_ors.py Show resolved Hide resolved
@zezha-msft zezha-msft force-pushed the feature/storage/ors branch from a8bd125 to 8ef4eab Compare June 4, 2020 07:35
@xiafu-msft
Copy link
Contributor

According to the doc it looks like list blobs also returns the header? https://microsoft.sharepoint.com/:w:/r/teams/AzureStorage/_layouts/15/Doc.aspx?sourcedoc=%7BBAE60A2A-CAB4-4BFE-9663-34BB25E5CBFC%7D&file=ORS_XFE.docx&action=default&mobileredirect=true&CID=449DD0DE-0B6D-4E6C-820B-700108448A11&wdLOR=cFE5F05E6-ECD4-4D99-8FAD-DB156014E6DD

@xiafu-msft xiafu-msft merged commit b68c071 into Azure:feature/storage-stg73 Jun 5, 2020
xiafu-msft pushed a commit that referenced this pull request Jun 10, 2020
@zezha-msft @xiafu-msft
[Storage][Blob] Added support for Object Replication (#11525)
0870aed
xiafu-msft added a commit that referenced this pull request Jul 7, 2020
@xiafu-msft @kasobol-msft
@zezha-msft @annatisch
Feature/storage stg73 (#12272)
900aa7d 
* [Storage]STG73

* [Blob][Swagger]Update Swagger (#10943)

* [Blob][Swagger]Regenerate Swagger Code

* fix container test failure caused by list_containers include type change

* [Storage] Internal avro parser. (#10764)

* initial avro parser

* try fixing test...

* falling in love with python compatibility...

* make linter happy.

* raise StopIteration when there is no more bytes instead of tracking file length.

* async avro parser

* fix syntax for Python 3.5

* get rid of 'readers_schema' as we only honor schema that has been written to file ('writer_schema').

* pr feedback

* trim unused code.

* pr feedback.

* simplify skip sync in next.

* move avro tests from _shared.

* Jumbo blob support (#11176)

* wip

* initial test coverage.

* wip.

* wip

* single upload.

* add async tests.

* disable 50k block tests.

* datalake append.

* async datalake

* disable tests that send large payload over network.

* pr feedback.

* Undelete share (#11394)

* Undelete container (#11339)

* [Storage][Blob] Added support for Object Replication (#11525)

* Blob versioning (#11154)

* [Blob][QuickQuery]Add Quick Query Support (#10946)

* [Blob][STG73]Blob Tags (#11418)

* regenerate code (#11964)

* fix the bug which caused only showing fatal error (#11997)

* [Storage][STG73]Address API Review Comments (#12111)

* [Storage][STG73]Address API Review Comments

* [Storage][STG73]dict<policy, rules> -> list(ObjectReplicationPolicy)

* fix blob tag_value test

* expose ObjectReplicationPolicy and ObjectReplicationRule, fix test

* fix test

* Changefeed (#10755)

* [ChangeFeed]Add ChangeFeed Package

* test_avro failure

* update dev_requirement.txt

* change namespace to azure.storage.blob.changefeed

* address comments

* optimize memory when reading changefeed events

* namespace change

* set up package change

* fix failed tests

* readme and kwargs

* Update sdk/storage/azure-storage-blob-changefeed/azure/storage/blob/changefeed/_change_feed_client.py

Co-authored-by: Rakshith Bhyravabhotla <sabhyrav@microsoft.com>

* address comments

* 'azure-storage-blob>=12.3.0' which does not match the frozen requirement 'azure-storage-blob~=1.3'

Co-authored-by: Rakshith Bhyravabhotla <sabhyrav@microsoft.com>

* [Storage-Blob] Quick Query API (#11991)

* Renamed query error

* Renamed query reader

* Updated config models

* Updated format request params

* Updated iterator

* fix the bug which caused only showing fatal error

* Updated Error message

* Fixed query helper

* Started test conversion

* small fix

* Fixed tests

* Updated error handling + json model

* Updated recordings

* Removed old recording

* Added iter tests

* Iter test recordings

* Fix test

* Remove extra recording

* Fix pylint

* Some docs cleanup

* Renamed iter_records -> iter_stream

* Review feedback

* Updated tests

* Missing commas

* Fix syntax

* Fix pylint

Co-authored-by: xiafu <xiafu@microsoft.com>

* tag sas (#12258)

* tag sas

* disable undelete_container

* pylint

* skip undelete_container tests

* [Blob][Versioning]Disable Versioning Live Test (#12281)

* [Blob][QQ]Default output_format to input_format (#12283)

* [Storage][Jumbo]Remove super (#12314)

* [Storage][JumboBlob]remove empty super()

* pypy3

* change sas version to latest

* set tags account location to central canada

* re-recording queue

* changefeed paths generator

* mark tests for vid as playback only

* fix changefeed

* fix pylint
make the test account location for tags to central canada

* add a delay before calling find_blobs_by_tags

* remove tags header

* mark a large file test playback only

* revert "mark a large file test playback only"
skip upload large file test
address comment

* move tag permission and filter_by_tags permission to kwargs

Co-authored-by: Kamil Sobol <61715331+kasobol-msft@users.noreply.github.com>
Co-authored-by: Ze Qian Zhang <zezha@microsoft.com>
Co-authored-by: Rakshith Bhyravabhotla <sabhyrav@microsoft.com>
Co-authored-by: annatisch <antisch@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rakshith91 rakshith91 rakshith91 left review comments

@annatisch annatisch annatisch left review comments

@xiafu-msft xiafu-msft xiafu-msft approved these changes

@amishra-dev amishra-dev Awaiting requested review from amishra-dev

Assignees
No one assigned
Labels
Storage Storage Service (Queues, Blobs, Files)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zezha-msft @xiafu-msft @annatisch @rakshith91