Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Datalake] Removed list_paths manual paging and deserialization #16309

Merged
merged 6 commits into from
Feb 1, 2021

Conversation

tasherif-msft
Copy link
Contributor

@tasherif-msft tasherif-msft commented Jan 22, 2021

Using the new generator the list paths generated operation is able to handle paging and property deserialization which allows us to remove a lot of manually added code. This resolves #16230

@ghost ghost added the Storage Storage Service (Queues, Blobs, Files) label Jan 22, 2021
@tasherif-msft tasherif-msft changed the title [Datalake] Removed path listing manual paging and deserialization [Datalake] Removed list_paths manual paging and deserialization Jan 24, 2021
@tasherif-msft
Copy link
Contributor Author

/azp run python - storage - tests

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@tasherif-msft
Copy link
Contributor Author

/azp run python - storage - tests

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@Azure Azure deleted a comment from azure-pipelines bot Jan 27, 2021
@tasherif-msft
Copy link
Contributor Author

/azp run python - storage - ci

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@@ -259,7 +259,7 @@ def test_create_largest_blob_from_stream_single_upload_without_network(self, res
payload_dropping_policy = PayloadDroppingPolicy()
credential_policy = _format_shared_key_credential(storage_account.name, storage_account_key)
self._setup(storage_account, storage_account_key, [payload_dropping_policy, credential_policy],
max_single_put_size=LARGEST_SINGLE_UPLOAD_SIZE)
max_single_put_size=LARGEST_SINGLE_UPLOAD_SIZE+1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this in an attempt to resolve the flaky test test_create_largest_blob_from_stream_single_upload_without_network which seems to sometimes do a PUT twice rather than once causing one of the assertions to fail.

@@ -385,7 +384,7 @@ def get_paths(self, path=None, # type: Optional[str]
recursive=True, # type: Optional[bool]
max_results=None, # type: Optional[int]
**kwargs):
# type: (...) -> ItemPaged[PathProperties]
# type: (...) -> AsyncItemPaged[PathProperties]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this a break? what if somebody checks type? should we alias?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have previously fixed incorrect type annotations. The return object in this method is AsyncItemPaged so this was a mistake that needed to be fixed.

@tasherif-msft tasherif-msft merged commit c94211b into Azure:master Feb 1, 2021
iscai-msft added a commit to iscai-msft/azure-sdk-for-python that referenced this pull request Feb 2, 2021
…into analyze_redesign

* 'master' of https://github.com/Azure/azure-sdk-for-python: (35 commits)
  Sync eng/common directory with azure-sdk-tools for PR 1353 (Azure#16465)
  Normalize the package name for Doc.Ms readme (Azure#16401)
  fix changelog and version (Azure#16445)
  we should always run integration to publish from our artifacts. if the build pipeline crashed too early in the pipeline, this will fail regardless, due to inability to pull the artifacts down (Azure#15058)
  Fix DateTime bug (Azure#16456)
  Resolve Regression Failures (Azure#16455)
  [text analytics] Expose 'string_index_type' parameter in all service client methods where applicable (Azure#16412)
  adding devtools to the appconfig dev_reqs to solve python core issue (Azure#16381)
  Copy job matrix functionality (Azure#16450)
  Add APIView KV variable group to prepare pipelines bot (Azure#16451)
  [Datalake] Added support for PurePosixPath (Azure#16400)
  Regenerate baseline because last one break. (Azure#16415)
  adding step to test for crlf line endings (Azure#16398)
  [Datalake] Removed list_paths manual paging and deserialization (Azure#16309)
  Sync eng/common directory with azure-sdk-tools for PR 1351 (Azure#16448)
  Update auto_codegen.py (Azure#16443)
  First release purview (Azure#16440)
  1ES pools update for release pipeline. (Azure#16419)
  Add Cloud environment for Teams user (Azure#16359)
  Sync eng/common directory with azure-sdk-tools for PR 1345 (Azure#16404)
  ...
iscai-msft added a commit to iscai-msft/azure-sdk-for-python that referenced this pull request Feb 2, 2021
…into enum-meta

* 'master' of https://github.com/Azure/azure-sdk-for-python: (128 commits)
  Communication identity api redesign (Azure#16420)
  fix EH samples and docs (Azure#16457)
  T2 redisenterprise 2021 02 02 (Azure#16472)
  Update automation_generate.sh (Azure#16470)
  Sync eng/common directory with azure-sdk-tools for PR 1353 (Azure#16465)
  Normalize the package name for Doc.Ms readme (Azure#16401)
  fix changelog and version (Azure#16445)
  we should always run integration to publish from our artifacts. if the build pipeline crashed too early in the pipeline, this will fail regardless, due to inability to pull the artifacts down (Azure#15058)
  Fix DateTime bug (Azure#16456)
  Resolve Regression Failures (Azure#16455)
  [text analytics] Expose 'string_index_type' parameter in all service client methods where applicable (Azure#16412)
  adding devtools to the appconfig dev_reqs to solve python core issue (Azure#16381)
  Copy job matrix functionality (Azure#16450)
  Add APIView KV variable group to prepare pipelines bot (Azure#16451)
  [Datalake] Added support for PurePosixPath (Azure#16400)
  Regenerate baseline because last one break. (Azure#16415)
  adding step to test for crlf line endings (Azure#16398)
  [Datalake] Removed list_paths manual paging and deserialization (Azure#16309)
  Sync eng/common directory with azure-sdk-tools for PR 1351 (Azure#16448)
  Update auto_codegen.py (Azure#16443)
  ...
tasherif-msft added a commit that referenced this pull request Feb 9, 2021
* Removed path listing manual deserialization

* removed imports

* removed imports

* linter

* lint

* fixing flaky large file upload test
openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-python that referenced this pull request Oct 15, 2021
PR to  make swagger changes for new event grid public SDK release (Azure#16309)

* create baseline commit

* Add all the changes for new public SDK release

* fix the version

* fixing prettier check issues

* address pr comments

* fix arm version in readme

* Add extra line

* fix lintdiff issue

* Adding suppression and fixing sdk readme

* fix issues with go sdk path
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Storage Storage Service (Queues, Blobs, Files)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Blob][Datalake] Remove manually added paging code for listing operations
2 participants