Progress on aztables #14676

christothes · 2021-05-18T22:00:48Z

Creating this PR for some early feedback on the API shape

fixups eliminate dependency on go-check format refactor testing to testify refactor tests to testify entity parsing wip Implement complex entity and struct entity support fb to/from map and structs delete unused file remove pointer return from GetRecordedVariable remove pointer return from GetRecordedVariable in aztables merge

christothes · 2021-05-18T22:06:11Z

sdk/tables/aztables/tableClient.go

-	resp, err := t.client.Create(ctx, TableProperties{&name}, nil, nil)
-	if resp == nil {
-		return TableResponseResponse{}, err
+func (t *TableClient) QueryAsStruct(opt QueryOptions, s FromMapper) StructEntityQueryResponsePager {


The response type for this operation is mostly just a draft, but I'm curious about any feedback related to this idea for returning query results as a well defined interface struct rather than the default map[string]interface{}. This allows developers to use "strongly typed" models rather than the map.

I'm not familiar with tables as a lib...do you know what the query results will be?

The developer would know, typically. Because a table is schema-less it is possible to get back a result with jagged schema. But if that was expected, you'd use the map result query instead.

christothes · 2021-05-18T22:13:24Z

sdk/tables/aztables/tableClient.go

-func (t TableClient) Delete(ctx context.Context, name string) (TableDeleteResponse, error) {
-	return t.client.Delete(ctx, name, nil)
+// AddEntity creates an entity from an arbitrary struct value.
+func (t *TableClient) AddEntity(ctx context.Context, entity interface{}) (*TableInsertEntityResponse, *runtime.ResponseError) {


This func is the analog for the QueryAsStruct method above. It allows adding of a "strongly typed" table entity struct rather than the default map shape.

sdk/internal/testframework/recording_sanitizer_test.go

sdk/tables/aztables/CHANGELOG.md

catalinaperalta · 2021-05-19T14:13:40Z

sdk/tables/aztables/recordings/TestServiceClient_Cosmos/TestCreateTable.yaml

+      Dataserviceversion:
+      - "3.0"
+      User-Agent:
+      - azsdk-go-tables/<version> azsdk-go-tables/<version> azcore/v0.13.4 (go1.16.2;


The telemetry string in the generated code needs to be updated to replace <version> with v0.1.0.
Also just noticed shouldn't it have generated with azsdk-go-aztables? instead of just ending with tables

sdk/tables/aztables/tableClient.go

catalinaperalta · 2021-05-19T14:40:03Z

sdk/tables/aztables/tableClient.go

 }

 // NewTableClient creates a TableClient object using the specified URL and request policy pipeline.
-func NewTableClient(serviceURL string, cred azcore.Credential, options *TableClientOptions) (*TableClient, error) {
-	con := newConnection(serviceURL, cred, options.getConnectionOptions())
+func NewTableClient(tableName string, serviceURL string, cred azcore.Credential, options *TableClientOptions) (*TableClient, error) {


Curious, since I heard you mention that azidentity cant be used with Tables....should the credential type here be different than an azcore.Credential interface then? Because azidentity does implement that. Could it lead to some confusion?

Good question. In other languages, the type is typically TokenCredential to indicate this. Based on the naming, I assumed this was not implied to be a token credential type. Is there not an equivalently named type in azcore?

catalinaperalta · 2021-05-19T14:48:26Z

sdk/tables/aztables/table_pagers.go

+	Err() error
+}
+
+type StructEntityQueryResponsePager interface {


Why do you have the word "Struct" in this interface name?

Same question for the other types that include it.

To differentiate it from the map version. The naming is really placeholder for now, but I'd be interested to hear any suggestions on how to differentiate these types by name.

check-enforcer · 2021-05-21T00:01:31Z

This repository is protected by Check Enforcer. The check-enforcer check-run will not pass until there is at least one more check-run successfully passing. Check Enforcer supports the following comment commands:

/check-enforcer evaluate; tells Check Enforcer to evaluate this pull request.
/check-enforcer override; by-pass Check Enforcer (approvals still required).

catalinaperalta

I think you can merge this into your branch. I do think we need a small more informal API review with Jeff. There are a few things I can see that would be good to discuss like some naming (for instance finding a replacement for struct in the pager), some return types (like the interfaces), etc. Could be done in the APIView tool to only focus on public surface area for the UX study.

catalinaperalta · 2021-05-21T22:28:49Z

sdk/tables/aztables/cosmos_patch_transform_policy.go

+)
+
+// CosmosPatchTransformPolicy transforms PATCH requests into POST requests with the "X-HTTP-Method":"MERGE" header set.
+type CosmosPatchTransformPolicy struct{}


Wondering if this should have a ctor...like NewCosmosPatchTransformPolicy? To match the policies in azcore. This is probably a bigger discussion that may require both Jeff and Joel's input as well. In my opinion it's a bit odd to call ctors for all the other and not just for this one..

I had the same thought - the only reason I didn't is that there is no config, so a standard instance with no arguments is all is needed. I'm fine to change it if we decided that it makes sense.

sdk/tables/aztables/table_client.go

sdk/tables/aztables/table_transactional_batch.go

sdk/tables/aztables/zc_tableConstants.go

* implement batch operations and all entity operations

* Initial generation of the aztables module (#14441) * initial generation * Update ci.yml * Introduce internal/testFramework and test aztables with it (#14491) * testFramework initial implementation * sanitizer fixes (#14600) * Progress on aztables (#14676) * implement batch operations and all entity operations * AsModels * rename * comments * comment tweaks, options as pointer * add EntityMapAsModel * refactors to AddEntity * update some code docs * doc fixes * doc fixes * doc improvements * rename testframework to recording * fixing tables to go from testframework -> recording * renaming recordingInstance -> r * file name change Co-authored-by: Christopher Scott <chriss@microsoft.com> Co-authored-by: catalinaperalta <catalinaperaltah@hotmail.com>

* Initial generation of the aztables module (#14441) * initial generation * Update ci.yml * Introduce internal/testFramework and test aztables with it (#14491) * testFramework initial implementation * sanitizer fixes (#14600) * Progress on aztables (#14676) * implement batch operations and all entity operations * AsModels * rename * comments * comment tweaks, options as pointer * add EntityMapAsModel * refactors to AddEntity * update some code docs * doc fixes * doc fixes * doc improvements * rename testframework to recording * move ci.yml * ci path * ci path * [Tables] Update from testframework -> recording (#14888) * Initial generation of the aztables module (#14441) * initial generation * Update ci.yml * Introduce internal/testFramework and test aztables with it (#14491) * testFramework initial implementation * sanitizer fixes (#14600) * Progress on aztables (#14676) * implement batch operations and all entity operations * AsModels * rename * comments * comment tweaks, options as pointer * add EntityMapAsModel * refactors to AddEntity * update some code docs * doc fixes * doc fixes * doc improvements * rename testframework to recording * fixing tables to go from testframework -> recording * renaming recordingInstance -> r * file name change Co-authored-by: Christopher Scott <chriss@microsoft.com> Co-authored-by: catalinaperalta <catalinaperaltah@hotmail.com> * [Tables] ci fix (#14956) * [Tables] Add post-request validation of entity (#14960) * adds a secondary validation if the entity doesnt have pk/rk * creating an errors file just for handling errors * removing extra line * moving error to constants file, renaming constants file * improving test * [Tables] changing QueryOptions param to be a ptr (#14961) * changing QueryOptions param to be a ptr * feedback from catalina * [Tables] linting Fixes (#15028) * small fixes * small fixes * small fixes * small fixes * small fixes * small fixes * adding error checking to transactional batching * adding error checking to test set up * adding more error checking * adding more error checking * fixing two tests * fixing error reporting * fixing error reporting * fixing error reporting * fixing last lint failure * [Tables] Adding swagger.md (#15016) * adding swagger * updating swagger with autorest command * more fixes * fixing directive * updating swagger.md * fixed swagger * fixing directive * add dataplane flag * updating swagger * commented out tests, passes remaining * addressing chris hamons feedback * undoing changes * undoing the rest of changes * updating generated code * formatting gen code * more formatting corrections * one last formatting correction * [Tables] Add get/set access policy (#15027) * adding a way to skip cosmos tests, adding access policies * adding support for getting/setting access policies * tests working * does not work for multiple access policies * adding post validation * added error validation, note about possible issue * set policy works * removing table access policy file * [Tables] Add GetStats, GetProperties, and SetProperties methods (#15112) * adding GetProperties * basic test of set properties * added logging and metrics tests * tests for setting service properties * adding test for retention too long * adding samples and docs * [Tables] Add support for Entity struct (#15003) * adding test for custom entity * further * starting access policy * working for basic entity * adding to test * new generated code, removed one more test * AddEntity now takes []bytes * adding directive to change entity to []byte, changes to compile and be able to pass * converting Value map[string]interface{} -> []byte in convenience layer * paging works the same way now too * fixing two tests, update complex and basic test entities, need to add type hints for int64/float64 types * adding test for delete method, changed delete etag to be nil * working test for merge * Added Merge test * fix up test query simple entity * complex query test * everything except for batch tests * basic batch functionality * fixing batch mixed test * fixed last batch test * added separate file for testing batch actions * fixing query on table service client to take a pointer as well * working on annotating entities * fixing recordings * working with odata annotations * working test for using odata to annotate entity * adding options to get * working with embedded entity now too * adding to entity, peeling off prints * converted to a type * first change to add EdmEntity type * updated unmarshalling * removing two files * fix editLink * removing sample * cleaning up * removing more code * cleaning up * formatting * formatting * [Tables] change Query to List (#15099) * adding List methods * moving helper to zt file * improving docstrings * changing all query methods to list * updating docstrings * [Tables] Update transactions method (#15125) * adding test for custom entity * further * starting access policy * working for basic entity * adding to test * new generated code, removed one more test * AddEntity now takes []bytes * adding directive to change entity to []byte, changes to compile and be able to pass * converting Value map[string]interface{} -> []byte in convenience layer * paging works the same way now too * fixing two tests, update complex and basic test entities, need to add type hints for int64/float64 types * adding test for delete method, changed delete etag to be nil * working test for merge * Added Merge test * fix up test query simple entity * complex query test * everything except for batch tests * basic batch functionality * fixing batch mixed test * fixed last batch test * added separate file for testing batch actions * fixing query on table service client to take a pointer as well * working on annotating entities * fixing recordings * working with odata annotations * working test for using odata to annotate entity * adding options to get * working with embedded entity now too * adding to entity, peeling off prints * converted to a type * first change to add EdmEntity type * updated unmarshalling * removing two files * fix editLink * removing sample * cleaning up * removing more code * cleaning up * re-record and remove methods * more cleanup * formatting * having issues with cosmos tests * Adding Table suffix to Create, Delete, and List methods on service client (#15181) * [Tables] Architect Feedback (#15196) * architect feedback * docstrings * feedback * fixing eng pipelines * figuring out pipeline failure * missing recording * restore eng changes * [EngSys] Test proxy (#15101) * [Tables] Add authentication via SAS credential (#15256) * changing recording to use test-proxy * updating client and test proxy * updated Transport Do to send actual request * whitespace * working implementation * updating internal recording * now I have a blank recording... * cleaning up * more clean up * transport Do is repeating, proxy is not parsing URL correctly * small changes * adding a policy to do the same thing * adding a second test for proxy and transport * further progress with Jeff, switching to only using a policy right now * cleaning up file * cleaning up * added more debugging, added two required headers that I was missing * proxy working on a single test, need to convert all tests to test proxy * cleaning up * moved over successfully :) * more helper functions * working for a subtest functionality * fixed up a service client test * converting more service tests * converted all service client tests * converted table client tests * fixed last two client tests * entity test * entity tests * converting access policy tests * all passing in record mode * batch tests * liveonly for batch tests * adding a sanitizer and more methods for recording * adding main method to start and stop the proxy automatically * adding test proxy step to pipeline * double dash on version arg * need a better way to get userful error reporting * explicitly running test-proxy in the background * adjust the proxy start and install to be compatible with specific process * dont run test proxy in background * didnt remove an import * header to proxy_test.go * fixes for pipeline * added loggin on accident * use nohup on linux machines * further simplify nohup usage * there are no exes on windows * fixing errors, passing in default env variable * force background for linux task. may need another iteration to place it within the quotes * fake aad credential * clean up after test-proxy * add print for record mode * another different attempt at starting the proxy * jobs apparently is an unrecognized command on our linux boxes * correct dumb typo in run tests * replace nuget install with docker * working fake credentials for client delete entity test * fixing fake credentials portion, passing locally * trying an insecureskipverify transport * debugging statements to find the cwd * removing blank recording * working implementation of sas * working sas credential for account signature * table sas is failing * leverage proxy tool, not docker * remove apostrophes * moving recordings one directory up * transitioning to using the sas written by storage team * using storage methods * copied storage code, sas needs to be appended in a different way * changing directory for test-proxy to start from * removing parenthesis * adding steps for validating ssl * switching to docker * adding start server script * working implementation if we can fix the prepending of 'recording' to the docker request finder * big refactor, moving stuff into recording file, cleaning up proxy_test.go file * last fixings * convert back to docker. windows and linux images now present * double condition * moving configuration to a separate file. preparing for eng/common move * tier 0 of trust. I don't expect this to work, but it's still worth a shot. maybe the devops agents do something differently * update condition to use variable syntax * guess I'm not using variables. removing * call start-server.ps1 * adjust build-test and configure-proxy to run the docker container in context * wrap in quotes for the container create. it's apparently a bad * adjust the initialize call * disable vet temporarily * looking for cert file in env variable PROXY_CERT * set proxy_cert environment variable to find that certificate * changing to crt file * use crt cert * want to see errors * powershell errors * correcting how the volume binds to the windows container * small modification for windows container * finally got the magic sauce * cleaning up to remove azcore from internal * removing recordings that use vcr * issue with the url creation * persist query params correctly in azcore.JoinPaths * removing print statements * return root * changing location of script * forgot the stop command * working table level sas implementation * adding start/end rk/pk functionality * removing print statements * removing more non-tables code * docstrings and removing storage only code * making sas tests live only * Fixing sas table name to be lowercase always, adding test to verify read only * Fixing sas table name to be lowercase always, adding test to verify read only * adding cosmos sas test * fix for params * apiview fixes * fixing test * undoing change * updates * error naming * fixed service client, was adding percallpolicies twice * simplification * aligning with main * converting from query pager to list pager for naming consistency * updating objects for consistency * jeffs feedback * small fixes * autorest.md changes * fixing tablename Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com> * fixing internal (#15315) * Fixes2 (#15316) Makes a fix to azcore in track2-tables branch * [Tables] bump deps (#15350) * Sync eng/common directory with azure-sdk-tools for PR 1912 (#15314) * Attempt to purge all vaults, managed HSMs Reverts #1910. Vaults and managed HSMs are automatically purged on their purge date. The point was to purge them daily to preserve capacity. The default purge date is +90 days. * Add timeout and more logging * Pass required -Resource * Fix log message * Ensure the $Resource is correctly captured Added comment to new code explaining why, since ScriptBlock.GetNewClosure() is not working as expected. * Add -ErrorAction to Receive-Job Worked without terminating when run locally, but failed on the first error in the AzDO agent. * Use $using:r instead of creating ScriptBlock More idiomatic for passing ScriptBlocks to jobs. * Resolve PR feedback * Change default DeleteAfterHours to 120 Resolves #1917 * Use the Az cmdlets built-in -AsJob Co-authored-by: Heath Stewart <heaths@microsoft.com> * Fix query batch processing (#15319) Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com> * fix: three dots path not work in linux in build script (#15297) * Release v56.3.0 1629350585 (#15323) * Generated from specification/apimanagement/resource-manager/readme.md tag package-preview-2021-01 (commit hash: 30d1537f0c9aa49e6e04401b1d16f612b31231e7) * Generated from specification/purview/resource-manager/readme.md tag package-2021-07-01 (commit hash: 30d1537f0c9aa49e6e04401b1d16f612b31231e7) * v56.3.0 * fix * Remove old unused update changelog script (#15326) Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com> * add clear output flag to build.ps1 (#15255) * add clear output flag * remove the path parameter and make sure the process-sdk function only runs in current directory * change flag name * [tools/generator] add automation and release command for track2 (#15271) * [tools/generator] add automation and release command for track2 * add copyright and do gofmt * Use semver to pump version, move git operation to repo package and fix some typo * Add specfic version to release cmd, seperate package for automation and remove test config file * add automation script and config for track2 * fix: three dots path not work in linux in build script * fix: ln pwsh.exe to pwsh to run in linux * fix: ignore dep check for go-git * chore: label name change for track2 * fix: wrong package path result in empty git commit * fix: repo addr replacement in autorest.md * chore: move script to eng folder and change name * fix: onboard generation need to replace NewClientMethod * chore: change all naming from track2 to v2 * fix: add new build constraint * fix: template compat with go 1.17 and change commit msg for release * chore: add v2 to readme begin regex to support future change of `track2` naming * chore: remove track2 naming * fix: duplicated and wrong expression * fix: add package title param to release cmd * fix: ci * Prevent ManagedIdentityCredential mutating GetToken arguments (#15331) * [Azcore] bumps version of internal (#15347) * bump internal to latest * upgrading internal * not the correct one * changelog update * [azidentity] bump azcore and internal dependencies (#15348) * updating to latest core and identity * trying to fix merge * undoing changes to identity that should not be there * more changes * more changes * removing a bunch more * one last file * formatting * linting checks * linting * formatting Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> Co-authored-by: Heath Stewart <heaths@microsoft.com> Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com> Co-authored-by: tadelesh <tadelesh.shi@live.cn> Co-authored-by: JiahuiPeng <46921893+804873052@users.noreply.github.com> Co-authored-by: Arcturus <dapzhang@microsoft.com> Co-authored-by: Charles Lowell <chlowe@microsoft.com> * [Tables] connection string parser (#15187) * adding conn str parser * Basic conn str parser * adding support for custom domains * invalid tests * adding self note * Update sdk/tables/aztable/connection_string.go Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com> * Update sdk/tables/aztable/connection_string.go Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com> * improving testing * finishing connection string portion * formatting and improved error handling * docstring addition for private method * connection string error * splitn instead of split * sas comment * updating proxy-server script * configure proxy with hash of file, correct file location * using certs in main * trying download with no password * trying without the pfx file altogether * adding pfx portion back * manual copy of testproxy to common * using common * fixing path * adding trust ps1 script * adding step to run proxy * formatting and go mod x/net * fixing for latest azcore * fixing rqeuire * removing eng stuff * conn str * running tests * run tests = true * adding build sourcesDir * relative path * fixing up path * build-test * moving byte array response inside table pager * env var playing to fix ci lint * changing package name Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com> * [Tables] simplify code space by moving generated to `internal` module (#15299) * changing to internal package, everything except batching works * back to working * converting more things to private * renamings * still working, checkpoint * most of tableclient done * finished table client options/responses * finished with service client * cors rule done * retentionpolicy * retentionpolicy and cors * last piece * final comments * updating with newest autorest * formatting * fixing go mod in internal package * update versions * flipping param order * convert internal to a subpackage * fixing telemetry and bumping go version * updating proxy, adding test-resources.json file * cert issues * formatting and x/net * fixing build-test * correcting file path * lint and formatting * go fmt * dropping coverage for internal * updates from pre-arch board * checking error * doing scopes work in generated code * linting * removing old file * consistent namings * removing types and simplifying pageresponse for list entities * formatting * renames, removing Table prefix, removing TableTRansactionError * fixed error type, placeholder while azcore is in PR * running tests live, fixing one test, converting to always using shared key * formatting * linter * fixing playback for storage * fixing playback credential * fixing serviceURL to expect tableName in service url * moving update mode into options for update/insert entity * return *Client from CreateTable method * cleaning up structs * flipping order of strings in conn str constructor * renaming param of transactionAction from ETag -> IfMatch * remove XMS prefix, change Value to Entities/Tables * moving GetTableSASToken to Client struct * making MinimalOdata the default, removing option to set it, new recordings * renaming ETag -> IfMatch for Options * formatting * renaming file, adding version to constants.go file * adding version * updating to latest azcore * updating autorest * formatting * updates for SAS * fixing tests, fixing conn str parser * formatting * missing error check * [Tables] Docs and cleanup (#15428) * removing to library * cleanup * fixing two tests for live * removing options, responses, and table_pagers files * removing table_ prefix on files * removing byte_array_response.go file * fixing up two reponses, adding proper comments and fixing samples * removing NewClientFromConnStr * wrong 'to' library * [Tables]Rename package (#15448)  - [ ] The purpose of this PR is explained in this or a referenced issue. - [ ] The PR does not update generated files. - These files are managed by the codegen framework at [Azure/autorest.go][]. - [ ] Tests are included and/or updated for code changes. - [ ] Updates to [CHANGELOG.md][] are included. - [ ] MIT license headers are included in each file. [Azure/autorest.go]: https://github.com/Azure/autorest.go [CHANGELOG.md]: https://github.com/Azure/azure-sdk-for-go/blob/main/CHANGELOG.md * [Tables] Merging internal (#15451) * updating * formatting Co-authored-by: Christopher Scott <chriss@microsoft.com> Co-authored-by: catalinaperalta <catalinaperaltah@hotmail.com> Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com> Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> Co-authored-by: Heath Stewart <heaths@microsoft.com> Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com> Co-authored-by: tadelesh <tadelesh.shi@live.cn> Co-authored-by: JiahuiPeng <46921893+804873052@users.noreply.github.com> Co-authored-by: Arcturus <dapzhang@microsoft.com> Co-authored-by: Charles Lowell <chlowe@microsoft.com> Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com>

* Initial generation of the aztables module (Azure#14441) * initial generation * Update ci.yml * Introduce internal/testFramework and test aztables with it (Azure#14491) * testFramework initial implementation * sanitizer fixes (Azure#14600) * Progress on aztables (Azure#14676) * implement batch operations and all entity operations * AsModels * rename * comments * comment tweaks, options as pointer * add EntityMapAsModel * refactors to AddEntity * update some code docs * doc fixes * doc fixes * doc improvements * rename testframework to recording * move ci.yml * ci path * ci path * [Tables] Update from testframework -> recording (Azure#14888) * Initial generation of the aztables module (Azure#14441) * initial generation * Update ci.yml * Introduce internal/testFramework and test aztables with it (Azure#14491) * testFramework initial implementation * sanitizer fixes (Azure#14600) * Progress on aztables (Azure#14676) * implement batch operations and all entity operations * AsModels * rename * comments * comment tweaks, options as pointer * add EntityMapAsModel * refactors to AddEntity * update some code docs * doc fixes * doc fixes * doc improvements * rename testframework to recording * fixing tables to go from testframework -> recording * renaming recordingInstance -> r * file name change Co-authored-by: Christopher Scott <chriss@microsoft.com> Co-authored-by: catalinaperalta <catalinaperaltah@hotmail.com> * [Tables] ci fix (Azure#14956) * [Tables] Add post-request validation of entity (Azure#14960) * adds a secondary validation if the entity doesnt have pk/rk * creating an errors file just for handling errors * removing extra line * moving error to constants file, renaming constants file * improving test * [Tables] changing QueryOptions param to be a ptr (Azure#14961) * changing QueryOptions param to be a ptr * feedback from catalina * [Tables] linting Fixes (Azure#15028) * small fixes * small fixes * small fixes * small fixes * small fixes * small fixes * adding error checking to transactional batching * adding error checking to test set up * adding more error checking * adding more error checking * fixing two tests * fixing error reporting * fixing error reporting * fixing error reporting * fixing last lint failure * [Tables] Adding swagger.md (Azure#15016) * adding swagger * updating swagger with autorest command * more fixes * fixing directive * updating swagger.md * fixed swagger * fixing directive * add dataplane flag * updating swagger * commented out tests, passes remaining * addressing chris hamons feedback * undoing changes * undoing the rest of changes * updating generated code * formatting gen code * more formatting corrections * one last formatting correction * [Tables] Add get/set access policy (Azure#15027) * adding a way to skip cosmos tests, adding access policies * adding support for getting/setting access policies * tests working * does not work for multiple access policies * adding post validation * added error validation, note about possible issue * set policy works * removing table access policy file * [Tables] Add GetStats, GetProperties, and SetProperties methods (Azure#15112) * adding GetProperties * basic test of set properties * added logging and metrics tests * tests for setting service properties * adding test for retention too long * adding samples and docs * [Tables] Add support for Entity struct (Azure#15003) * adding test for custom entity * further * starting access policy * working for basic entity * adding to test * new generated code, removed one more test * AddEntity now takes []bytes * adding directive to change entity to []byte, changes to compile and be able to pass * converting Value map[string]interface{} -> []byte in convenience layer * paging works the same way now too * fixing two tests, update complex and basic test entities, need to add type hints for int64/float64 types * adding test for delete method, changed delete etag to be nil * working test for merge * Added Merge test * fix up test query simple entity * complex query test * everything except for batch tests * basic batch functionality * fixing batch mixed test * fixed last batch test * added separate file for testing batch actions * fixing query on table service client to take a pointer as well * working on annotating entities * fixing recordings * working with odata annotations * working test for using odata to annotate entity * adding options to get * working with embedded entity now too * adding to entity, peeling off prints * converted to a type * first change to add EdmEntity type * updated unmarshalling * removing two files * fix editLink * removing sample * cleaning up * removing more code * cleaning up * formatting * formatting * [Tables] change Query to List (Azure#15099) * adding List methods * moving helper to zt file * improving docstrings * changing all query methods to list * updating docstrings * [Tables] Update transactions method (Azure#15125) * adding test for custom entity * further * starting access policy * working for basic entity * adding to test * new generated code, removed one more test * AddEntity now takes []bytes * adding directive to change entity to []byte, changes to compile and be able to pass * converting Value map[string]interface{} -> []byte in convenience layer * paging works the same way now too * fixing two tests, update complex and basic test entities, need to add type hints for int64/float64 types * adding test for delete method, changed delete etag to be nil * working test for merge * Added Merge test * fix up test query simple entity * complex query test * everything except for batch tests * basic batch functionality * fixing batch mixed test * fixed last batch test * added separate file for testing batch actions * fixing query on table service client to take a pointer as well * working on annotating entities * fixing recordings * working with odata annotations * working test for using odata to annotate entity * adding options to get * working with embedded entity now too * adding to entity, peeling off prints * converted to a type * first change to add EdmEntity type * updated unmarshalling * removing two files * fix editLink * removing sample * cleaning up * removing more code * cleaning up * re-record and remove methods * more cleanup * formatting * having issues with cosmos tests * Adding Table suffix to Create, Delete, and List methods on service client (Azure#15181) * [Tables] Architect Feedback (Azure#15196) * architect feedback * docstrings * feedback * fixing eng pipelines * figuring out pipeline failure * missing recording * restore eng changes * [EngSys] Test proxy (Azure#15101) * [Tables] Add authentication via SAS credential (Azure#15256) * changing recording to use test-proxy * updating client and test proxy * updated Transport Do to send actual request * whitespace * working implementation * updating internal recording * now I have a blank recording... * cleaning up * more clean up * transport Do is repeating, proxy is not parsing URL correctly * small changes * adding a policy to do the same thing * adding a second test for proxy and transport * further progress with Jeff, switching to only using a policy right now * cleaning up file * cleaning up * added more debugging, added two required headers that I was missing * proxy working on a single test, need to convert all tests to test proxy * cleaning up * moved over successfully :) * more helper functions * working for a subtest functionality * fixed up a service client test * converting more service tests * converted all service client tests * converted table client tests * fixed last two client tests * entity test * entity tests * converting access policy tests * all passing in record mode * batch tests * liveonly for batch tests * adding a sanitizer and more methods for recording * adding main method to start and stop the proxy automatically * adding test proxy step to pipeline * double dash on version arg * need a better way to get userful error reporting * explicitly running test-proxy in the background * adjust the proxy start and install to be compatible with specific process * dont run test proxy in background * didnt remove an import * header to proxy_test.go * fixes for pipeline * added loggin on accident * use nohup on linux machines * further simplify nohup usage * there are no exes on windows * fixing errors, passing in default env variable * force background for linux task. may need another iteration to place it within the quotes * fake aad credential * clean up after test-proxy * add print for record mode * another different attempt at starting the proxy * jobs apparently is an unrecognized command on our linux boxes * correct dumb typo in run tests * replace nuget install with docker * working fake credentials for client delete entity test * fixing fake credentials portion, passing locally * trying an insecureskipverify transport * debugging statements to find the cwd * removing blank recording * working implementation of sas * working sas credential for account signature * table sas is failing * leverage proxy tool, not docker * remove apostrophes * moving recordings one directory up * transitioning to using the sas written by storage team * using storage methods * copied storage code, sas needs to be appended in a different way * changing directory for test-proxy to start from * removing parenthesis * adding steps for validating ssl * switching to docker * adding start server script * working implementation if we can fix the prepending of 'recording' to the docker request finder * big refactor, moving stuff into recording file, cleaning up proxy_test.go file * last fixings * convert back to docker. windows and linux images now present * double condition * moving configuration to a separate file. preparing for eng/common move * tier 0 of trust. I don't expect this to work, but it's still worth a shot. maybe the devops agents do something differently * update condition to use variable syntax * guess I'm not using variables. removing * call start-server.ps1 * adjust build-test and configure-proxy to run the docker container in context * wrap in quotes for the container create. it's apparently a bad * adjust the initialize call * disable vet temporarily * looking for cert file in env variable PROXY_CERT * set proxy_cert environment variable to find that certificate * changing to crt file * use crt cert * want to see errors * powershell errors * correcting how the volume binds to the windows container * small modification for windows container * finally got the magic sauce * cleaning up to remove azcore from internal * removing recordings that use vcr * issue with the url creation * persist query params correctly in azcore.JoinPaths * removing print statements * return root * changing location of script * forgot the stop command * working table level sas implementation * adding start/end rk/pk functionality * removing print statements * removing more non-tables code * docstrings and removing storage only code * making sas tests live only * Fixing sas table name to be lowercase always, adding test to verify read only * Fixing sas table name to be lowercase always, adding test to verify read only * adding cosmos sas test * fix for params * apiview fixes * fixing test * undoing change * updates * error naming * fixed service client, was adding percallpolicies twice * simplification * aligning with main * converting from query pager to list pager for naming consistency * updating objects for consistency * jeffs feedback * small fixes * autorest.md changes * fixing tablename Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com> * fixing internal (Azure#15315) * Fixes2 (Azure#15316) Makes a fix to azcore in track2-tables branch * [Tables] bump deps (Azure#15350) * Sync eng/common directory with azure-sdk-tools for PR 1912 (Azure#15314) * Attempt to purge all vaults, managed HSMs Reverts Azure#1910. Vaults and managed HSMs are automatically purged on their purge date. The point was to purge them daily to preserve capacity. The default purge date is +90 days. * Add timeout and more logging * Pass required -Resource * Fix log message * Ensure the $Resource is correctly captured Added comment to new code explaining why, since ScriptBlock.GetNewClosure() is not working as expected. * Add -ErrorAction to Receive-Job Worked without terminating when run locally, but failed on the first error in the AzDO agent. * Use $using:r instead of creating ScriptBlock More idiomatic for passing ScriptBlocks to jobs. * Resolve PR feedback * Change default DeleteAfterHours to 120 Resolves Azure#1917 * Use the Az cmdlets built-in -AsJob Co-authored-by: Heath Stewart <heaths@microsoft.com> * Fix query batch processing (Azure#15319) Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com> * fix: three dots path not work in linux in build script (Azure#15297) * Release v56.3.0 1629350585 (Azure#15323) * Generated from specification/apimanagement/resource-manager/readme.md tag package-preview-2021-01 (commit hash: 30d1537f0c9aa49e6e04401b1d16f612b31231e7) * Generated from specification/purview/resource-manager/readme.md tag package-2021-07-01 (commit hash: 30d1537f0c9aa49e6e04401b1d16f612b31231e7) * v56.3.0 * fix * Remove old unused update changelog script (Azure#15326) Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com> * add clear output flag to build.ps1 (Azure#15255) * add clear output flag * remove the path parameter and make sure the process-sdk function only runs in current directory * change flag name * [tools/generator] add automation and release command for track2 (Azure#15271) * [tools/generator] add automation and release command for track2 * add copyright and do gofmt * Use semver to pump version, move git operation to repo package and fix some typo * Add specfic version to release cmd, seperate package for automation and remove test config file * add automation script and config for track2 * fix: three dots path not work in linux in build script * fix: ln pwsh.exe to pwsh to run in linux * fix: ignore dep check for go-git * chore: label name change for track2 * fix: wrong package path result in empty git commit * fix: repo addr replacement in autorest.md * chore: move script to eng folder and change name * fix: onboard generation need to replace NewClientMethod * chore: change all naming from track2 to v2 * fix: add new build constraint * fix: template compat with go 1.17 and change commit msg for release * chore: add v2 to readme begin regex to support future change of `track2` naming * chore: remove track2 naming * fix: duplicated and wrong expression * fix: add package title param to release cmd * fix: ci * Prevent ManagedIdentityCredential mutating GetToken arguments (Azure#15331) * [Azcore] bumps version of internal (Azure#15347) * bump internal to latest * upgrading internal * not the correct one * changelog update * [azidentity] bump azcore and internal dependencies (Azure#15348) * updating to latest core and identity * trying to fix merge * undoing changes to identity that should not be there * more changes * more changes * removing a bunch more * one last file * formatting * linting checks * linting * formatting Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> Co-authored-by: Heath Stewart <heaths@microsoft.com> Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com> Co-authored-by: tadelesh <tadelesh.shi@live.cn> Co-authored-by: JiahuiPeng <46921893+804873052@users.noreply.github.com> Co-authored-by: Arcturus <dapzhang@microsoft.com> Co-authored-by: Charles Lowell <chlowe@microsoft.com> * [Tables] connection string parser (Azure#15187) * adding conn str parser * Basic conn str parser * adding support for custom domains * invalid tests * adding self note * Update sdk/tables/aztable/connection_string.go Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com> * Update sdk/tables/aztable/connection_string.go Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com> * improving testing * finishing connection string portion * formatting and improved error handling * docstring addition for private method * connection string error * splitn instead of split * sas comment * updating proxy-server script * configure proxy with hash of file, correct file location * using certs in main * trying download with no password * trying without the pfx file altogether * adding pfx portion back * manual copy of testproxy to common * using common * fixing path * adding trust ps1 script * adding step to run proxy * formatting and go mod x/net * fixing for latest azcore * fixing rqeuire * removing eng stuff * conn str * running tests * run tests = true * adding build sourcesDir * relative path * fixing up path * build-test * moving byte array response inside table pager * env var playing to fix ci lint * changing package name Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com> * [Tables] simplify code space by moving generated to `internal` module (Azure#15299) * changing to internal package, everything except batching works * back to working * converting more things to private * renamings * still working, checkpoint * most of tableclient done * finished table client options/responses * finished with service client * cors rule done * retentionpolicy * retentionpolicy and cors * last piece * final comments * updating with newest autorest * formatting * fixing go mod in internal package * update versions * flipping param order * convert internal to a subpackage * fixing telemetry and bumping go version * updating proxy, adding test-resources.json file * cert issues * formatting and x/net * fixing build-test * correcting file path * lint and formatting * go fmt * dropping coverage for internal * updates from pre-arch board * checking error * doing scopes work in generated code * linting * removing old file * consistent namings * removing types and simplifying pageresponse for list entities * formatting * renames, removing Table prefix, removing TableTRansactionError * fixed error type, placeholder while azcore is in PR * running tests live, fixing one test, converting to always using shared key * formatting * linter * fixing playback for storage * fixing playback credential * fixing serviceURL to expect tableName in service url * moving update mode into options for update/insert entity * return *Client from CreateTable method * cleaning up structs * flipping order of strings in conn str constructor * renaming param of transactionAction from ETag -> IfMatch * remove XMS prefix, change Value to Entities/Tables * moving GetTableSASToken to Client struct * making MinimalOdata the default, removing option to set it, new recordings * renaming ETag -> IfMatch for Options * formatting * renaming file, adding version to constants.go file * adding version * updating to latest azcore * updating autorest * formatting * updates for SAS * fixing tests, fixing conn str parser * formatting * missing error check * [Tables] Docs and cleanup (Azure#15428) * removing to library * cleanup * fixing two tests for live * removing options, responses, and table_pagers files * removing table_ prefix on files * removing byte_array_response.go file * fixing up two reponses, adding proper comments and fixing samples * removing NewClientFromConnStr * wrong 'to' library * [Tables]Rename package (Azure#15448)  - [ ] The purpose of this PR is explained in this or a referenced issue. - [ ] The PR does not update generated files. - These files are managed by the codegen framework at [Azure/autorest.go][]. - [ ] Tests are included and/or updated for code changes. - [ ] Updates to [CHANGELOG.md][] are included. - [ ] MIT license headers are included in each file. [Azure/autorest.go]: https://github.com/Azure/autorest.go [CHANGELOG.md]: https://github.com/Azure/azure-sdk-for-go/blob/main/CHANGELOG.md * [Tables] Merging internal (Azure#15451) * updating * formatting Co-authored-by: Christopher Scott <chriss@microsoft.com> Co-authored-by: catalinaperalta <catalinaperaltah@hotmail.com> Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com> Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> Co-authored-by: Heath Stewart <heaths@microsoft.com> Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com> Co-authored-by: tadelesh <tadelesh.shi@live.cn> Co-authored-by: JiahuiPeng <46921893+804873052@users.noreply.github.com> Co-authored-by: Arcturus <dapzhang@microsoft.com> Co-authored-by: Charles Lowell <chlowe@microsoft.com> Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com>

christothes added 6 commits May 17, 2021 10:47

merge conflicts

0ea3fa2

batch working with Storage

6358502

batch working

02881ec

sort batch headers for recording consistency

84d21c3

batch response parsing

8d29b40

christothes requested review from catalinaperalta and jhendrixMSFT as code owners May 18, 2021 22:00

christothes self-assigned this May 18, 2021

christothes commented May 18, 2021

View reviewed changes

catalinaperalta reviewed May 19, 2021

View reviewed changes

christothes added 7 commits May 19, 2021 10:19

update mod and sum

031b0ec

newline

f1dc560

ioutil

96c9314

renames

c077d26

changelog version

6411bd3

batch error parsing, need to fix mixed batch

9a890a2

fixed batch - need to add cosmos merge operation policy

fe45615

christothes added 7 commits May 21, 2021 12:19

mixed batches

719341c

fmt

9d2274f

batch errors

6fa31f0

DeleteEntity

ba52922

update and upsert

f9b8273

upsert

8e07904

GetEntity - with generated code issue

d72bee5

catalinaperalta approved these changes May 21, 2021

View reviewed changes

christothes added 2 commits May 21, 2021 17:55

regenerate client

6c8b3a8

fb

e7f7059

christothes merged commit afe8e9e into Azure:track2-tables May 21, 2021

christothes added a commit that referenced this pull request Jun 28, 2021

Progress on aztables (#14676)

fed39eb

* implement batch operations and all entity operations

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Progress on aztables #14676

Progress on aztables #14676

christothes commented May 18, 2021

christothes May 18, 2021

catalinaperalta May 19, 2021

christothes May 19, 2021

christothes May 18, 2021 •

edited

Loading

catalinaperalta May 19, 2021

catalinaperalta May 19, 2021

christothes May 19, 2021

catalinaperalta May 19, 2021

catalinaperalta May 19, 2021

christothes May 19, 2021

check-enforcer bot commented May 21, 2021

catalinaperalta left a comment

catalinaperalta May 21, 2021

christothes May 21, 2021

Progress on aztables #14676

Progress on aztables #14676

Conversation

christothes commented May 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christothes May 18, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

check-enforcer bot commented May 21, 2021

catalinaperalta left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christothes May 18, 2021 •

edited

Loading