Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merging the latest master changes into the spet_release branch. (#1294)
* Updating issue template for adding issue SLO (#1216) * adding bug SLO * adding bug SLO * Fixing comment * Integration tests --key-file flag and GOOGLE_APPLICATION_CREDENTIALS env with admin permission tests (#1167) * updating go version * empty commit * local commit * local changes * local changes * local changes * adding key file tests * testing * testing * testing * testing * local changes * local changes * local changes * local changes * testing * testing * testing * testing * testing * adding test for admin creds * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * formating * testing defer statement * testing defer statement for deleting credentials * adding comment * testing with error * testing with error * testing with error * removing testing statement * adding testbucket and mntdir in commnd * adding comment * updating bucket name * updating bucket name * removing unnecessary changes * removing unnecessary changes * removing unnecessary changes * formatting * conflict * adding error handling * testing * small fix * removing creds tests from implicit and explicit dir tests * testing * testing * testing * testing * removing testing statement * adding creds tests in operations back * Testing * Testing * Testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * adding remaining changes * adding remaining changes * adding remaining changes * testing service account * testing service account * testing service account * adding comments * removing unnecessary changes * formatting * testing * testing * testing * testing * removing without key file tests * small fix * formalizing for reuse * small fix * removing unnecessary changes * formatting * updating comment * updating comment * updating comment * fixing comments * adding comment * testing * testing * adding condintion for service account already exsit * adding condintion for service account already exsit * testing time * running tests only for operations * Changing the number of epoch based on previous observation. (#1222) * Adding rpm digest while creating rpm package (#1215) * removing defer as not working properly in for loop (#1223) * Added CD pipeline scripts (#1224) * Added scripts * review comments * move cd_scripts to tools/ * Integration tests for key-file flag with viewer permission tests (#1228) * testing with viewer permission * testing on operations * testing on operations * final testing * testing * testing * Get dir attribute integration tests (#1220) * adding test for dir attribute * renaming * fixing bytes comparison * adding more tests * adding sub dir attribute tests * fix failure * adding comment * adding const variable * removing unnecessary file * fixing comment * fix lint * Persistent mounting integration tests (#1218) * adding test for persistent mounting * adding test for persistent mounting * adding test for persistent mounting * adding test for persistent mounting * Testing * Testing * Testing * Testing * Testing * Testing * Testing * Testing * Testing * testing * formating argument * adding -o flag * adding mounting test in every package * small fix * removing --o in * removing --o in * adding commands to run tests for mountedDirectory flag * small fix * fixing commands * adding comment * small fix * small fix * adding comment * adding comments and small fixed * fixing lint * fixing merge conflict * smallfi x * testing * testing * small fix * removing unnecessary commands * removing extra line * removing extra line * Move directory integration tests (#1230) * move dir tests * test move dir tests * test move dir tests * small fixed * fixing typo * fixed comment * Add copyright file (#1225) * added copyright file added debian directory lintian fixes Updated main dockerfile * review comments * use gcs-fuse-maintainers@google.com email * Kokoro perf test fix - Integration tests (#1239) * testing * testing * testing * adding comment for a reason * creating function to copy dir with root permission * common function * fix * testing * revert changes * revert changes * adding remove package * adding remove package * minor changes to test script (#1238) * Sequentially read large file (#1233) * testing sequential read tests * testing sequential read tests * testing sequential read tests * testing sequential read tests 100Mb * adding it in other separate package as it is taking more time * adding comments and fixing lint * adding comments * adding comments * adding comments * adding commands in script * formating * adding commands in script * fixing comments * small fix * removing unnecessary commands * small fix * adding remove copied bin dir function call * formating * updating comment * Creds test error fixing Integration tests (#1241) * testing * testing * testing * testing * creating service account if does not exist * removing unnecessary params * formatting * formatting * adding relavant comments * updating path * fixing comment * adding command for removing file * small fix in script to update path of file * Upgrade to go version 1.20.5 (#1196) Brief list of changes * Changes gcsfuse binary to be statically-linked, removes dependency on libc. * Prefixed/exported CGO_ENABLED=0 for every go build|run|test for every gcsfuse program * Replaced all golang:1.20.4 based base images to corresponding golang:1.20.5 based images in all docker files and all docker run commands * Updated golang 1.20.4 dependencies to golang 1.20.5 * Updated troubleshooting guide * Replaced os.user.Current() with os.Getuid() and os.Getgid() to remove dependency on glibc * Added unit tests for perms.go * Removed libc dependency from DEBIAN/CONTROL * Dynamic mounting integration tests (#1242) * testing * testing * adding dynamic mounting testing * testing * testing * separating unmounting * small fix * unblocking remaning tests * small fix(updating echo message) * fixing lint tests and adding comments * updating file path * small fix * testing * testing * testing * unblocking code for testing * testing * increasing time as taking more than 10m to run * adding random suffix at the end of bucket name as with fixed bucket it can fail when multiple instance run inetgration tests at a time * assigning it to variable * small fix * small fix * updating const var name * testing * testing * unblocking code * updating comment * minor change to test script (#1243) * Sequential write large file integration tests (#1240) * adding test for writing file of 100MB * adding test for writing file of 100MB * adding commands in script * adding commands in script * small fix * increasing chunk size * small fix * using min function * small fix * adding comment * fixing lint * writing 500MB in more redable way * fixing comments * fixing comments * changing chunksize to 20MB * testing sync * testing * merging * merging * testing sync * small fix * Kokoro perf test fix - gcloud storage command not working (#1248) * Update semantics.md -- fix typos (#1251) * Random read integration tests (#1247) * adding tests for random read files * fixing comments * creating common function * small fix * fixing comment * creating common function * small fix * merge * fixing comments and fixing typos in naming * renaming function name * renaming function name * Read multiple files at a time integration tests (#1249) * adding tests * updating comment * fixing comments and fixing typos in naming * renaming function name * renaming * fix lint test * Update gcloud version (#1259) Update jacobsa/gcloud dependency version This is to account for jacobsa/gcloud#34 . Also minor gitignore change: Ignore vscode settings folder (.vscode) * Removing rate limit dependency (#1253) * local changes * local changes * local changes * local changes * removed token bucket and added unit tests * fixing lint * small fix- renaming * small fix- renaming * fix lint * adding licence * testing * back to changes * fixing comments * removing throttle_test * adding throttle test * fixing comment * fixing lint * fixed comment * lint tests * lint tests * fixing comment * empty commit * empty commit * Upgrading go-client module from 1.29.0 to 1.31.0 (#1263) * Upgrading go-client module from 1.29.0 to 1.31.0 * Resolving merge conflict * Bump requests in /perfmetrics/scripts/load_tests/python (#1268) Bumps [requests](https://github.com/psf/requests) from 2.30.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.30.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.30.0 to 2.31.0 in /perfmetrics/scripts (#1267) Bumps [requests](https://github.com/psf/requests) from 2.30.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.30.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump certifi in /perfmetrics/scripts/load_tests/python (#1269) Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updating fuse library version (#1273) * updated architecture to all (#1270) * Bump certifi from 2023.5.7 to 2023.7.22 in /perfmetrics/scripts (#1266) Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Read gzip content-encoding objects as compressed (#1255) Enable read-compressed for gzip objects Details of changes * Add field contentEncoding in internal.storage.MinObject * Passes ReadCompressed as true for objects with contentEncoding set to "gzip" * Added corresponding unit tests. * Update fsouza/fake-gcs-server to v1.40.3 - to fix the read-behavior of objects with content-encoding gzip in fake-server, used in unit tests. * gzip integration tests (#1256) Add read-flow integration tests for gzip objects This adds read-flow integration tests for the gzip read-compressed changes added in previous PR #1255 . Tests added (2*5 = 10 tests) * Two types of operations each * List, Stat and full-file read * On five types of objects each * Object with text content, uploaded with gzip compression with content-encoding: gzip, and cache-control: no-transform * Object with text content, uploaded with gzip compression with content-encoding: gzip, and cache-control: '' * Object with gzip content, uploaded with content-encoding: '' * Object with gzip content, uploaded with gzip compression (i.e. doubly compressed) with content-encoding: gzip, and cache-control: no-transform * Object with gzip content, uploaded with gzip compression (i.e. doubly compressed) with content-encoding: gzip, and cache-control: '' Write-flow tests will be added in another commit shortly after. This commit also adds some helper functions needed for gzip integration tests Functions added * CreateLocalTempFile - creates a temporary local file of given size, gzip/non-gzip * DiffFiles * GetGcsObjectSize * DownloadGcsObject * UploadGcsObject * DeleteGcsObject * ClearCacheControlOnGcsObject * Add gzip write-flow tests (#1262) It adds write-flow integration tests for GCSfuse-mounted objects with content-encoding gzip. These tests replace an existing gzip object in a mounted bucket, then verify the size of the re-uploaded GCS object. Tests added (1*5 = 5 tests) * One type of operation * Overwrite * On five types of objects each * Object with text content, uploaded with gzip compression with content-encoding: gzip, and cache-control: no-transform * Object with text content, uploaded with gzip compression with content-encoding: gzip, and cache-control: '' * Object with gzip content, uploaded with content-encoding: '' * Object with gzip content, uploaded with gzip compression (i.e. doubly compressed) with content-encoding: gzip, and cache-control: no-transform * Object with gzip content, uploaded with gzip compression (i.e. doubly compressed) with content-encoding: gzip, and cache-control: '' This also adds a necessary CopyFileAllowOverwrite alternative utility to the existing CopyFile utility(which doesn't allow file overwrite) in tools/integration_tests/util/operations/file_operations.go . * Fix kokoro job failure for gzip integration tests (#1276) Kokoro VM is based on a specialized linux image created by the Kokoro team for running jobs on it. This has an old version of gcloud on it, and thus the gcsfuse integration test jobs use `gcloud alpha` on it as opposed to `gcloud` command. To make matters worse, on the kokoro VM, `gcloud alpha` doesn't support the `-Z` option as well as the options for updating object metadata. Both these features are required for integration tests for GCS objects with content-encoding gzip in GCSFuse. Hence, for both these features, switching from `gcloud alpha` to equivalent `gsutil` commands. * fixing requirements (#1281) * Replace gcloud with gsutil in file-op utils (#1280) Replaces `gcloud` or `gcloud alpha` with `gsutil` in all gcs related utilities in file_operations utils in integration_tests. This is to avoid failures related to compatibility in kokoro VMs which support an old version of gcloud cli. * Deprecate --enable-storage-client-library flag (#1258) * local changes * local changes * local changes * local changes * removed token bucket and added unit tests * fixing lint * small fix- renaming * small fix- renaming * fix lint * adding licence * testing * back to changes * fixing comments * removing throttle_test * depricating from enable storage client library flag * temporary using env variable to set end point * fixing linux tests * adding throttle test * fixing comment * fixing lint * fixed comment * lint tests * lint tests * removing endpoint changes * removing endpoint changes * removing unnecessary changes * removing connection * fixing comment * empty commit * empty commit * removing endponti changes * fixing comments * formating * removing debug http * fixing linux tests * adding back debug_http flag * fixing comments * renaming function * fixing comments * fixing comments * fixing comments in integration test scrip * fixing comments in integration test scrip * fixing comments in integration test scrip * fixing comments in integration test script * updating vendor client * fixing comment * removing concurrent read * empty commit * replacing mountWithConn to mountWithStorageHandle * empty commit * testing * testing * back to original changes * Enabling --endpoint flag for go-storage-client library (#1264) * Enabling --endpoint flag for go-storage-client library * Fixing lint issue * Added more unit test * fixing end pointd escription * Refactoring and resolving comments * refactoring * Reverting previous refactoring, created client_helper for all client creation logic * treating passing actual GCSURL like custom point, by design in go-storage-lib * Fixing unit test, removing unused token-source * Minor refactoring, adding dummy key-file * Making the default client non-nil so that we can get dummy token-source * Fixing unit test * Review comments updated * Review comments * Incorporating review comments * Fixing build issue * Incorporating review comments * Renamed client helper to client * Incorporating review comments * Adding more unit test * Review comments * Fixing flag unit test * Fixed gcsfuse hang * Triggering performance test and fixing unit test * Changing the timeout from 15m to 24m for integraotin test of one package (#1284) * Changing the timeout from 15m to 24m for integraotin test of one package * Adding the timout flag for continuous test also * Added an experimental flag to trigger JSON flow of go-storage-client (#1265) * Enabling --endpoint flag for go-storage-client library * Fixing lint issue * Added more unit test * fixing end pointd escription * Refactoring and resolving comments * refactoring * Reverting previous refactoring, created client_helper for all client creation logic * treating passing actual GCSURL like custom point, by design in go-storage-lib * Fixing unit test, removing unused token-source * Minor refactoring, adding dummy key-file * Making the default client non-nil so that we can get dummy token-source * Fixing unit test * Review comments updated * Review comments * Incorporating review comments * Fixing build issue * Incorporating review comments * Renamed client helper to client * Incorporating review comments * Adding more unit test * Review comments * Support json read in GCSFuse * Rebased with support endpoint branch * Incorporating review comments * reverting formatting issue * Fixing build issue * Fixing build failure * Correcting a comment * Fixing typo * Enabling --endpoint flag for go-storage-client library * Fixing lint issue * Added more unit test * fixing end pointd escription * Refactoring and resolving comments * refactoring * Reverting previous refactoring, created client_helper for all client creation logic * treating passing actual GCSURL like custom point, by design in go-storage-lib * Fixing unit test, removing unused token-source * Minor refactoring, adding dummy key-file * Making the default client non-nil so that we can get dummy token-source * Fixing unit test * Review comments updated * Review comments * Incorporating review comments * Fixing build issue * Incorporating review comments * Renamed client helper to client * Incorporating review comments * Adding more unit test * Review comments * Fixing flag unit test * Fixed gcsfuse hang * Triggering performance test and fixing unit test * Fixing linting issue * Trigger perf test * Added equivalent command in mounted-directory run test * Fixing review comments: replacing printf error log to returning error (#1285) * Updating gcsufse --help (#1286) * Adding changes to run integration and perf tests with label (#1283) * adding changes to separate integration tests and perf tests * fix braces error * fix random string error * testing * remove changes * merge * merge * fixing comments * fixing comments * fixing comments * fixing comments * testing * testing * echo bucket name * testing by adding git stash * removing testing commanda * testing by adding git stash * testing by adding git stash * testing by adding git stash * testing by adding git stash * testing kokoro * testing kokoro * testing kokoro * testing kokoro * undo testing changes * fixing comment * testing with both the labels * removing testing changes * removing comment * Removing extra logs in kokoro continous test (#1289) * removing extra logs of go tar * removing extra logs of go tar * timeout increase for linting issue * Revert "timeout increase for linting issue" This reverts commit 8d5e9af. * Adding yaml configurations * Trigger CI integration test * Trigger CI integration test1 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Tulsishah <46474643+Tulsishah@users.noreply.github.com> Co-authored-by: Ashmeen Kaur <57195160+ashmeenkaur@users.noreply.github.com> Co-authored-by: Nitin Garg <gargnitin@google.com> Co-authored-by: Nitin Garg <113666283+gargnitingoogle@users.noreply.github.com> Co-authored-by: Minwoo Byeon (Dylan) <mwbyun@hotmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
- Loading branch information
7 people
authored
Aug 23, 2023
1 parent
765092f
commit 7282937