Merging the latest master changes into the spet_release branch. #1294
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* adding bug SLO * adding bug SLO * Fixing comment
…env with admin permission tests (#1167) * updating go version * empty commit * local commit * local changes * local changes * local changes * adding key file tests * testing * testing * testing * testing * local changes * local changes * local changes * local changes * testing * testing * testing * testing * testing * adding test for admin creds * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * formating * testing defer statement * testing defer statement for deleting credentials * adding comment * testing with error * testing with error * testing with error * removing testing statement * adding testbucket and mntdir in commnd * adding comment * updating bucket name * updating bucket name * removing unnecessary changes * removing unnecessary changes * removing unnecessary changes * formatting * conflict * adding error handling * testing * small fix * removing creds tests from implicit and explicit dir tests * testing * testing * testing * testing * removing testing statement * adding creds tests in operations back * Testing * Testing * Testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * adding remaining changes * adding remaining changes * adding remaining changes * testing service account * testing service account * testing service account * adding comments * removing unnecessary changes * formatting * testing * testing * testing * testing * removing without key file tests * small fix * formalizing for reuse * small fix * removing unnecessary changes * formatting * updating comment * updating comment * updating comment * fixing comments * adding comment * testing * testing * adding condintion for service account already exsit * adding condintion for service account already exsit * testing time * running tests only for operations
* Added scripts * review comments * move cd_scripts to tools/
* testing with viewer permission * testing on operations * testing on operations * final testing * testing * testing
* adding test for dir attribute * renaming * fixing bytes comparison * adding more tests * adding sub dir attribute tests * fix failure * adding comment * adding const variable * removing unnecessary file * fixing comment * fix lint
* adding test for persistent mounting * adding test for persistent mounting * adding test for persistent mounting * adding test for persistent mounting * Testing * Testing * Testing * Testing * Testing * Testing * Testing * Testing * Testing * testing * formating argument * adding -o flag * adding mounting test in every package * small fix * removing --o in * removing --o in * adding commands to run tests for mountedDirectory flag * small fix * fixing commands * adding comment * small fix * small fix * adding comment * adding comments and small fixed * fixing lint * fixing merge conflict * smallfi x * testing * testing * small fix * removing unnecessary commands * removing extra line * removing extra line
* move dir tests * test move dir tests * test move dir tests * small fixed * fixing typo * fixed comment
* added copyright file added debian directory lintian fixes Updated main dockerfile * review comments * use gcs-fuse-maintainers@google.com email
* testing * testing * testing * adding comment for a reason * creating function to copy dir with root permission * common function * fix * testing * revert changes * revert changes * adding remove package * adding remove package
* testing sequential read tests * testing sequential read tests * testing sequential read tests * testing sequential read tests 100Mb * adding it in other separate package as it is taking more time * adding comments and fixing lint * adding comments * adding comments * adding comments * adding commands in script * formating * adding commands in script * fixing comments * small fix * removing unnecessary commands * small fix * adding remove copied bin dir function call * formating * updating comment
* testing * testing * testing * testing * creating service account if does not exist * removing unnecessary params * formatting * formatting * adding relavant comments * updating path * fixing comment * adding command for removing file * small fix in script to update path of file
Brief list of changes * Changes gcsfuse binary to be statically-linked, removes dependency on libc. * Prefixed/exported CGO_ENABLED=0 for every go build|run|test for every gcsfuse program * Replaced all golang:1.20.4 based base images to corresponding golang:1.20.5 based images in all docker files and all docker run commands * Updated golang 1.20.4 dependencies to golang 1.20.5 * Updated troubleshooting guide * Replaced os.user.Current() with os.Getuid() and os.Getgid() to remove dependency on glibc * Added unit tests for perms.go * Removed libc dependency from DEBIAN/CONTROL
…g-1.20.5 Upgrade to go version 1.20.5 (#1196)
* testing * testing * adding dynamic mounting testing * testing * testing * separating unmounting * small fix * unblocking remaning tests * small fix(updating echo message) * fixing lint tests and adding comments * updating file path * small fix * testing * testing * testing * unblocking code for testing * testing * increasing time as taking more than 10m to run * adding random suffix at the end of bucket name as with fixed bucket it can fail when multiple instance run inetgration tests at a time * assigning it to variable * small fix * small fix * updating const var name * testing * testing * unblocking code * updating comment
* adding test for writing file of 100MB * adding test for writing file of 100MB * adding commands in script * adding commands in script * small fix * increasing chunk size * small fix * using min function * small fix * adding comment * fixing lint * writing 500MB in more redable way * fixing comments * fixing comments * changing chunksize to 20MB * testing sync * testing * merging * merging * testing sync * small fix
* adding tests for random read files * fixing comments * creating common function * small fix * fixing comment * creating common function * small fix * merge * fixing comments and fixing typos in naming * renaming function name * renaming function name
* adding tests * updating comment * fixing comments and fixing typos in naming * renaming function name * renaming * fix lint test
Update jacobsa/gcloud dependency version This is to account for jacobsa/gcloud#34 . Also minor gitignore change: Ignore vscode settings folder (.vscode)
* local changes * local changes * local changes * local changes * removed token bucket and added unit tests * fixing lint * small fix- renaming * small fix- renaming * fix lint * adding licence * testing * back to changes * fixing comments * removing throttle_test * adding throttle test * fixing comment * fixing lint * fixed comment * lint tests * lint tests * fixing comment * empty commit * empty commit
* Upgrading go-client module from 1.29.0 to 1.31.0 * Resolving merge conflict
Bumps [requests](https://github.com/psf/requests) from 2.30.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.30.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [requests](https://github.com/psf/requests) from 2.30.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.30.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
It adds write-flow integration tests for GCSfuse-mounted objects with content-encoding gzip. These tests replace an existing gzip object in a mounted bucket, then verify the size of the re-uploaded GCS object. Tests added (1*5 = 5 tests) * One type of operation * Overwrite * On five types of objects each * Object with text content, uploaded with gzip compression with content-encoding: gzip, and cache-control: no-transform * Object with text content, uploaded with gzip compression with content-encoding: gzip, and cache-control: '' * Object with gzip content, uploaded with content-encoding: '' * Object with gzip content, uploaded with gzip compression (i.e. doubly compressed) with content-encoding: gzip, and cache-control: no-transform * Object with gzip content, uploaded with gzip compression (i.e. doubly compressed) with content-encoding: gzip, and cache-control: '' This also adds a necessary CopyFileAllowOverwrite alternative utility to the existing CopyFile utility(which doesn't allow file overwrite) in tools/integration_tests/util/operations/file_operations.go .
Kokoro VM is based on a specialized linux image created by the Kokoro team for running jobs on it. This has an old version of gcloud on it, and thus the gcsfuse integration test jobs use `gcloud alpha` on it as opposed to `gcloud` command. To make matters worse, on the kokoro VM, `gcloud alpha` doesn't support the `-Z` option as well as the options for updating object metadata. Both these features are required for integration tests for GCS objects with content-encoding gzip in GCSFuse. Hence, for both these features, switching from `gcloud alpha` to equivalent `gsutil` commands.
Replaces `gcloud` or `gcloud alpha` with `gsutil` in all gcs related utilities in file_operations utils in integration_tests. This is to avoid failures related to compatibility in kokoro VMs which support an old version of gcloud cli.
* local changes * local changes * local changes * local changes * removed token bucket and added unit tests * fixing lint * small fix- renaming * small fix- renaming * fix lint * adding licence * testing * back to changes * fixing comments * removing throttle_test * depricating from enable storage client library flag * temporary using env variable to set end point * fixing linux tests * adding throttle test * fixing comment * fixing lint * fixed comment * lint tests * lint tests * removing endpoint changes * removing endpoint changes * removing unnecessary changes * removing connection * fixing comment * empty commit * empty commit * removing endponti changes * fixing comments * formating * removing debug http * fixing linux tests * adding back debug_http flag * fixing comments * renaming function * fixing comments * fixing comments * fixing comments in integration test scrip * fixing comments in integration test scrip * fixing comments in integration test scrip * fixing comments in integration test script * updating vendor client * fixing comment * removing concurrent read * empty commit * replacing mountWithConn to mountWithStorageHandle * empty commit * testing * testing * back to original changes
* Enabling --endpoint flag for go-storage-client library * Fixing lint issue * Added more unit test * fixing end pointd escription * Refactoring and resolving comments * refactoring * Reverting previous refactoring, created client_helper for all client creation logic * treating passing actual GCSURL like custom point, by design in go-storage-lib * Fixing unit test, removing unused token-source * Minor refactoring, adding dummy key-file * Making the default client non-nil so that we can get dummy token-source * Fixing unit test * Review comments updated * Review comments * Incorporating review comments * Fixing build issue * Incorporating review comments * Renamed client helper to client * Incorporating review comments * Adding more unit test * Review comments * Fixing flag unit test * Fixed gcsfuse hang * Triggering performance test and fixing unit test
…age (#1284) * Changing the timeout from 15m to 24m for integraotin test of one package * Adding the timout flag for continuous test also
…1265) * Enabling --endpoint flag for go-storage-client library * Fixing lint issue * Added more unit test * fixing end pointd escription * Refactoring and resolving comments * refactoring * Reverting previous refactoring, created client_helper for all client creation logic * treating passing actual GCSURL like custom point, by design in go-storage-lib * Fixing unit test, removing unused token-source * Minor refactoring, adding dummy key-file * Making the default client non-nil so that we can get dummy token-source * Fixing unit test * Review comments updated * Review comments * Incorporating review comments * Fixing build issue * Incorporating review comments * Renamed client helper to client * Incorporating review comments * Adding more unit test * Review comments * Support json read in GCSFuse * Rebased with support endpoint branch * Incorporating review comments * reverting formatting issue * Fixing build issue * Fixing build failure * Correcting a comment * Fixing typo * Enabling --endpoint flag for go-storage-client library * Fixing lint issue * Added more unit test * fixing end pointd escription * Refactoring and resolving comments * refactoring * Reverting previous refactoring, created client_helper for all client creation logic * treating passing actual GCSURL like custom point, by design in go-storage-lib * Fixing unit test, removing unused token-source * Minor refactoring, adding dummy key-file * Making the default client non-nil so that we can get dummy token-source * Fixing unit test * Review comments updated * Review comments * Incorporating review comments * Fixing build issue * Incorporating review comments * Renamed client helper to client * Incorporating review comments * Adding more unit test * Review comments * Fixing flag unit test * Fixed gcsfuse hang * Triggering performance test and fixing unit test * Fixing linting issue * Trigger perf test * Added equivalent command in mounted-directory run test
* adding changes to separate integration tests and perf tests * fix braces error * fix random string error * testing * remove changes * merge * merge * fixing comments * fixing comments * fixing comments * fixing comments * testing * testing * echo bucket name * testing by adding git stash * removing testing commanda * testing by adding git stash * testing by adding git stash * testing by adding git stash * testing by adding git stash * testing kokoro * testing kokoro * testing kokoro * testing kokoro * undo testing changes * fixing comment * testing with both the labels * removing testing changes * removing comment
* removing extra logs of go tar * removing extra logs of go tar
This reverts commit 8d5e9af.
raj-prince
requested review from
sethiay,
vadlakondaswetha,
Tulsishah,
gargnitingoogle and
ashmeenkaur
raj-prince
changed the title
sethiay
approved these changes
Aug 23, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Link to the issue in case of a bug fix.
NA
Testing details