Changing the number of epoch based on previous run #1222
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sethiay
approved these changes
Jul 12, 2023
vadlakondaswetha
added a commit
that referenced
this pull request
Jul 18, 2023
* Updating issue template for adding issue SLO (#1216) * adding bug SLO * adding bug SLO * Fixing comment * Integration tests --key-file flag and GOOGLE_APPLICATION_CREDENTIALS env with admin permission tests (#1167) * updating go version * empty commit * local commit * local changes * local changes * local changes * adding key file tests * testing * testing * testing * testing * local changes * local changes * local changes * local changes * testing * testing * testing * testing * testing * adding test for admin creds * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * formating * testing defer statement * testing defer statement for deleting credentials * adding comment * testing with error * testing with error * testing with error * removing testing statement * adding testbucket and mntdir in commnd * adding comment * updating bucket name * updating bucket name * removing unnecessary changes * removing unnecessary changes * removing unnecessary changes * formatting * conflict * adding error handling * testing * small fix * removing creds tests from implicit and explicit dir tests * testing * testing * testing * testing * removing testing statement * adding creds tests in operations back * Testing * Testing * Testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * adding remaining changes * adding remaining changes * adding remaining changes * testing service account * testing service account * testing service account * adding comments * removing unnecessary changes * formatting * testing * testing * testing * testing * removing without key file tests * small fix * formalizing for reuse * small fix * removing unnecessary changes * formatting * updating comment * updating comment * updating comment * fixing comments * adding comment * testing * testing * adding condintion for service account already exsit * adding condintion for service account already exsit * testing time * running tests only for operations * Changing the number of epoch based on previous observation. (#1222) * Adding rpm digest while creating rpm package (#1215) * removing defer as not working properly in for loop (#1223) --------- Co-authored-by: Tulsishah <46474643+Tulsishah@users.noreply.github.com> Co-authored-by: Prince Kumar <princer@google.com>
raj-prince
added a commit
that referenced
this pull request
Aug 23, 2023
* Updating issue template for adding issue SLO (#1216) * adding bug SLO * adding bug SLO * Fixing comment * Integration tests --key-file flag and GOOGLE_APPLICATION_CREDENTIALS env with admin permission tests (#1167) * updating go version * empty commit * local commit * local changes * local changes * local changes * adding key file tests * testing * testing * testing * testing * local changes * local changes * local changes * local changes * testing * testing * testing * testing * testing * adding test for admin creds * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * formating * testing defer statement * testing defer statement for deleting credentials * adding comment * testing with error * testing with error * testing with error * removing testing statement * adding testbucket and mntdir in commnd * adding comment * updating bucket name * updating bucket name * removing unnecessary changes * removing unnecessary changes * removing unnecessary changes * formatting * conflict * adding error handling * testing * small fix * removing creds tests from implicit and explicit dir tests * testing * testing * testing * testing * removing testing statement * adding creds tests in operations back * Testing * Testing * Testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * adding remaining changes * adding remaining changes * adding remaining changes * testing service account * testing service account * testing service account * adding comments * removing unnecessary changes * formatting * testing * testing * testing * testing * removing without key file tests * small fix * formalizing for reuse * small fix * removing unnecessary changes * formatting * updating comment * updating comment * updating comment * fixing comments * adding comment * testing * testing * adding condintion for service account already exsit * adding condintion for service account already exsit * testing time * running tests only for operations * Changing the number of epoch based on previous observation. (#1222) * Adding rpm digest while creating rpm package (#1215) * removing defer as not working properly in for loop (#1223) * Added CD pipeline scripts (#1224) * Added scripts * review comments * move cd_scripts to tools/ * Integration tests for key-file flag with viewer permission tests (#1228) * testing with viewer permission * testing on operations * testing on operations * final testing * testing * testing * Get dir attribute integration tests (#1220) * adding test for dir attribute * renaming * fixing bytes comparison * adding more tests * adding sub dir attribute tests * fix failure * adding comment * adding const variable * removing unnecessary file * fixing comment * fix lint * Persistent mounting integration tests (#1218) * adding test for persistent mounting * adding test for persistent mounting * adding test for persistent mounting * adding test for persistent mounting * Testing * Testing * Testing * Testing * Testing * Testing * Testing * Testing * Testing * testing * formating argument * adding -o flag * adding mounting test in every package * small fix * removing --o in * removing --o in * adding commands to run tests for mountedDirectory flag * small fix * fixing commands * adding comment * small fix * small fix * adding comment * adding comments and small fixed * fixing lint * fixing merge conflict * smallfi x * testing * testing * small fix * removing unnecessary commands * removing extra line * removing extra line * Move directory integration tests (#1230) * move dir tests * test move dir tests * test move dir tests * small fixed * fixing typo * fixed comment * Add copyright file (#1225) * added copyright file added debian directory lintian fixes Updated main dockerfile * review comments * use gcs-fuse-maintainers@google.com email * Kokoro perf test fix - Integration tests (#1239) * testing * testing * testing * adding comment for a reason * creating function to copy dir with root permission * common function * fix * testing * revert changes * revert changes * adding remove package * adding remove package * minor changes to test script (#1238) * Sequentially read large file (#1233) * testing sequential read tests * testing sequential read tests * testing sequential read tests * testing sequential read tests 100Mb * adding it in other separate package as it is taking more time * adding comments and fixing lint * adding comments * adding comments * adding comments * adding commands in script * formating * adding commands in script * fixing comments * small fix * removing unnecessary commands * small fix * adding remove copied bin dir function call * formating * updating comment * Creds test error fixing Integration tests (#1241) * testing * testing * testing * testing * creating service account if does not exist * removing unnecessary params * formatting * formatting * adding relavant comments * updating path * fixing comment * adding command for removing file * small fix in script to update path of file * Upgrade to go version 1.20.5 (#1196) Brief list of changes * Changes gcsfuse binary to be statically-linked, removes dependency on libc. * Prefixed/exported CGO_ENABLED=0 for every go build|run|test for every gcsfuse program * Replaced all golang:1.20.4 based base images to corresponding golang:1.20.5 based images in all docker files and all docker run commands * Updated golang 1.20.4 dependencies to golang 1.20.5 * Updated troubleshooting guide * Replaced os.user.Current() with os.Getuid() and os.Getgid() to remove dependency on glibc * Added unit tests for perms.go * Removed libc dependency from DEBIAN/CONTROL * Dynamic mounting integration tests (#1242) * testing * testing * adding dynamic mounting testing * testing * testing * separating unmounting * small fix * unblocking remaning tests * small fix(updating echo message) * fixing lint tests and adding comments * updating file path * small fix * testing * testing * testing * unblocking code for testing * testing * increasing time as taking more than 10m to run * adding random suffix at the end of bucket name as with fixed bucket it can fail when multiple instance run inetgration tests at a time * assigning it to variable * small fix * small fix * updating const var name * testing * testing * unblocking code * updating comment * minor change to test script (#1243) * Sequential write large file integration tests (#1240) * adding test for writing file of 100MB * adding test for writing file of 100MB * adding commands in script * adding commands in script * small fix * increasing chunk size * small fix * using min function * small fix * adding comment * fixing lint * writing 500MB in more redable way * fixing comments * fixing comments * changing chunksize to 20MB * testing sync * testing * merging * merging * testing sync * small fix * Kokoro perf test fix - gcloud storage command not working (#1248) * Update semantics.md -- fix typos (#1251) * Random read integration tests (#1247) * adding tests for random read files * fixing comments * creating common function * small fix * fixing comment * creating common function * small fix * merge * fixing comments and fixing typos in naming * renaming function name * renaming function name * Read multiple files at a time integration tests (#1249) * adding tests * updating comment * fixing comments and fixing typos in naming * renaming function name * renaming * fix lint test * Update gcloud version (#1259) Update jacobsa/gcloud dependency version This is to account for jacobsa/gcloud#34 . Also minor gitignore change: Ignore vscode settings folder (.vscode) * Removing rate limit dependency (#1253) * local changes * local changes * local changes * local changes * removed token bucket and added unit tests * fixing lint * small fix- renaming * small fix- renaming * fix lint * adding licence * testing * back to changes * fixing comments * removing throttle_test * adding throttle test * fixing comment * fixing lint * fixed comment * lint tests * lint tests * fixing comment * empty commit * empty commit * Upgrading go-client module from 1.29.0 to 1.31.0 (#1263) * Upgrading go-client module from 1.29.0 to 1.31.0 * Resolving merge conflict * Bump requests in /perfmetrics/scripts/load_tests/python (#1268) Bumps [requests](https://github.com/psf/requests) from 2.30.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.30.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.30.0 to 2.31.0 in /perfmetrics/scripts (#1267) Bumps [requests](https://github.com/psf/requests) from 2.30.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.30.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump certifi in /perfmetrics/scripts/load_tests/python (#1269) Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updating fuse library version (#1273) * updated architecture to all (#1270) * Bump certifi from 2023.5.7 to 2023.7.22 in /perfmetrics/scripts (#1266) Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Read gzip content-encoding objects as compressed (#1255) Enable read-compressed for gzip objects Details of changes * Add field contentEncoding in internal.storage.MinObject * Passes ReadCompressed as true for objects with contentEncoding set to "gzip" * Added corresponding unit tests. * Update fsouza/fake-gcs-server to v1.40.3 - to fix the read-behavior of objects with content-encoding gzip in fake-server, used in unit tests. * gzip integration tests (#1256) Add read-flow integration tests for gzip objects This adds read-flow integration tests for the gzip read-compressed changes added in previous PR #1255 . Tests added (2*5 = 10 tests) * Two types of operations each * List, Stat and full-file read * On five types of objects each * Object with text content, uploaded with gzip compression with content-encoding: gzip, and cache-control: no-transform * Object with text content, uploaded with gzip compression with content-encoding: gzip, and cache-control: '' * Object with gzip content, uploaded with content-encoding: '' * Object with gzip content, uploaded with gzip compression (i.e. doubly compressed) with content-encoding: gzip, and cache-control: no-transform * Object with gzip content, uploaded with gzip compression (i.e. doubly compressed) with content-encoding: gzip, and cache-control: '' Write-flow tests will be added in another commit shortly after. This commit also adds some helper functions needed for gzip integration tests Functions added * CreateLocalTempFile - creates a temporary local file of given size, gzip/non-gzip * DiffFiles * GetGcsObjectSize * DownloadGcsObject * UploadGcsObject * DeleteGcsObject * ClearCacheControlOnGcsObject * Add gzip write-flow tests (#1262) It adds write-flow integration tests for GCSfuse-mounted objects with content-encoding gzip. These tests replace an existing gzip object in a mounted bucket, then verify the size of the re-uploaded GCS object. Tests added (1*5 = 5 tests) * One type of operation * Overwrite * On five types of objects each * Object with text content, uploaded with gzip compression with content-encoding: gzip, and cache-control: no-transform * Object with text content, uploaded with gzip compression with content-encoding: gzip, and cache-control: '' * Object with gzip content, uploaded with content-encoding: '' * Object with gzip content, uploaded with gzip compression (i.e. doubly compressed) with content-encoding: gzip, and cache-control: no-transform * Object with gzip content, uploaded with gzip compression (i.e. doubly compressed) with content-encoding: gzip, and cache-control: '' This also adds a necessary CopyFileAllowOverwrite alternative utility to the existing CopyFile utility(which doesn't allow file overwrite) in tools/integration_tests/util/operations/file_operations.go . * Fix kokoro job failure for gzip integration tests (#1276) Kokoro VM is based on a specialized linux image created by the Kokoro team for running jobs on it. This has an old version of gcloud on it, and thus the gcsfuse integration test jobs use `gcloud alpha` on it as opposed to `gcloud` command. To make matters worse, on the kokoro VM, `gcloud alpha` doesn't support the `-Z` option as well as the options for updating object metadata. Both these features are required for integration tests for GCS objects with content-encoding gzip in GCSFuse. Hence, for both these features, switching from `gcloud alpha` to equivalent `gsutil` commands. * fixing requirements (#1281) * Replace gcloud with gsutil in file-op utils (#1280) Replaces `gcloud` or `gcloud alpha` with `gsutil` in all gcs related utilities in file_operations utils in integration_tests. This is to avoid failures related to compatibility in kokoro VMs which support an old version of gcloud cli. * Deprecate --enable-storage-client-library flag (#1258) * local changes * local changes * local changes * local changes * removed token bucket and added unit tests * fixing lint * small fix- renaming * small fix- renaming * fix lint * adding licence * testing * back to changes * fixing comments * removing throttle_test * depricating from enable storage client library flag * temporary using env variable to set end point * fixing linux tests * adding throttle test * fixing comment * fixing lint * fixed comment * lint tests * lint tests * removing endpoint changes * removing endpoint changes * removing unnecessary changes * removing connection * fixing comment * empty commit * empty commit * removing endponti changes * fixing comments * formating * removing debug http * fixing linux tests * adding back debug_http flag * fixing comments * renaming function * fixing comments * fixing comments * fixing comments in integration test scrip * fixing comments in integration test scrip * fixing comments in integration test scrip * fixing comments in integration test script * updating vendor client * fixing comment * removing concurrent read * empty commit * replacing mountWithConn to mountWithStorageHandle * empty commit * testing * testing * back to original changes * Enabling --endpoint flag for go-storage-client library (#1264) * Enabling --endpoint flag for go-storage-client library * Fixing lint issue * Added more unit test * fixing end pointd escription * Refactoring and resolving comments * refactoring * Reverting previous refactoring, created client_helper for all client creation logic * treating passing actual GCSURL like custom point, by design in go-storage-lib * Fixing unit test, removing unused token-source * Minor refactoring, adding dummy key-file * Making the default client non-nil so that we can get dummy token-source * Fixing unit test * Review comments updated * Review comments * Incorporating review comments * Fixing build issue * Incorporating review comments * Renamed client helper to client * Incorporating review comments * Adding more unit test * Review comments * Fixing flag unit test * Fixed gcsfuse hang * Triggering performance test and fixing unit test * Changing the timeout from 15m to 24m for integraotin test of one package (#1284) * Changing the timeout from 15m to 24m for integraotin test of one package * Adding the timout flag for continuous test also * Added an experimental flag to trigger JSON flow of go-storage-client (#1265) * Enabling --endpoint flag for go-storage-client library * Fixing lint issue * Added more unit test * fixing end pointd escription * Refactoring and resolving comments * refactoring * Reverting previous refactoring, created client_helper for all client creation logic * treating passing actual GCSURL like custom point, by design in go-storage-lib * Fixing unit test, removing unused token-source * Minor refactoring, adding dummy key-file * Making the default client non-nil so that we can get dummy token-source * Fixing unit test * Review comments updated * Review comments * Incorporating review comments * Fixing build issue * Incorporating review comments * Renamed client helper to client * Incorporating review comments * Adding more unit test * Review comments * Support json read in GCSFuse * Rebased with support endpoint branch * Incorporating review comments * reverting formatting issue * Fixing build issue * Fixing build failure * Correcting a comment * Fixing typo * Enabling --endpoint flag for go-storage-client library * Fixing lint issue * Added more unit test * fixing end pointd escription * Refactoring and resolving comments * refactoring * Reverting previous refactoring, created client_helper for all client creation logic * treating passing actual GCSURL like custom point, by design in go-storage-lib * Fixing unit test, removing unused token-source * Minor refactoring, adding dummy key-file * Making the default client non-nil so that we can get dummy token-source * Fixing unit test * Review comments updated * Review comments * Incorporating review comments * Fixing build issue * Incorporating review comments * Renamed client helper to client * Incorporating review comments * Adding more unit test * Review comments * Fixing flag unit test * Fixed gcsfuse hang * Triggering performance test and fixing unit test * Fixing linting issue * Trigger perf test * Added equivalent command in mounted-directory run test * Fixing review comments: replacing printf error log to returning error (#1285) * Updating gcsufse --help (#1286) * Adding changes to run integration and perf tests with label (#1283) * adding changes to separate integration tests and perf tests * fix braces error * fix random string error * testing * remove changes * merge * merge * fixing comments * fixing comments * fixing comments * fixing comments * testing * testing * echo bucket name * testing by adding git stash * removing testing commanda * testing by adding git stash * testing by adding git stash * testing by adding git stash * testing by adding git stash * testing kokoro * testing kokoro * testing kokoro * testing kokoro * undo testing changes * fixing comment * testing with both the labels * removing testing changes * removing comment * Removing extra logs in kokoro continous test (#1289) * removing extra logs of go tar * removing extra logs of go tar * timeout increase for linting issue * Revert "timeout increase for linting issue" This reverts commit 8d5e9af. * Adding yaml configurations * Trigger CI integration test * Trigger CI integration test1 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Tulsishah <46474643+Tulsishah@users.noreply.github.com> Co-authored-by: Ashmeen Kaur <57195160+ashmeenkaur@users.noreply.github.com> Co-authored-by: Nitin Garg <gargnitin@google.com> Co-authored-by: Nitin Garg <113666283+gargnitingoogle@users.noreply.github.com> Co-authored-by: Minwoo Byeon (Dylan) <mwbyun@hotmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
raj-prince
added a commit
that referenced
this pull request
Feb 29, 2024
* Updating issue template for adding issue SLO (#1216) * adding bug SLO * adding bug SLO * Fixing comment * Integration tests --key-file flag and GOOGLE_APPLICATION_CREDENTIALS env with admin permission tests (#1167) * updating go version * empty commit * local commit * local changes * local changes * local changes * adding key file tests * testing * testing * testing * testing * local changes * local changes * local changes * local changes * testing * testing * testing * testing * testing * adding test for admin creds * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * testing * formating * testing defer statement * testing defer statement for deleting credentials * adding comment * testing with error * testing with error * testing with error * removing testing statement * adding testbucket and mntdir in commnd * adding comment * updating bucket name * updating bucket name * removing unnecessary changes * removing unnecessary changes * removing unnecessary changes * formatting * conflict * adding error handling * testing * small fix * removing creds tests from implicit and explicit dir tests * testing * testing * testing * testing * removing testing statement * adding creds tests in operations back * Testing * Testing * Testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * create service account key testing * adding remaining changes * adding remaining changes * adding remaining changes * testing service account * testing service account * testing service account * adding comments * removing unnecessary changes * formatting * testing * testing * testing * testing * removing without key file tests * small fix * formalizing for reuse * small fix * removing unnecessary changes * formatting * updating comment * updating comment * updating comment * fixing comments * adding comment * testing * testing * adding condintion for service account already exsit * adding condintion for service account already exsit * testing time * running tests only for operations * Changing the number of epoch based on previous observation. (#1222) * Adding rpm digest while creating rpm package (#1215) * removing defer as not working properly in for loop (#1223) --------- Co-authored-by: Tulsishah <46474643+Tulsishah@users.noreply.github.com> Co-authored-by: Prince Kumar <princer@google.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Changing the number of epochs:
Pytorch - from 100 to 80 - since initial number took more than 7 days. Expectation is to run the test for 7 days, hence reducing the number of epochs.
TF - 675 - from 1000 to 675 - since initial number took more than 7 days. Expectation is to run the test for 7 days, hence reducing the number of epochs.
Link to the issue in case of a bug fix.
NA
Testing details