From c75f4f040b7f66d48d1be261b18f38c628164397 Mon Sep 17 00:00:00 2001 From: Piotr Adamczyk Date: Tue, 23 Feb 2021 18:36:37 +0100 Subject: [PATCH 1/9] docs: Explore test-targets-for-shards SDD --- ...t-target-for-shards-with-extra-features.md | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 docs/feature/1532-explore-test-target-for-shards-with-extra-features.md diff --git a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md new file mode 100644 index 0000000000..c148d8a0b4 --- /dev/null +++ b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md @@ -0,0 +1,56 @@ +# Auto discover rest of tests when using test-target-for-shards option + +Currently, when using `test-target-for-shards` you need to specify all tests manually to split to shards. If you do not +do that then only listed tests are executed. +It will be good to provide an option to auto-discover other than specified tests to make `test-target-for-shards` +better and easier to use. + +# References + +- [Conversation on Slack](https://firebase-community.slack.com/archives/C72V6UW8M/p1611322014000600) +- [1532](https://github.com/Flank/flank/issues/1532) + +# Motivation + +`test-target-for-shards` is not so easy to use, this change will improve it when using Flank and give users more +flexibility and control over creating shards. + +# Goals + +1. New option is available on the flank configuration level +1. User could specify which packages/class/tests run on specific shards by using a new option +1. Other not specified tests are automatically added to the left shard/shards + +# Design + +- Add new option under Flank configuration, possible ideas are: `plan-sharding`, `tests-for-shards`. +- New option should behave the same as [`test-target-for-shards`](https://cloud.google.com/sdk/gcloud/reference/beta/firebase/test/android/run#--test-targets-for-shard). +- After that Flank should split the test into shards specified by the user and make another shard/shards with the left one. +- Flank should fast fail if `disable-sharding` is set to `true` or `max-test-shards` is lower than specified test shards + by user + 1. + +# API + +It is hard to plan API for that change because the codebase could be different when doing this task and also usage of this +the option will be applied to many places in code. +Please follow up `Design` section for the implementation plan. + +# Results + +A new option which split sharding based on user input or alternative flag (described in `Alternative Considered`) which +will change behavior of `test-target-for-shards`. + +# Dependencies + +There are not any dependencies that could have a real impact on this task. However, it will be best to do this after +[Flank's refactor](https://github.com/Flank/flank/issues/1317) to make it easier. + +# Testing + +1. Add some tests(not all) as a value to a new option +1. Verify that they are correctly split into shards and there is one more shard with rest of tests(not specified) + +# Alternatives Considered + +Add a new flag to Flank configuration, which overrides `test-target-for-shards` and automatically add the rest of the tests to +separate shard/shards(if max tests shards are greater than left count). From 16d103b16b288863ea661f5c87649cf88ebaceb4 Mon Sep 17 00:00:00 2001 From: piotradamczyk5 <65554637+piotradamczyk5@users.noreply.github.com> Date: Wed, 24 Feb 2021 10:18:46 +0100 Subject: [PATCH 2/9] Update docs/feature/1532-explore-test-target-for-shards-with-extra-features.md Co-authored-by: Michael Wright --- .../1532-explore-test-target-for-shards-with-extra-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md index c148d8a0b4..2946ff136e 100644 --- a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md +++ b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md @@ -24,7 +24,7 @@ flexibility and control over creating shards. # Design - Add new option under Flank configuration, possible ideas are: `plan-sharding`, `tests-for-shards`. -- New option should behave the same as [`test-target-for-shards`](https://cloud.google.com/sdk/gcloud/reference/beta/firebase/test/android/run#--test-targets-for-shard). +- New option should at least behave the same as [`test-target-for-shards`](https://cloud.google.com/sdk/gcloud/reference/beta/firebase/test/android/run#--test-targets-for-shard). - After that Flank should split the test into shards specified by the user and make another shard/shards with the left one. - Flank should fast fail if `disable-sharding` is set to `true` or `max-test-shards` is lower than specified test shards by user + 1. From 5da37e71f809733e4791996ae21747a2eb18a6fe Mon Sep 17 00:00:00 2001 From: piotradamczyk5 <65554637+piotradamczyk5@users.noreply.github.com> Date: Wed, 24 Feb 2021 10:18:51 +0100 Subject: [PATCH 3/9] Update docs/feature/1532-explore-test-target-for-shards-with-extra-features.md Co-authored-by: Michael Wright --- .../1532-explore-test-target-for-shards-with-extra-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md index 2946ff136e..ccbf8a9adf 100644 --- a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md +++ b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md @@ -31,7 +31,7 @@ flexibility and control over creating shards. # API -It is hard to plan API for that change because the codebase could be different when doing this task and also usage of this +It is hard to plan for this change to the API because by the time this is worked upon the codebase may have changed significantly influencing the usage of this request. the option will be applied to many places in code. Please follow up `Design` section for the implementation plan. From cbd7699b4d1123049d918e09807f73d12216678d Mon Sep 17 00:00:00 2001 From: piotradamczyk5 <65554637+piotradamczyk5@users.noreply.github.com> Date: Wed, 24 Feb 2021 10:18:56 +0100 Subject: [PATCH 4/9] Update docs/feature/1532-explore-test-target-for-shards-with-extra-features.md Co-authored-by: Michael Wright --- .../1532-explore-test-target-for-shards-with-extra-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md index ccbf8a9adf..7d9d20af3a 100644 --- a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md +++ b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md @@ -25,7 +25,7 @@ flexibility and control over creating shards. - Add new option under Flank configuration, possible ideas are: `plan-sharding`, `tests-for-shards`. - New option should at least behave the same as [`test-target-for-shards`](https://cloud.google.com/sdk/gcloud/reference/beta/firebase/test/android/run#--test-targets-for-shard). -- After that Flank should split the test into shards specified by the user and make another shard/shards with the left one. +- The left over tests should then be sorted into other shards or a single shard based upon the configuration provided by the user. - Flank should fast fail if `disable-sharding` is set to `true` or `max-test-shards` is lower than specified test shards by user + 1. From fa6ab9d98299015e97a629f1a53161777789ff4f Mon Sep 17 00:00:00 2001 From: piotradamczyk5 <65554637+piotradamczyk5@users.noreply.github.com> Date: Wed, 24 Feb 2021 10:19:01 +0100 Subject: [PATCH 5/9] Update docs/feature/1532-explore-test-target-for-shards-with-extra-features.md Co-authored-by: Michael Wright --- .../1532-explore-test-target-for-shards-with-extra-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md index 7d9d20af3a..04decbd741 100644 --- a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md +++ b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md @@ -37,7 +37,7 @@ Please follow up `Design` section for the implementation plan. # Results -A new option which split sharding based on user input or alternative flag (described in `Alternative Considered`) which +A new option will be created which will split sharding based on user input or an alternative flag (described in `Alternative Considered`) which will change behavior of `test-target-for-shards`. # Dependencies From 057caac822729adea0097f73d5e5dfcd4426dc71 Mon Sep 17 00:00:00 2001 From: piotradamczyk5 <65554637+piotradamczyk5@users.noreply.github.com> Date: Wed, 24 Feb 2021 10:19:06 +0100 Subject: [PATCH 6/9] Update docs/feature/1532-explore-test-target-for-shards-with-extra-features.md Co-authored-by: Michael Wright --- .../1532-explore-test-target-for-shards-with-extra-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md index 04decbd741..41ce9992d5 100644 --- a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md +++ b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md @@ -32,7 +32,7 @@ flexibility and control over creating shards. # API It is hard to plan for this change to the API because by the time this is worked upon the codebase may have changed significantly influencing the usage of this request. -the option will be applied to many places in code. +This option will be applied to many places in code. Please follow up `Design` section for the implementation plan. # Results From 0b1338fc068a5c93d6619a4fb74fbe35fc70e225 Mon Sep 17 00:00:00 2001 From: piotradamczyk5 <65554637+piotradamczyk5@users.noreply.github.com> Date: Wed, 24 Feb 2021 10:19:11 +0100 Subject: [PATCH 7/9] Update docs/feature/1532-explore-test-target-for-shards-with-extra-features.md Co-authored-by: Michael Wright --- .../1532-explore-test-target-for-shards-with-extra-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md index 41ce9992d5..d4064dd930 100644 --- a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md +++ b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md @@ -42,7 +42,7 @@ will change behavior of `test-target-for-shards`. # Dependencies -There are not any dependencies that could have a real impact on this task. However, it will be best to do this after +There are no dependencies that will have a impact on this task. It is recommended to do this after [Flank's refactor](https://github.com/Flank/flank/issues/1317) to make it easier. # Testing From ceb76bdd628bcffa98e400f9a04a6a816c90cfaf Mon Sep 17 00:00:00 2001 From: piotradamczyk5 <65554637+piotradamczyk5@users.noreply.github.com> Date: Wed, 24 Feb 2021 10:19:18 +0100 Subject: [PATCH 8/9] Update docs/feature/1532-explore-test-target-for-shards-with-extra-features.md Co-authored-by: Michael Wright --- .../1532-explore-test-target-for-shards-with-extra-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md index d4064dd930..f353618512 100644 --- a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md +++ b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md @@ -23,7 +23,7 @@ flexibility and control over creating shards. # Design -- Add new option under Flank configuration, possible ideas are: `plan-sharding`, `tests-for-shards`. +- Add new option under Flank configuration, possible ideas are: `plan-sharding`, `tests-for-shards`, `smart-test-targets-for-shards`. - New option should at least behave the same as [`test-target-for-shards`](https://cloud.google.com/sdk/gcloud/reference/beta/firebase/test/android/run#--test-targets-for-shard). - The left over tests should then be sorted into other shards or a single shard based upon the configuration provided by the user. - Flank should fast fail if `disable-sharding` is set to `true` or `max-test-shards` is lower than specified test shards From b12e6c29dfb10d72d894efd979adc50c5f5123ef Mon Sep 17 00:00:00 2001 From: piotradamczyk5 <65554637+piotradamczyk5@users.noreply.github.com> Date: Wed, 24 Feb 2021 10:19:23 +0100 Subject: [PATCH 9/9] Update docs/feature/1532-explore-test-target-for-shards-with-extra-features.md Co-authored-by: Michael Wright --- .../1532-explore-test-target-for-shards-with-extra-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md index f353618512..06202bde6e 100644 --- a/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md +++ b/docs/feature/1532-explore-test-target-for-shards-with-extra-features.md @@ -19,7 +19,7 @@ flexibility and control over creating shards. 1. New option is available on the flank configuration level 1. User could specify which packages/class/tests run on specific shards by using a new option -1. Other not specified tests are automatically added to the left shard/shards +1. Other tests that are not specified are then automatically added to a different set of shards/shard # Design