Rate Limit Exceeded

[miguelslemos]

Describe the bug
After bump from 20.05.2 to 20.06.2 I started to see some issues related to rate limit. Also, there are no significant changes on the shard size or test amount.

java.io.IOException: Request failed
  at ftl.http.ExecuteWithRetryKt$executeWithRetry$$inlined$withRetry$1.invokeSuspend(ExecuteWithRetry.kt:36)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
  at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:241)
  at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:270)
  at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:79)
  at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54)
  at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
  at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:36)
  at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
  at ftl.http.ExecuteWithRetryKt.executeWithRetry(ExecuteWithRetry.kt:41)
  at ftl.gc.GcToolResults.getStepResult(GcToolResults.kt:98)
  at ftl.json.SavedMatrix.finished(SavedMatrix.kt:90)
  at ftl.json.SavedMatrix.update(SavedMatrix.kt:65)
  at ftl.json.MatrixMapKt.update(MatrixMap.kt:47)
  at ftl.run.NewTestRunKt$newTestRun$2$2.invokeSuspend(NewTestRun.kt:24)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
  at kotlinx.coroutines.ResumeModeKt.resumeUninterceptedMode(ResumeMode.kt:45)
  at kotlinx.coroutines.internal.ScopeCoroutine.afterCompletionInternal(Scopes.kt:32)
  at kotlinx.coroutines.JobSupport.completeStateFinalization(JobSupport.kt:310)
  at kotlinx.coroutines.JobSupport.tryFinalizeSimpleState(JobSupport.kt:276)
  at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:807)
  at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:787)
  at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:111)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
  at kotlinx.coroutines.ResumeModeKt.resumeUninterceptedMode(ResumeMode.kt:45)
  at kotlinx.coroutines.internal.ScopeCoroutine.afterCompletionInternal(Scopes.kt:32)
  at kotlinx.coroutines.JobSupport.completeStateFinalization(JobSupport.kt:310)
  at kotlinx.coroutines.JobSupport.tryFinalizeFinishingState(JobSupport.kt:236)
  at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:849)
  at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:811)
  at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:787)
  at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:111)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
  at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:241)
  at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:270)
  at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:79)
  at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54)
  at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
  at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:36)
  at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
  at ftl.cli.firebase.test.android.AndroidRunCommand.run(AndroidRunCommand.kt:48)
  at picocli.CommandLine.executeUserObject(CommandLine.java:1769)
  at picocli.CommandLine.access$900(CommandLine.java:145)
  at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2150)
  at picocli.CommandLine$RunLast.handle(CommandLine.java:2144)
  at picocli.CommandLine$RunLast.handle(CommandLine.java:2108)
  at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975)
  at picocli.CommandLine.execute(CommandLine.java:1904)
  at ftl.Main$Companion$main$1.invoke(Main.kt:53)
  at ftl.Main$Companion$main$1.invoke(Main.kt:43)
  at ftl.util.Utils.withGlobalExceptionHandling(Utils.kt:130)
  at ftl.Main$Companion.main(Main.kt:49)
  at ftl.Main.main(Main.kt)
C used by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 429 Too Many Requests
{ 
  "code" : 429,
  "errors" : [ {
    "domain" : "global",
    "message" : "Quota exceeded for quota group 'default' and limit 'Queries per user per 100 seconds' of service 'toolresults.googleapis.com' for consumer 'project_number:x'.",
    "reason" : "rateLimitExceeded"
  } ],
  "message" : "Quota exceeded for quota group 'default' and limit 'Queries per user per 100 seconds' of service 'toolresults.googleapis.com' for consumer 'project_number:x'.",
  "status" : "RESOURCE_EXHAUSTED"
} 
  at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
  at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
  at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
  at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:443)
  at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1092)
  at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
  at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
  at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
  at ftl.http.ExecuteWithRetryKt$executeWithRetry$$inlined$withRetry$1.invokeSuspend(ExecuteWithRetry.kt:41)
  ... 52 more   

To Reproduce

Steps to reproduce the behavior:

1. Flank.jar firebase test android run --num-flaky-test-attempts=0 --full-junit-result=true
2. See error

Expected behavior

Should not make too many requests or throttle request if rate limit happen

Additional context
We are running another 7 flank execution in parallel. Total ~6k tests.

    flank:
      max-test-shards: 15
      shard-time: 160
      num-test-runs: 1
      smart-flank-gcs-path: gs://x/unit-tests.xml
      smart-flank-disable-upload: false
      files-to-download:
      test-targets-always-run:
      disable-sharding: false
      project: x
      local-result-dir: results
      full-junit-result: true
      # Android Flank Yml
      keep-file-path: false
      additional-app-test-apks:
      run-timeout: -1
      legacy-junit-result: false
      ignore-failed-tests: false
      output-style: multi

RunTests

 Smart Flank cache hit: 100% (88 / 88)
  Shard times: 93s, 93s

  Uploadingx-app-debug.apk .
  Uploading x-androidTest.apk .
  88 tests / 2 shards

[bootstraponline]

Thanks for reporting! I recommend opening a support ticket with Firebase for now to increase your limit. I've escalated this to the FTL team.

[bootstraponline]

For the Flank team, let's create a document that outlines the API usage for different scenarios. We should know exactly how many API requests are used when fetching tool results. Possibly we could instrument the Flank client itself to print this data.

[pawelpasterz]

Flank should respect limits (as of a day 5th August 2020) :

API	Unit	Default Limit	Max Limit
Cloud Testing API	requests/100seconds/project	100_000	200_000
Cloud Tool Results API	requests/100seconds/user	2_000	4_000

Above are default and max-allowed values*

*according to Firebase team there are some extra cases when the team is able to increase limits above max-allowed but those are uncommon