🐛 [firebase_core] Is not compatible with headless integration testing.

[polina-c]

Firebase fails to setup connection for integration test for headless device (It succeeds for -d macos though).

Steps to repro:

1. Clone PR: Setup integration testing. date-game-com/date-game#2
2. cd pkgs/date_game
3. Run flutter test integration_test/app_test.dart -d flutter-tester
4. Get error.

Output for print('!!!! defaultTargetPlatform: $defaultTargetPlatform') is !!!! defaultTargetPlatform: TargetPlatform.android.

Error:

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following PlatformException was thrown running a test:
PlatformException(channel-error, Unable to establish connection on channel., null, null)

When the exception was thrown, this was the stack:
#0      FirebaseCoreHostApi.initializeCore (package:firebase_core_platform_interface/src/pigeon/messages.pigeon.dart:210:7)
<asynchronous suspension>
#1      MethodChannelFirebase._initializeCore (package:firebase_core_platform_interface/src/method_channel/method_channel_firebase.dart:29:44)
<asynchronous suspension>
#2      MethodChannelFirebase.initializeApp (package:firebase_core_platform_interface/src/method_channel/method_channel_firebase.dart:73:7)
<asynchronous suspension>
#3      Firebase.initializeApp (package:firebase_core/src/firebase.dart:43:31)
<asynchronous suspension>
#4      main (package:date_game/main.dart:10:3)
<asynchronous suspension>
#5      main.<anonymous closure> (file:///Users/polinach/_/date-game/pkgs/date_game/integration_test/app_test.dart:22:5)
<asynchronous suspension>
#6      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:165:15)
<asynchronous suspension>
#7      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:982:5)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

The test description was:
  Smoke
════════════════════════════════════════════════════════════════════════════════════════════════════
00:06 +0 -1: Smoke [E]                                                                                                                
  Test failed. See exception logs above.
  The test description was: Smoke

[darshankawar]

Thanks for the report @polina-c
I do see same behavior as reported. The repo uses gradle-7.5-all.zip so #10825 (comment) isn't related.

[polina-c]

Thank you for checking. The same issue applies to unit testing.
To repro issue with unit tests:

1. Clone Crud date-game-com/date-game#4
2. cd pkgs/date_game
3. flutter test test/logic/shared/primitives/crud_test.dart

[polina-c]

Is there workaround?

I really want to test cover user flows, so that GitHub bots could verify them on presubmit.
May be flutter-tester is wrong device for integration tests that use firestore?

What is recommended device for testing features, that use firestore, on GitHub bots?

[darshankawar]

Tagging @Lyokone @russellwheatley for their input on this.

[russellwheatley]

Hi @polina-c, just to clarify, presumably you still mean running integration tests on an emulator, simulator or Chrome driver without the GUI also running?

[polina-c]

I am interested to run tests by github bots on presubmit.
I want to manipulate UI, i.e. to use pumpWidget, finders, gestures and so on.

But I am open in terms of what test engine and what device to use. I created some implementation (see PR), but it is just one option.

Does it answer the question?

[russellwheatley]

Ok, I think I understand what you mean. We don't have any integration tests that currently handle UI input/feedback, etc. Having said that, we are interested in this draft PR submitted by a member of the Patrol team: #11378

It uses Patrol to handle API that requires UI input/feedback. Read more about Patrol here: https://pub.dev/packages/patrol

Hope that helps!

[polina-c]

Let me make sure there is no misunderstanding.
Flutter Framework has concept of integration tests where the flutter application is run by test engine, button clicks are emulated and results are verified. Applications in such tests have access to internet and are fully functional.

However, calls to firebase backend fail, because platform is detected as android, while it is not android.

It would be very beneficial to enable Flutter integration tests to to verify user flows that work with authentication and database.

Does it change something?

[russellwheatley]

If it is running headless, it won't run on a GUI (i.e. emulator/simulator). FlutterFire is ultimately a wrapper around the native SDKs which require a non-headless environment to communicate with the native platforms.

[polina-c]

thank you!
Which platform is recommended for testing interaction with Firebase on GitHub bots, instead of headless, then?

[russellwheatley]

I don't know enough about GitHub bot capabilities to give you an answer on that I'm afraid. I can point you in the direction of where our e2e tests are ran via GitHub actions, this is android & this is iOS

a summary of what is happening in each action:

1. Setup nodejs and java to run Firebase emulators.
2. We cache certain elements to speed up CI run (e.g. emulator, Pod files, Firebase emulator)
3. Run melos boostrap to install dependencies and link packages.
4. Run Firebase emulator to interact with e2e tests.
5. Boot up emulator for android or simulator for iOS.
6. Run flutter test integration_test/e2e_test.dart which runs our e2e test suite here.

[polina-c]

Thanks.
I think flutterfire documentation should contain instruction how to do presubmit integration test coverage, as it is necessary to build reliable flutter/firebase applications.

[Lyokone]

Hello @polina-c, after discussing it internally, we will not add instructions on how to integrate CI/CD with FlutterFire because we feel it would duplicate informations that can be found elsewhere and that are highly dependent on other CI platforms like GitHub Actions, Codemagic or else ... Please found the current official instructions here: https://firebase.google.com/docs/test-lab/flutter/integration-testing-with-flutter