Error Testing in Flutter plugin

[hanwencheng]

Is it possible to write foreign function test in dart for a flutter plugin?

I use ffi to import rust functions, and they did work in the example application. But failed in the test.

it comes with error:

Invalid argument(s): Failed to lookup symbol (dlsym(RTLD_DEFAULT, my_function_name): symbol not found)
  dart:ffi                                                          DynamicLibrary.lookup

More Context:

The way I used for calling the library is

final DynamicLibrary nativeSubstrateSignLib = Platform.isAndroid
    ? DynamicLibrary.open("myPackage.so")
    : DynamicLibrary.process();

For more information, the very simple code base is here

[mkustermann]

/cc @dcharkes

[dcharkes]

This Medium post mentions a workaround for the symbols not being visible:

Note: XCode will not bundle the library unless it detects explicit usage within the workspace. Since our Dart code calling it is out of the scope of XCode, we need to write a dummy Swift function that makes some fake usage.

Does this workaround work for you?

(cc similar issues: flutter/flutter#33227 (comment) and flutter/flutter#33227 (comment).)

[hanwencheng]

I acutally applied the walkaround in the medium post and only testing for the function which exposed in the dummy Swift function, the library could still not be found.

The weird thing is that calling the library in Simualtor is all good, just test is failing.

Thanks for mention the #33227 thread, I tried to remove s.static_framework = true this in podspec, but it does not work, this setting seems useful for the building but not for the test problem.

[dcharkes]

@hanwencheng can you describe in which combinations the symbols can and cannot be found? Which combination of options work and doesnt work?

• debug/release/profile mode
• Android/iOS
• On devices/simulator
• Testing/running app

[hanwencheng]

Ok, here is the complete testing result:

It works on

• debug mode on Andorid on Device
• debug mode on Android on Emulator
• debug mode on iOS on Simulator
• Integration test on iOS Simulator (call native functions)

It failed on

• debug mode on iOS on device (crash with unknown error) (probably not related to this issue ticket)
• Unit Test (run flutter test on plugin's root folder) failed with error

Invalid argument(s): Failed to lookup symbol (dlsym(RTLD_DEFAULT, my_function_name): symbol not found)
  dart:ffi                                                          DynamicLibrary.lookup

[dcharkes]

I see. So far, we've been testing by actually running an app on device or emulator/simulator and performing a FFI operations. I will take a look at how unit testing works in Flutter.

Do you have a minimal repro that I could easily reproduce?

[hanwencheng]

This one could be useful: https://github.com/hanwencheng/substrate_sign_flutter/blob/master/test/substrate_sign_flutter_test.dart

[dcharkes]

I can reproduce this locally.

$ flutter test --verbose
[...]
[   +1 ms] /Users/dacoharkes/flt/engine/flutter/bin/cache/artifacts/engine/darwin-x64/flutter_tester --disable-observatory --enable-checked-mode
--verify-entry-points --enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling --non-interactive --use-test-fonts
--packages=/Users/dacoharkes/ffi-samples/hanwencheng/substrate_sign_flutter/example/.packages
/var/folders/6l/j64crzh16j7djnq676q0994h00klzg/T/flutter_test_listener.l3dCla/listener.dart.dill

We need to make flutter_tester aware that it needs to ship the native library, if that is possible. I'll dig a bit deeper. (Though I see you've already found a workaround with flutter drive --target=test_driver/app.dart.)

[hanwencheng]

We need to make flutter_tester aware that it needs to ship the native library.

AFAIK, in React Native it is not possible to directly use a custom native library for unit tests, It would be exciting if it is possible with Dart / Flutter.

[dcharkes]

After some more digging:

• flutter test only loads Dart code, and does not build any plugin code.
• Libraries used by dart:ffi are bundled either through Gradle (android) or CocoaPods (iOS).

That's why they do not work together right now.

We could possibly make unit tests work for dynamically linked libraries (using DynamicLibrary.open(...). However, that would not work for static linking (using DynamicLibrary.process()). We would also need to specify the paths the dynamic libraries in that case, either by passing them as arguments to flutter test or by putting them in a config file (possibly pubspec.yaml).

That last option, is also what we would need for solving dart-lang/sdk#36712. And then we could generate the right CocoaPods and Maven configs from that. But that is something further out on the horizon.

I suggest you use the workaround with flutter drive for the time being. Does that work for you @hanwencheng?

[hanwencheng]

Now it is totally fine for me to use flutter drive, thanks for the clarifying.

And I try to understand:

We would also need to specify the paths the dynamic libraries in that case, either by passing them as arguments to flutter test or by putting them in a config file (possibly pubspec.yaml).

This is the requirement for enabling statically linked libraries in the unit test, it that correct? if it is so, will "supporting dynamically linked libraries in the unit test" be included in the roadmap?

[dcharkes]

It also is a requirement for dynamically linked libraries, if the path from the executable to the library is different in the unit test and the deployed app. In the unit test the executable is the flutter_tester, while in the deployed app it is bundled with the libraries in the apk on Android for example.

It might be possible that dynamically linked libraries already work in unit tests if you DynamicLibrary.open(...) them with an absolute path on your host computer because the unit tester runs on your host. However, that path will of course not work in Android apks, so you'd have to somehow branch on whether you're running unit tests.

[derolf]

I have a "better" workaround for this (works on Mac):

1. Before running the unittest, build a dynamic lib using make:

cmake foo/bar/CMakeLists.txt -B build/test
cd build/test
make

2. When you load the lib, handle unit testing:

DynamicLibrary _open() {
  if (Platform.environment.containsKey('FLUTTER_TEST')) {
    return DynamicLibrary.open('build/test/libfoo.dylib');
  }
  ...
}

[dcharkes]

Closing issue as both flutter drive works and flutter test works with the last workaround.

[ahimta]

@derolf thanks for the tip. Unfortunately, passing an absolute path for a Rust shared-library built for Android x86-64 (tried built for Linux x86-64 too) on Ubuntu 20.04 LTS doesn't seem to work. I can confirm that the FFI integration works correctly in the Android app in an Android x86-64 emulator so the tests are the culprit here and flutter drive would probably work just fine but it probably wouldn't work as easily in many CI environments.

@dcharkes the biggest issue here is that DynamicLibrary.open() and flutter test have a seemingly adversarial behavior:

1. You can't know whether the file passed to DynamicLibrary.open() is compatible with the running environment or even exists (I tried random paths and the call still succeeds). And you only discover when the DynamicLibrary#lookup() call fails.
2. You can't know whether DynamicLibrary.open() loaded the library successfully. This is more acceptable as libraries can be loaded lazily but not a valid excuse for not doing simple and fast checks.
3. flutter test prints the error-message related to DynamicLibrary.open() and doesn't exit and goes ahead and runs the rest of the tests and you have to scroll to the top of the flutter test output (if you're lucky or waste much time otherwise) to notice that this is why the tests fail. Some test frameworks catch some errors/exceptions but, in my case, the native code is loaded statically (as a final global variable) before the tests even start and, even if they're loaded dynamically, this should probably crash the tests if it's an uncaught exception by the application.

It may seem like a joke but DynamicLibrary.open() and flutter test behavior is suitable for CTF competitions and code obfuscation.

But I might be wrong here and there maybe very good reasons for this. In this case, error messages should probably explain this and maybe provide links for further details. This isn't a luxury and can save time for project maintainers as these details are often forgotten and can waste a lot of a maintainer's time when they work on them or diagnose a related issue.

This should be a guiding principle: fail as early as possible and have the code do the checks that can be automated and only require a person to diagnose for cases that can't be handled easily or are likely false positives. And provide rationals (e.g: error messages, docs, code comments) only as a last resort as they often indicate an underlying issue.

[Sakari369]

Spent couple of days here wondering why my prebuilt static library did not work on iOS, when on macOS desktop target it worked perfectly. Seems the issue was with dead code stripping, XCode aggressively removing my whole C++ library if there was no calling to in the actual iOS runner code.

Added a Bridging header to my iOS Runner project, where I defined one function signature that exists in my C++ library and then called that in a dummy function inside the iOS AppDelegate swift code.

This solved the problem!

I would have not found this without finding this issue, would be really beneficial if this dead code stripping was mentioned somewhere in the iOS native code section.

[ahimta]

@Sakari369 I remember the dead-code stripping with iOS was mentioned in the docs when I wrote my previous comment. But not sure whether it still exists in the docs right now.

But this is only one symptom of the underlying disease of failing many steps too late and making diagnosis exponentially more difficult.

[Sakari369]

Okay, yeah to be honest maybe not something Dart should worry about, I was thinking I was on a Flutter issue when I wrote the comment..

But looking at for example https://dart.dev/guides/libraries/objective-c-interop, I can't see any mentions of such things.
Well, googling works, and it's a complex thing to really document well, so many moving pieces and even the whole Intel / arm64 thing on macOS is big enough thing to cause confusion.