Add GenAI inference task (#26)

* Adds mediapipe_core package (#11)

* adds mediapipe_core package

* adds makefile for all packages

* fixes typo in Makefile

* Apply suggestions from code review

* Update Makefile

* Apply suggestions from code review

* updated generated code's location and license (for 3P status)

* code review responses

including:
  * comments
  * licensing
  * resolving testing nits

* updated README

* setup sharing of base analysis_options

* Update packages/mediapipe-core/lib/src/containers.dart

Co-authored-by: Kate Lovett <katelovett@google.com>

* Update packages/mediapipe-core/lib/src/containers.dart

Co-authored-by: Kate Lovett <katelovett@google.com>

* add free methods to core structs

* code review updates to options

* adds publish blocker

* Add GitHub actions for CI/CD (#14)

* adds CI/CD for mediapipe_core

* add newlines

* moves file into workflows dir

* uncomment flutter doctor

* testing PR config to run this now

* added master and beta CI scripts

* add executable permissions to CI scripts

* adds ffiwrapper to ci/cd

---------

Co-authored-by: Kate Lovett <katelovett@google.com>

* Add utility to collect headers from google/mediapipe (#10)

* adds cmd to pull header files from google/mediapipe

* polish and missing parts from git surgery

* More comments and touch ups

* Apply suggestions from code review

* moves build command into `tool/` directory

and renames folder `build_cmd` -> `builder`

* complete build_cmd -> builder rename

* Update readme

* Added licenses

* Adds DownloadModelCommand

---------

Co-authored-by: Kate Lovett <katelovett@google.com>

* [FFI] MediaPipe SDKs finder automation (#16)

* adds sdks_finder command to builder utility

* propagates changes to existing commands

* adds manifest files generated by new sdks_finder command

* updates in response to code review

* Adds mediapipe_text package (#12)

* adds mediapipe_text package

* Update .vscode/settings.json

added newline

* resync headers

* regenerated core bindings

* Apply suggestions from code review

* Adds example to `mediapipe-task-text` (#15)

* initial commit of example

* build file changes from `flutter pub get`

* update main.dart

* removes commented code

* updates example for isolates design

* Use `native-assets` to vendor MediaPipe SDK (#9)

* adding bare structure for native assets

* add MVP / first draft of build.dart

* build.dart updates

* update build.dart

TODO: stream file

* model memory troubleshooting

* vendoring script tweak

* remove development logging

* removes pointless build method

* Add utility to collect headers from google/mediapipe (#10)

* adds cmd to pull header files from google/mediapipe

* polish and missing parts from git surgery

* More comments and touch ups

* Apply suggestions from code review

* moves build command into `tool/` directory

and renames folder `build_cmd` -> `builder`

* complete build_cmd -> builder rename

* Update readme

* Added licenses

* Adds DownloadModelCommand

---------

Co-authored-by: Kate Lovett <katelovett@google.com>

* adds mediapipe_text package

* Update .vscode/settings.json

added newline

* resync headers

* regenerated core bindings

* native assets troubleshooting

this commit is broken

* Removes redundant count field

* update build.dart for correct bindings path

* download text classification model for CI

* better memory freeing in executor

* added SafeArea to example

* added CI to PRs into text package

* ci tooling change

* remove accidentally commited model

* more CI shenanigans

* lowers minimum Dart version for builder

* added smoke test for text example

* Added CI/CD for examples

* More CI tweaks

* entering "please work" territory

* d'oh

* trying more random stuff

* one more time

* it'd be funny if this helped

* more print statements

* enable reaching new print statements

* more logging

* see what's in build dir

* another test

* adding flutter config list

* turn off fail-fast for beta and master

* moar logs

* way moar prints

* moare things

* commit rest of rename

* moar whatevers

* adds manifest files generated by new sdks_finder command

* adds sdks_finder command to builder utility

* propagates changes to existing commands

* updates in response to code review

* updates to build.dart and tests

* add Android runtime

* sdks_finder logging improvement for when build folders change names

* refreshed symbols from google/mediapipe

* cleanup

* loosens closeness thresholds in integration tests

* separate build commands for macos architectures

* restores fail-fast setting to CI

* removed stale logging statements from CI

* removes accidentally committed lines

* add formatting of sdk_downloads.dart for CI

* fixes broken example test

* code touch ups from @Piinks code review

---------

Co-authored-by: Kate Lovett <katelovett@google.com>

* added base Dart class ClassificationResult to consolidate results logic in `mediapipe_core`

in doing so, removed meaningless TextClassifierResult.timestamp field

* Update Makefile

* Update Makefile

* Update packages/mediapipe-task-text/build.dart

Co-authored-by: Kate Lovett <katelovett@google.com>

* Update packages/mediapipe-task-text/lib/src/tasks/text_classification/text_classification_executor.dart

Co-authored-by: Kate Lovett <katelovett@google.com>

* code review responses

* formatting

* removes stale comment

* adds Dart to Native converters, with tests

* changes from code review

* updates mediapipe-core to prepare for IO/web split

* Improves memory management in core tests

* updated ffigen / bindings

* refactors text package for better memory management and eventual web/io split

* removed stale test

* cleanup on aisle COMMENTS

* Comments and documentation improvements

* Moved log statement

* Removed native memory management helpers in favor of `free` extensions on pointers

* Renamed abstract classes to have Base prefix

* sorted out class constructors

* moved `fake` constructor to default unnamed constructor
* leaning on the fact that the native constructors will be hidden by conditional exports, reducing confusion

* Improved docstrings explaining memory ownership

* Convert lists to lazy iterable / generators

* added missing licenses

* Update packages/mediapipe-task-text/example/test/widgets_test.dart

Co-authored-by: Kate Lovett <katelovett@google.com>

* Update packages/mediapipe-task-text/example/lib/main.dart

Co-authored-by: Kate Lovett <katelovett@google.com>

* completed return style change

* CI troubleshooting

* moved around debugging code

* logging tweak

* cat native-assets.yaml

* moar logs

* removed bad echo

* fixed native-assets typo

should be underscore!

* Removes CI debugging statements

---------

Co-authored-by: Kate Lovett <katelovett@google.com>

* Native Assets CI fix (#20)

* adds native assets debugging statements

* Try only downloading target arch

* Revert "adds native assets debugging statements"

This reverts commit b2bc215.

---------

Co-authored-by: Daco Harkes <dacoharkes@google.com>

* Text Embedding task (#21)

* updated and re-ran generators

* added embedding concepts to mediapipe-core

* fixed embedding header file and bindings

* adds text embedding classes to text pkg

* updates example with text embedding

* removed dead file

* added more embedding tests

* added embedding model download to CI script

* touch ups

* Update packages/mediapipe-core/lib/src/io/containers.dart

Co-authored-by: Kate Lovett <katelovett@google.com>

* Update packages/mediapipe-task-text/example/.gitignore

Co-authored-by: Kate Lovett <katelovett@google.com>

* Update packages/mediapipe-task-text/example/lib/text_embedding_demo.dart

Co-authored-by: Kate Lovett <katelovett@google.com>

* moved worker dispose method to base class

* docstring & comment improvements

* throw exceptions in impossible code paths instead of returning null

* class hierarchy improvements

* fixed outdates tests

* cleaned up dispose methods

* various tidying

* fixed deprecation warning

* moves repeated widgets into helper method

---------

Co-authored-by: Kate Lovett <katelovett@google.com>

* updated and re-ran generators

* fixed embedding header file and bindings

* adds text embedding classes to text pkg

* moved worker dispose method to base class

* class hierarchy improvements

* cleaned up dispose methods

* initial commit of language detection task

* finishes language detection impl

* adds language detection demo

* backfilling improvements to classification and embedding

* adds language detection tests

* add new model download to CI script

* fixes stale classification widget test, adds language detection widget test

* initial inference commit of flutter create -t package

* rename inference folder

* adds build tooling for inference headers and ffigen

* rename "inference" to "genai"

* adds initial inference impl

* flutter creat example for genai

* sdks crawling update

* adds genai
* adds model name to output file
* looks backwards through time until a build is good instead of betting the house that the most recent build was good

* adds android project to genai example

* adds ios project to genai example

* updates to macos project in genai example

* inference impl improvements

* inference example - nearly working

* various cleanup

* inference demo updates

* model path debugging

* inference demo improvements

* improvements to inference demo

* genai inference example improvements

* updates to inference and example

* fixed issue with chat screens sometimes not refreshing while the LLM was responding

problem came from state.transcript changes not busting the hashCode cache

* updated SDK manifests

* removed unused ffi utils copy param

---------

Co-authored-by: Kate Lovett <katelovett@google.com>
Co-authored-by: Daco Harkes <dacoharkes@google.com>

Author: 3 people

.github/workflows/main.yaml

@@ -5,7 +5,7 @@ permissions: read-all
 
 on:
   pull_request:
-    branches: [ main ]
+    branches: [main]
   push:
     branches: [main, ffi-wrapper, ffi-wrapper-text-pkg]
   schedule:

Makefile

@@ -57,4 +57,17 @@ test_text:
 	cd packages/mediapipe-task-text/example && flutter test
 
 example_text:
-	cd packages/mediapipe-task-text/example && flutter run -d macos
+	cd packages/mediapipe-task-text/example && flutter run -d macos
+
+# GenAI ---
+generate_genai:
+	cd packages/mediapipe-task-genai && dart --enable-experiment=native-assets run ffigen --config=ffigen.yaml
+
+# Example genai invocation.
+# Note that `GEMMA_4B_CPU_URI` can either be a local path or web URL. Similar values exist for
+# 8B and GPU variants.
+#
+# For desktop development, standard environment variables like this work great.
+# $ GEMMA_4B_CPU_URI=/path/to/gemma-2b-it-cpu-int4.bin flutter run -d [macos, windows, linux]
+# For emulator or attached device testing, use `--dart-define` for the same values.
+# $ flutter run -d [<device_id>] --dart-define=GEMMA_4B_CPU_URI=https://url/to.com/gemma-2b-it-cpu-int4.bin

packages/mediapipe-core/lib/src/interface/task_options.dart

@@ -6,9 +6,12 @@ import 'dart:typed_data';
 
 import 'package:equatable/equatable.dart';
 
-/// {@template TaskOptions}
+/// {@template BaseOptions}
 /// Root class for options classes for MediaPipe tasks.
-///
+/// {@endtemplate}
+abstract class Options extends Equatable {}
+
+/// {@template TaskOptions}
 /// Implementing classes will contain two [BaseInnerTaskOptions] subclasses,
 /// including a descendent of the universal options struct, [BaseBaseOptions].
 /// The second field will be task-specific.
@@ -17,7 +20,7 @@ import 'package:equatable/equatable.dart';
 /// This implementation is not immutable to track whether `dispose` has been
 /// called. All values used by pkg:equatable are in fact immutable.
 // ignore: must_be_immutable
-abstract class BaseTaskOptions extends Equatable {
+abstract class BaseTaskOptions extends Options {
   /// {@macro TaskOptions}
   BaseTaskOptions();
 

packages/mediapipe-core/lib/src/io/ffi_utils.dart

@@ -90,11 +90,11 @@ extension DartAwarePointerChars on Pointer<Pointer<Char>> {
   ///
   /// See also:
   ///  * [toDartString], for a non-list equivalent.
-  List<String?> toDartStrings(int length) {
+  List<String> toDartStrings(int length) {
     if (isNullPointer) {
       throw Exception('Unexpectedly called `toDartStrings` on nullptr');
     }
-    final dartStrings = <String?>[];
+    final dartStrings = <String>[];
     int counter = 0;
     while (counter < length) {
       dartStrings.add(this[counter].toDartString());

packages/mediapipe-core/lib/src/io/task_options.dart

@@ -15,7 +15,7 @@ import 'third_party/mediapipe/generated/mediapipe_common_bindings.dart'
 /// should manage their [InnerTaskOptions] fields. The two suggested methods are
 /// [copyToNative] and [dispose].
 /// {@endtemplate}
-mixin TaskOptions<T extends Struct> on BaseTaskOptions {
+mixin TaskOptions<T extends Struct> on Options {
   /// {@template TaskOptions.copyToNative}
   /// Copies these task options into native memory. Any fields of type
   /// [InnerTaskOptions] should have their `assignToStruct` method called.

packages/mediapipe-core/test/io/task_options_test.dart

@@ -86,13 +86,17 @@ void main() {
         expect(ptr.ref.score_threshold, lessThan(0.90001));
         expect(ptr.ref.category_allowlist_count, 3);
         expect(
-          ptr.ref.category_allowlist.toDartStrings(3),
+          ptr.ref.category_allowlist.toDartStrings(
+            3,
+          ),
           ['good', 'great', 'best'],
         );
 
         expect(ptr.ref.category_denylist_count, 4);
         expect(
-          ptr.ref.category_denylist.toDartStrings(4),
+          ptr.ref.category_denylist.toDartStrings(
+            4,
+          ),
           ['bad', 'terrible', 'worst', 'honestly come on'],
         );
       });

packages/mediapipe-task-audio/sdk_downloads.dart

@@ -1,2 +1,4 @@
 // Generated file. Do not manually edit.
-final Map<String, Map<String, String>> sdkDownloadUrls = {};
+// Used by the flutter toolchain (via build.dart) during compilation of any
+// Flutter app using this package.
+final Map<String, Map<String, Map<String, String>>> sdkDownloadUrls = {};

packages/mediapipe-task-genai/.gitignore

@@ -0,0 +1,29 @@
+# Miscellaneous
+*.class
+*.log
+*.pyc
+*.swp
+.DS_Store
+.atom/
+.buildlog/
+.history
+.svn/
+migrate_working_dir/
+
+# IntelliJ related
+*.iml
+*.ipr
+*.iws
+.idea/
+
+# The .vscode folder contains launch configuration and tasks you configure in
+# VS Code which you may wish to be included in version control, so this line
+# is commented out by default.
+#.vscode/
+
+# Flutter/Dart/Pub related
+# Libraries should not include pubspec.lock, per https://dart.dev/guides/libraries/private-files#pubspeclock.
+/pubspec.lock
+**/doc/api/
+.dart_tool/
+build/

packages/mediapipe-task-genai/.metadata

@@ -0,0 +1,10 @@
+# This file tracks properties of this Flutter project.
+# Used by Flutter tool to assess capabilities and perform upgrades etc.
+#
+# This file should be version controlled and should not be manually edited.
+
+version:
+  revision: "bd909542a33ab1d5249363a2434ae50ee468094f"
+  channel: "master"
+
+project_type: package

packages/mediapipe-task-genai/CHANGELOG.md

@@ -0,0 +1,3 @@
+## 0.0.1
+
+* TODO: Describe initial release.