Remove Flex OP dependency and run inference against new IDDetector #5404

ccen-stripe · 2022-08-13T01:34:17Z

Summary

The new IDDetector removed NMS from in graph in order to remove the flex op dependency.
Now the NMS algorithm needs to be perform from Android side.

Note: this change should not be submitted until the server side model is updated

Changes for the model inference:

Before

Input - one tensor
- tensor image of (1, 224, 224, 3), normalized to [-1, 1)
Output - two tensors
- best bounding box(1, 4)
- category(NO_ID, PASSPORT, ID_FRONT, ID_BACK, INVALID) scores of the box, (1, 5)

After

Input - one tensor
- tensor image of (1, 224, 224, 3), normalized to [0, 1)
Output - two tensors
- all bounding boxes (392, 4)
- all category(PASSPORT, ID_FRONT, ID_BACK, INVALID) scores (392, 4) - note NO_ID is removed

The algorithm is to find the best scores from category scores, then return its dimension 1 index [0, 392) as bestIndex.
Use that index to find best bounding box and best category

Motivation

Reduce SDK binary size

Testing

Added tests
Modified tests
Manually verified

Screenshots

Before	After
before screenshot	after screenshot

Changelog

github-actions · 2022-08-13T01:39:56Z

Diffuse output:

OLD: identity-example-release-base.apk (signature: none)
NEW: identity-example-release-pr.apk (signature: none)

          │            compressed             │            uncompressed            
          ├───────────┬──────────┬────────────┼───────────┬───────────┬────────────
 APK      │ old       │ new      │ diff       │ old       │ new       │ diff       
──────────┼───────────┼──────────┼────────────┼───────────┼───────────┼────────────
      dex │   6.3 MiB │  6.3 MiB │     -239 B │  17.8 MiB │  17.8 MiB │     -516 B 
     arsc │   1.1 MiB │  1.1 MiB │        0 B │   1.1 MiB │   1.1 MiB │        0 B 
 manifest │   1.9 KiB │  1.9 KiB │        0 B │   6.6 KiB │   6.6 KiB │        0 B 
      res │   385 KiB │  385 KiB │        0 B │ 589.2 KiB │ 589.2 KiB │        0 B 
   native │ 115.9 MiB │  5.7 MiB │ -110.2 MiB │ 377.5 MiB │  14.4 MiB │ -363.1 MiB 
    asset │   9.3 KiB │  9.3 KiB │       -7 B │     9 KiB │     9 KiB │       -7 B 
    other │  28.2 KiB │ 28.2 KiB │        0 B │  27.1 KiB │  27.1 KiB │        0 B 
──────────┼───────────┼──────────┼────────────┼───────────┼───────────┼────────────
    total │ 123.7 MiB │ 13.5 MiB │ -110.2 MiB │   397 MiB │  33.8 MiB │ -363.1 MiB 

         │          raw           │            unique             
         ├────────┬────────┬──────┼────────┬────────┬─────────────
 DEX     │ old    │ new    │ diff │ old    │ new    │ diff        
─────────┼────────┼────────┼──────┼────────┼────────┼─────────────
   files │      2 │      2 │    0 │        │        │             
 strings │ 116688 │ 116682 │   -6 │ 106868 │ 106862 │ -6 (+6 -12) 
   types │  16789 │  16787 │   -2 │  15738 │  15736 │ -2 (+0 -2)  
 classes │  14128 │  14126 │   -2 │  14128 │  14126 │ -2 (+0 -2)  
 methods │ 107754 │ 107747 │   -7 │ 104765 │ 104758 │ -7 (+2 -9)  
  fields │  76629 │  76627 │   -2 │  76385 │  76383 │ -2 (+3 -5)  

 ARSC    │ old  │ new  │ diff 
─────────┼──────┼──────┼──────
 configs │  259 │  259 │  0   
 entries │ 4760 │ 4760 │  0

APK

      compressed      │     uncompressed     │                                                 
─────────┬────────────┼─────────┬────────────┤                                                 
 size    │ diff       │ size    │ diff       │ path                                            
─────────┼────────────┼─────────┼────────────┼─────────────────────────────────────────────────
         │  -30.5 MiB │         │ -107.9 MiB │ - lib/x86_64/libtensorflowlite_flex_jni.so      
         │    -30 MiB │         │ -105.6 MiB │ - lib/x86/libtensorflowlite_flex_jni.so         
         │  -26.3 MiB │         │  -86.5 MiB │ - lib/arm64-v8a/libtensorflowlite_flex_jni.so   
         │  -23.5 MiB │         │  -63.2 MiB │ - lib/armeabi-v7a/libtensorflowlite_flex_jni.so 
 2.3 MiB │     -239 B │ 6.9 MiB │     -516 B │ ∆ classes2.dex                                  
 8.7 KiB │       -7 B │ 8.5 KiB │       -7 B │ ∆ assets/dexopt/baseline.prof                   
─────────┼────────────┼─────────┼────────────┼─────────────────────────────────────────────────
 2.3 MiB │ -110.2 MiB │ 6.9 MiB │ -363.1 MiB │ (total)

DEX

STRINGS:

   old    │ new    │ diff        
  ────────┼────────┼─────────────
   106868 │ 106862 │ -6 (+6 -12) 
  + :
  ���
  ��
  ���
  ��_
  ���
  ���
  ���
  ���
  ���
  �� 
  ���
  ���
  
  ���
  ������2�0�B���¢����R���������0�����0�0�¢��
  ������R��	��0�X�T¢��
  R��
  ��0�X�T¢��
  R�����0�X�T¢��
  R����0�X�T¢��
  R��
  ��0�X�T¢��
  R�����0�¢��
  ������R�����0�X�T¢��
  R���������0�0�¢��
  ������R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�¢��
  ��� �!R����0�X�T¢��
  R��#��0�¢��
  ���_�%¨�&
  + LIST_OF_INDICES
  + OUTPUT_SIZE
  + SMAP
  IDDetectorAnalyzer.kt
  Kotlin
  *S Kotlin
  *F
  + 1 IDDetectorAnalyzer.kt
  com/stripe/android/identity/ml/IDDetectorAnalyzer
  + 2 _Collections.kt
  kotlin/collections/CollectionsKt___CollectionsKt
  *L
  1#1,167:1
  1936#2,14:168
  1549#2:182
  1620#2,3:183
  *S KotlinDebug
  *F
  + 1 IDDetectorAnalyzer.kt
  com/stripe/android/identity/ml/IDDetectorAnalyzer
  *L
  83#1:168,14
  104#1:182
  104#1:183,3
  *E
  
  + access_getLIST_OF_INDICES_cp
  + getLIST_OF_INDICES
  
  - 2
  ���
  ��
  ���
  ��_
  ���
  ���
  ��	
  ���
  ���
  ���
  
  ���
  ������2�0�B���¢����R���������0�����0�0�¢��
  ������R��	��0�X�T¢��
  R��
  ��0�X�T¢��
  R�����0�X�T¢��
  R����0�X�T¢��
  R��
  ��0�X�T¢��
  R�����0�X�T¢��
  R�����0�¢��
  ������R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�X�T¢��
  R�����0�¢��
  ������R�����0�¢��
  ��� �!R����0�X�T¢��
  ¨�#
  - FlexDelegate.java
  - INDEX_NO_ID
  - Lorg/tensorflow/lite/flex/FlexDelegate;
  - Lorg/tensorflow/lite/flex/R;
  - SMAP
  IDDetectorAnalyzer.kt
  Kotlin
  *S Kotlin
  *F
  + 1 IDDetectorAnalyzer.kt
  com/stripe/android/identity/ml/IDDetectorAnalyzer
  + 2 _Collections.kt
  kotlin/collections/CollectionsKt___CollectionsKt
  + 3 _Arrays.kt
  kotlin/collections/ArraysKt___ArraysKt
  *L
  1#1,146:1
  1959#2,14:147
  11385#3:161
  11720#3,3:162
  *S KotlinDebug
  *F
  + 1 IDDetectorAnalyzer.kt
  com/stripe/android/identity/ml/IDDetectorAnalyzer
  *L
  73#1:147,14
  96#1:161
  96#1:162,3
  *E
  
  - TFLITE_FLEX_LIB
  - initTensorFlowForTesting
  - nativeCreateDelegate
  - nativeDeleteDelegate
  - nativeInitTensorFlow
  - tensorflowlite_flex_jni
  

TYPES:

   old   │ new   │ diff       
  ───────┼───────┼────────────
   15738 │ 15736 │ -2 (+0 -2) 
  - Lorg/tensorflow/lite/flex/FlexDelegate;
  - Lorg/tensorflow/lite/flex/R;
  

METHODS:

   old    │ new    │ diff       
  ────────┼────────┼────────────
   104765 │ 104758 │ -7 (+2 -9) 
  + com.stripe.android.identity.ml.IDDetectorAnalyzer_Companion getLIST_OF_INDICES() → List
  + com.stripe.android.identity.ml.IDDetectorAnalyzer access_getLIST_OF_INDICES_cp() → List
  
  - org.tensorflow.lite.flex.FlexDelegate <clinit>()
  - org.tensorflow.lite.flex.FlexDelegate <init>()
  - org.tensorflow.lite.flex.FlexDelegate close()
  - org.tensorflow.lite.flex.FlexDelegate getNativeHandle() → long
  - org.tensorflow.lite.flex.FlexDelegate initTensorFlowForTesting()
  - org.tensorflow.lite.flex.FlexDelegate nativeCreateDelegate() → long
  - org.tensorflow.lite.flex.FlexDelegate nativeDeleteDelegate(long)
  - org.tensorflow.lite.flex.FlexDelegate nativeInitTensorFlow() → long
  - org.tensorflow.lite.flex.R <init>()
  

FIELDS:

   old   │ new   │ diff       
  ───────┼───────┼────────────
   76385 │ 76383 │ -2 (+3 -5) 
  + com.stripe.android.identity.ml.IDDetectorAnalyzer_analyze_1 L_2: Object
  + com.stripe.android.identity.ml.IDDetectorAnalyzer LIST_OF_INDICES: List
  + com.stripe.android.identity.ml.IDDetectorAnalyzer OUTPUT_SIZE: int
  
  - com.stripe.android.identity.ml.IDDetectorAnalyzer INDEX_NO_ID: int
  - com.stripe.android.identity.ml.IDDetectorAnalyzer THRESHOLD: float
  - org.tensorflow.lite.flex.FlexDelegate INVALID_DELEGATE_HANDLE: long
  - org.tensorflow.lite.flex.FlexDelegate TFLITE_FLEX_LIB: String
  - org.tensorflow.lite.flex.FlexDelegate delegateHandle: long

stathis-stripe

LGTM

awush-stripe

Nice! this should drive down the SDK size a bit

ccen-stripe · 2022-08-19T18:28:52Z

Sorry needs to be rebase

ccen-stripe · 2022-08-19T18:29:56Z

Nice! this should drive down the SDK size a bit

Yes, the compressed size dropped from 123.7 MiB to 13.5 MiB 😱

ccen-stripe force-pushed the ccen/IDDetectorNoFlex branch from 41fddba to 0f3f2f4 Compare August 18, 2022 18:33

ccen-stripe changed the title ~~test TFLite without flex dep~~ Remove Flex OP dependency and run inference against new IDDetector Aug 18, 2022

ccen-stripe requested a review from stathis-stripe August 18, 2022 18:55

ccen-stripe marked this pull request as ready for review August 18, 2022 18:55

ccen-stripe requested a review from awush-stripe as a code owner August 18, 2022 18:55

ccen-stripe assigned stathis-stripe Aug 18, 2022

ccen-stripe requested review from brnunes-stripe, michelleb-stripe and jameswoo-stripe as code owners August 18, 2022 19:05

stathis-stripe previously approved these changes Aug 18, 2022

View reviewed changes

stripe-ci assigned ccen-stripe and unassigned stathis-stripe Aug 18, 2022

awush-stripe previously approved these changes Aug 18, 2022

View reviewed changes

ccen-stripe added 5 commits August 19, 2022 11:27

remove Flex OP dependency

a0c02f5

update scores

93ebd3f

update comments

c0fcd0a

lint

af32f52

add changelog

d7520e8

ccen-stripe dismissed stale reviews from awush-stripe and stathis-stripe via d7520e8 August 19, 2022 18:28

ccen-stripe force-pushed the ccen/IDDetectorNoFlex branch from 1e9bdbf to d7520e8 Compare August 19, 2022 18:28

ccen-stripe requested a review from awush-stripe August 19, 2022 18:28

awush-stripe approved these changes Aug 19, 2022

View reviewed changes

ccen-stripe merged commit ab343f5 into master Aug 19, 2022

ccen-stripe deleted the ccen/IDDetectorNoFlex branch August 19, 2022 20:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove Flex OP dependency and run inference against new IDDetector #5404

Remove Flex OP dependency and run inference against new IDDetector #5404

ccen-stripe commented Aug 13, 2022 •

edited

Loading

github-actions bot commented Aug 13, 2022 •

edited

Loading

stathis-stripe left a comment

awush-stripe left a comment

ccen-stripe commented Aug 19, 2022

ccen-stripe commented Aug 19, 2022

Remove Flex OP dependency and run inference against new IDDetector #5404

Remove Flex OP dependency and run inference against new IDDetector #5404

Conversation

ccen-stripe commented Aug 13, 2022 • edited Loading

Summary

Before

After

Motivation

Testing

Screenshots

Changelog

github-actions bot commented Aug 13, 2022 • edited Loading

stathis-stripe left a comment

Choose a reason for hiding this comment

awush-stripe left a comment

Choose a reason for hiding this comment

ccen-stripe commented Aug 19, 2022

ccen-stripe commented Aug 19, 2022

ccen-stripe commented Aug 13, 2022 •

edited

Loading

github-actions bot commented Aug 13, 2022 •

edited

Loading