[lite/micro] quantized dense layer is not supported.

[csukuangfj]

The following error is thrown while running a quantized tf lite model converted from a tensorflow model using tf lite micro.

Quantized FullyConnected expects output data type uint8 or int16
Node FULLY_CONNECTED (number 1) failed to invoke with status 1

Environment information

• The master branch is used to compile lite/micro.
• OS: Ubuntu 16.04
• Gcc version: 5.4.0

tensorflow version

1.13.1

The script to convert the tensorflow model to a tflite model is as follows:

#!/usr/bin/env python

from __future__ import print_function, absolute_import, division

import argparse
import os
import warnings

import numpy as np

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
warnings.simplefilter(action='ignore', category=FutureWarning)

import tensorflow as tf
tf.logging.set_verbosity(tf.logging.ERROR)


def test():
    # first build a graph
    g = tf.Graph()

    with g.as_default():
        in_shape = [1, 10, 9, 3]
        x = tf.placeholder(tf.float32, shape=in_shape, name="input")

        w = np.random.rand(3, 3)
        b = np.random.rand(9)

        layer = tf.layers.conv2d(inputs=x,
                                 filters=9,
                                 kernel_size=w.shape,
                                 name="layer1",
                                 kernel_initializer=tf.constant_initializer(w),
                                 bias_initializer=tf.constant_initializer(b))
        relu = tf.nn.relu(layer)

        relu = tf.reshape(relu, (1, -1))

        w2 = np.random.rand(relu.shape[-1])
        b2 = np.random.rand(9)
        y = tf.layers.dense(inputs=relu,
                            units=9,
                            kernel_initializer=tf.constant_initializer(w2),
                            bias_initializer=tf.constant_initializer(b2),
                            name="output")

    with tf.Session(graph=g) as sess:
        sess.run(tf.global_variables_initializer())
        converter = tf.lite.TFLiteConverter.from_session(sess,
                                                         input_tensors=[x],
                                                         output_tensors=[y])

        print("before conversion")
        in_data = np.arange(270).reshape(x.shape)
        out = sess.run(y, feed_dict={x: in_data})
        print(out)

        converter.post_training_quantize = True  # for 1.13.1

        tflite_model = converter.convert()
        model_filename = "xxx.tflite"
        open(model_filename, "wb").write(tflite_model)

        cc_src = "xxx.h"
        os.system("xxd -i {} > {}".format(model_filename, cc_src))

    # after conversion
    interpreter = tf.lite.Interpreter(model_path=model_filename)
    interpreter.allocate_tensors()
    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()
    input_shape = input_details[0]['shape']
    interpreter.set_tensor(input_details[0]['index'],
                           in_data.astype(np.float32))
    interpreter.invoke()
    output_data = interpreter.get_tensor(output_details[0]['index'])
    print("after conversion")
    print(output_data)


if __name__ == '__main__':
    print(tf.__version__)
    print(tf.__git_version__)
    print(tf.__compiler_version__)
    np.random.seed(777)
    test()

Output of the above script is:

1.13.1
b'v1.13.1-0-g6612da8951'
4.8.5
before conversion
[[327913.94 327228.56 327169.47 327523.12 326526.3  327757.97 328415.53
  326741.97 327555.22]]
after conversion
[[327600.28 326899.12 326834.88 327196.44 326194.53 327430.94 328063.88
  326408.12 327227.72]]

[njeffrie]

It looks like you're generating a model which uses hybrid quantization. Micro currently only supports full-quanitzed or full-float models. If you want to model to work today, the best way forward is to avoid quantizing the model in your conversion step. When we finish up landing the int8 quantized version of CONV2D, you will be able to run a fully quantized model. In order to fully quantize your model, you will need to provide a representative dataset.

[csukuangfj]

@njeffrie
could you tell me how to use full-quantized only? I don't care whether it is full-quantized or hybrid-quantized.