Skip to content

Latest commit

 

History

History
277 lines (227 loc) · 14.7 KB

add_function.md

File metadata and controls

277 lines (227 loc) · 14.7 KB

Adding a new function

Overview

This document demonstrates how to add new functions (also called as layer functions) to NNabla.
The following instruction demonstrates the new layer addition procedure for Ubuntu 16.04 platform.
However instructions for macOS and Windows platforms are also same except NNabla building part.

Prerequisites

Download the Neural Network Libraries from https://github.com/sony/nnabla and build from source.
For detailed instructions on building NNabla from source please see Build Manuals.

Work-flow for adding a new function

Following are the steps to be followed to add new layer function.

  1. Write a definition in YAML.
    1. Write function definition in functions.yaml.
    2. Write type specification of the new function in function_types.yaml.
  2. Generate new function code.
  3. Implement new function logic.
  4. Write unit test cases for new function.
  5. Write benchmark unit test cases for new function (optional).
  6. Write function documentation.

Write a definition in YAML

Function definition (functions.yaml)

First step for adding new layer function is to define the function definition in functions.yaml.

Following is the typical syntax for function definition in functions.yaml.

  class_name_of_new_function:
    snake_name: name_of_new_function
    doc: |2

      Write description for the functionality of new layer function.
    inputs:
      input_1:
        doc: Describe the input parameters.
        optional: boolean value indicating if input is optional.
        parameter: boolean value indicating if input is parameter.
      input_2:
        .
        .
        .  
      input_n:
    arguments:
      arg_1:
        doc: Describe the argument.
        type: Describe type of the argument.
        default: default value of the argument
      arg_2:
        .
        .
        .
      arg_n:
    outputs:
      y:
        doc: Describe the output parameter.

Following is example function definition of sum function in nnabla.

  Sum:
    snake_name: sum
    doc: |2
      Reduces a matrix along a specified axis with the sum function.
    inputs:
      x:
        doc: N-D array.
    arguments:
      axes:
        doc: Axes to be reduced. If empty list is given, all dimensions are reduced
          to scalar.
        type: repeated int64
        default: range(x.ndim)
      keep_dims:
        doc: Flag whether the reduced axis is kept.
        type: bool
        default: 'False'
    outputs:
      y:
	doc: N-D array

Note: Please ignore function_ids . This entry is the function id and its history used by internal code generation, and it is updated when compile NNabla.

Template type specification (function_types.yaml)

After adding the function definition, next step is to add a type specification of the new function to function_types.yaml.
A function class generated by the code generator and most of the existing functions are written as a template class, in which storage types are defined as template argument(s) of the class.

Using the storage type defined in function_types.yaml file, the build system will generate a source file in which template instantiation with specified types are written.
Also, the typed functions are registered to a function factory so that a typed function can be queried with a type configuration name.

NNabla supports two storage types, which will be used internally to cast input and output data while executing. Following are the storage types currently supported in nnabla.

  1. float: Represents any basic data types. E.g. float, int.
  2. half: Represents any basic data types (E.g. int) or 16-bit floating point internal data type Half.

We can set the storage type while setting the context as follows.

    # Get context.
    from nnabla.ext_utils import get_extension_context
    # for 32-bit floating point data type, set type_config='float'
    ctx = get_extension_context(
        args.context, device_id=args.device_id, type_config='float')

			    or

    # for 16-bit floating point data type, set type_config='half'
    ctx = get_extension_context(
        args.context, device_id=args.device_id, type_config='half')

    nn.set_default_context(ctx)

Following is the syntax of function_types.yaml.

class_name_of_new_function:
  storage_type_name: [data_type_1, data_type_2 .. data_type_n ]
  storage_type_name: [data_type_1, data_type_2 .. data_type_n ]

Following is example for type specification of Embed function in nnabla.

Embed:
  float: [int, float]
  half: [int, Half]

Generate code template and interfaces by CMake

After editing the .yaml files, next step is to generate the template class for new function.
By running cmake as described in the build instruction (Please refer Build Manuals.), our build system will create the following files using the function definition.

cd build
cmake ..

  • Function class template (created only if they don't exist, and should be added to the Git version control)

    • include/nbla/function/${snake_name}.hpp
    • src/nbla/function/generic/${snake_name}.cpp
    • python/src/nnabla/backward_function/${snake_name}.py

  • Template type instantiation of functions (overwritten)

    • src/nbla/function/${snake_name}.cpp

  • Initialization, function registration to factories (overwritten)

    • src/nbla/init.cpp

  • Python interfaces at python/src/nnabla (overwritten)

    • _version.py
    • function.pxd
    • function.pyx
    • function_bases.py

  • Serializer/Deserializer (overwritten)

    • python/src/nnabla/utils/load_function.py
    • python/src/nnabla/utils/save_function.py
    • src/nbla_utils/nnp_impl_create_function.cpp

Although the function template are useful, element-wise operation functions such as an activation function can be written by using macro functions.
Various macros are provided for easily defining functions with common structures. Please look inside existing functions for examples of these macros.
If you are adding new layer function which does element-wise operations, then you need not implement the generated template class.
Instead you can use the element-wise macro provided by nnabla.
Following are the steps to implement the element-wise operation based new function.

  1. In the ${snake_name}.cpp register the function with NBLA_REGISTER_FUNCTION_SOURCE(..) macro.
    Delete the auto generated setup_impl,forward_impl,backward_impl functions from generated ${snake_name}.cpp.
  2. Update the ${snake_name}.hpp with new function implementation using element-wise macro.
    NBLA_DEFINE_TRANSFORM_UNARY macro is used for element-wise operations.
    Please refer implementation of add_scalar function at cpp file and hpp file.

Write your function implementation

The template files generated using cmake contains the class definition for new function. The new function class inherits from the abstract base class BaseFunction which in turn inherits from Function class.
In order to add new layer function to nnabla you must implement setup, forward, and backward functions.

Implement the following three functions in the C++ file by adding your function logic:

  1. void setup_impl(const Variables &inputs, const Variables &outputs):
    Usually this function is used to initialize output parameters.
    The setup_impl is called only once during the forward pass of the training.
    It can be called more than once during the backward pass of the training.
    It is recommended to Keep the setup_impl implementation light weight.
  2. void forward_impl(const Variables &inputs, const Variables &outputs):
    In this function implement the new function logic for the forward pass.
  3. void backward_impl(const Variables &inputs, const Variables &outputs, const vector &propagate_down, const vector &accum):
    In this function implement the new function logic for the backward pass.
    Apart from input and output parameters, the backward_impl() function comes with 2 special parameters. I.e.
    1. propagate_down: The parameter propagate_down tells whether the gradients for the backward pass needs to be calculated or not. If its value is false then no need to calculate the gradients, the function can simply return.
    2. accum: The accum parameter tells whether to accumulate gradients from previous layer or not. If it is true then accumulate by summing the gradients. If it is set to false then just update the gradients.
      Note: The variable names must be valid names for Cython, Python and C++.
      For example, the variable name lambda cannot be used for keywords, since it is an invalid name in Cython and Python, although it is valid in C++.

Optionally, if you need the higher-order gradients of the new function added,

  1. Implement def ${snake_name}_backward(self, inputs, **kwargs) in python/src/nnabla/backward_function/${snake_name}.py:
    This method implements the backward pass of the function added in the python-layer, meaning the gradients of the function. See sigmoid.py, relu.py, and/or affine.py for details.
  2. Call nnabla.backward_functions.register(<FunctionName>, ${snake_name}_backward):
    If you add the new function based on the functions.yaml, this registration is automatically done at the compile-time, you do not need to do any registration. If you add a new PythonFunction (python-layer-only way to add a function), you have to do registration manually. See backward_functions.py.tmpl and backward_function.py for more details.

Write unit testing

It is good practice to test your new function implementation by writing the unit test cases.
You can add unit test cases for C++ version of your function implementation as well as for python version.

For adding C++ unit test cases: All C++ based unit-tests are found here.
NNabla uses Google Test unit test framework for C++ unit testing. You can learn more about Google Test from here.
Following are steps for adding C++ based test cases:

  1. Design your test cases.
  2. Create test_${snake_name}.cpp under nnabla/src/nbla/test folder.
  3. Implement the designed test cases in test_${snake_name}.cpp.
  4. //TODO need to write how to execute C++ unit test without using nnabla-builder.

For adding python unit test cases: All python based function unit-tests are found here.
NNabla uses pytest unit test framework for python unit testing. You can learn more about pytest from here.
Following are steps for adding C++ based test cases:

  1. Design your test cases.
  2. Create test_${snake_name}.py under nnabla/python/test/function folder.
  3. Implement the designed test cases in test_${snake_name}.py
  4. Execute the test cases by following instructions under Verify installation by unit testing section of Build Manuals.

NOTE:
Unit testing in C++ is often demanded by embedded engineers.
But writing unit testing in C++ requires more effort, and imposes double-maintenance cost over Python unit-tests.
Also, testing in Python is much easier to debug.
To reduce the effort required for C++ test-case implementation, currently we are thinking of creating a framework which can generate C++ unit test code of each functions from our Python testing framework (function_tester).
However, we do not have any specific plan to add this feature soon. Contributions are welcome.

Write benchmark unit testing

Write benchmark test cases to measure the performance of your new function implementation.
It is optional to write benchmark test cases, but it is good practice to measure the performance of your new function.
Following are steps for adding benchmark python based test cases:

  1. Create the test_${snake_name}.py file at python/benchmark/function folder (i.e. here).
  2. Implement the benchmark related test cases into test_${snake_name}.py file.
  3. Please follow the instructions in Benchmark readme for the execution of the benchmark test cases.

(DO NOT FORGET!) Add a function doc to sphinx.

It is very important to explain the behavior of input and output parameters of your new function implementation.
You can add the new function documentation to following files to generate the usage document.

  1. function_types.yaml.
  2. include/nbla/function/${snake_name}.hpp
  3. doc/python/api/function.rst

You need to edit the doc/python/api/function.rst (i.e. here) to make docs appear in the Python API docs.
Add the new layer function entry manually in function.rst file for including documentation in the Python API page.

After editing the above files with details of new function usage, compile the files to generate the documents.

cd ${nnabla_root_directory}
mkdir ${doc_dir_name}
cd ${doc_dir_name}
cmake -DPYTHON_VERSION=3.10 -DBUILD_CPP_LIB=ON -DBUILD_CPP_UTILS=OFF -DNNABLA_UTILS_STATIC_LINK_DEPS=ON -DBUILD_PYTHON_PACKAGE=ON ..
cd ..
make -C ${doc_dir_name} all wheel doc

The above command will generate doxygen based documentation and sphinx documentation under ${nnabla_root_directory}/${doc_dir_name}/doc/.

You want to add a CUDA function too?

We know many developers use CUDA to accelerate processing speed. See contribution guide of CUDA extension.