Code Generation Overview

High Level Overview

The basic flow for our code generation is as follows:

1. For each metadata folder in source/codegen/metadata we run code generation
2. Our build runs generate_service.py as seen in CMakeLists.txt passing in the metadata driver folder
3. generate_service.py generates files like the proto, service.h, service.cpp, etc. for the driver using corresponding .mako file templates, the driver's metadata folder, and some python helper functions.
4. The build generates gRPC C++ source files from the .proto file we generated in the previous step using google's protoc (proto compiler) as seen here in CMakeLists.txt
5. All of the generated code will get placed in the source/generated directory. You shouldn't need to do edit this folder by hand, instead run a successful CMake build to have it re-generate.
6. Everything is built together into the server executable here including our generated files and hand-written ones.

Metadata Source and Manual Updates

Metadata Source

A lot of our metadata was originally based on the nimi-python repo's meatadata. The first step was to copy one of the metadata folders like niscope's metadata over to our source/codegen/metadata folder. Currently some of our metadata was based on nimi-python with manual updates while some is generated from our scrapigen repo. Eventually, we hope to plug into the hapigen process to have our metadata sourced from there like other driver derived metadata does. But until then, the manual updates to nimi-python metadata and those pulled from the scrapigen repo are stored in the source/codegen/metadata folder.

Required Updates

1. Methods for initializing a session should be marked with 'init_method': True.
   1. This is so the correct service body is generated to register the new session with the session_repository so clients can operate on it through the SessionUtilities service.

Common manual updates

1. Add driver API methods that were left out in nimi-python
2. Remove methods present in nimi-python metadata not in API header file (these were special for the nimi python interface)
3. Mark private methods public You can look through the existing CHANGES.md file to get a feel what kind of manual updates were made or look at the history for the metadata folder itself.

Custom Service Implementations

The build expects a _service.custom.cpp file in source/custom directory to contain any custom service implementations. This can be empty like for niswitch_service.custom.cpp.

If you end up needing a custom implementation for a method because it's not generating correctly, you can

1. Mark the method in functions.py as 'codegen_method': 'CustomCode' such as Fetch for niscope.
2. Add a custom implementation in the _service.custom.cpp file such as the Fetch implementation added for niscope.

Proto Generation

As stated in the High Level View, generate_service.py invokes the generation of the driver's .proto file. Specifically, it will pass the driver's metadata folder to the proto.mako template to generate the .proto file.

You can look at proto.mako for specifics but generally it operates on the different metadata's python files it takes in to know what service methods to make, the naming of the service, namespaces, methods, etc. And for each service method it makes a Request and Response message to capture the API method's input and output parameters, respectively.

It will also generate proto enums corresponding to the attributes.py and enums.py that feed into the service methods (these correspond to #define constants in the driver's .h file)

Protoc Compiler for Proto Files

After the proto file for the driver is generated, there's another step to compile it into the language you're consuming it in. For our examples, this is python but for our service implementation built with the server this will be C++.

You can see the generation of the C++ interface to the proto file incorporated in the build here in CMakeLists.txt. This uses the .proto file we generated earlier using generate_service.py and then incorporates the built files into the server executable build for our service implementation to reference.

Service Implementation Generation

Similar to how the proto file is generated, through generate_service.py we generate the service.h, service.cpp, and supporting files using their corresponding .mako templates.

The service.h and service.cpp define the service specified in the proto file and the corresponding methods. These are registered with the server to be able to respond to incoming requests on the service. For most of the methods in the metadata, we can generically generate a corresponding service method implementation. The exceptions to this rule have been marked so we know to skip over them and hand-code their implementation in a service.custom.cpp file that gets pulled into the build.