Worker Indexing Pipeline

The Python Worker is being modified to enable it to perform indexing on a function app. This new functionality will enable Python developers to structure their function app directories in the way that they see fit.

Changes Needed For Implementation

New GRPC Messages

Link to first GRPC Protobuf PR

Link to second GRPC Protobuf PR

The first step to implement this pipeline is to ensure that proper communication logic is put in place between the host and the worker. To do this, we need new GRPC messages to handle worker indexing requests and responses. Here are the messages that have been created to facilitate this communication:

FunctionsMetadataRequest (host to worker)

Sends the function app directory to the worker.

{
    "directory": "some/script/path"
}

FunctionMetadataResponses (worker to host)

Includes a list of FunctionLoadRequest messages and an overall StatusResult attribute.

{
    "results": [
        {
            "functionId": "1234",
            "metadata": {
                          "name": "HttpTrigger",
                          "directory": "some/file/path",
                          "script_file": "some_file",
                          "entry_point": "some_entry_point",
                          "rawBindings": "[
                              'binding1',
                              'binding2'
                          ]",
                          "is_proxy": "false",
                          "status": "success"
                      },
            "managedDependencyEnabled": "true"
        }, 
        {
            "functionId": "1235",
            "metadata": {
                          "name": "HttpTrigger2",
                          "directory": "some/file/path2",
                          "script_file": "some_file2",
                          "entry_point": "some_entry_point2",
                          "rawBindings": "[
                              'binding1',
                              'binding2'
                          ]",
                          "is_proxy": "false",
                          "status": "success"
                      },
            "managedDependencyEnabled": "true"
        }
    ], 
    "overall_status": "success"
}

---

New handler in dispatcher.py for incoming FunctionsMetadataRequest messages

A new handler is needed to initiate the indexing process and return the function metadata through a new WorkerFunctionMetadataResponse message. A basic skeleton of the handler is shown below:

async def _handle_functions_metadata_request(self, req):
        metadata_request = req.functions_metadata_request
        directory = metadata_request.directory

        # get list of function metadata
        function_metadata = someFile.index_functions(directory)

        return protos.StreamingMessage(
            request_id=req.request_id,
            function_metadata_responses=protos.FunctionMetadataResponses(
                results=function_metadata,
                overall_status=protos.StatusResult(
                    status=protos.StatusResult.Success)))

---

New function in loader.py to house the indexing logic

This function needs to find all script files within the given function app directory.

From there, it needs to import the modules of each of these script files and check to see if they contain callable functions or not.

Finally, the callable functions needs to be indexed by storing the decorator parameters as defined binding and trigger objects.