PWV Audit Reproducible Build Pipeline Suggested Changes

[pwvpwvpwv]

Reproducible Build Pipeline Audit/Suggestions

Updates made

The build system PR currently includes the following updates:

1. Updates the manifest file, so that only one language can be specified.
2. Created a Dockerfile.mustache which can be used as a template, and is compiled before building docker image.
3. Created a build manifest file which specifies required configuration variables, their types, and the shape of the payload expected.
4. Created a configuration file (web3api.build.yaml) which specifies the dockerfile to use, along with variables being sent to the mustache template for outputting the final dockerfile.

Pros

• Highly customizable build process with customization available for any user via mustache templates allowing the Dockerfile to be updated easily.
• Everything out of the box just works for developers and build should be the same across any system.
• Any form of customization is saved locally in the repository, allowing for shared build processes with all people working on a project.

Cons

• It is relatively complex, and may be daunting for some developers. Particularly those not familiar with Docker.
• The templating of the Dockerfile can lead to confusing errors, as there is a middleware process that is building the actual Dockerfile process being used.
• Limits build systems to a single language - IE you can't have some modules that are written in Rust, once support is available, and others written in AssemblyScript.
• We find the configuration/manifest files to be verbose, and difficult to write by hand.

Concerns

Development Overhead

Docker can be daunting for some developers, and one of the goals of this project is to lower the necessary knowledge for developers to get up and running as quickly and efficiently as possible. However, we find the current system as implemented isn't very straight-forward.

The use of the Mustache templating engine is a bit difficult to read in the docker container. We feel the {{}} template tags feel verbose and make it difficult to understand what is happening. It also requires a lot of reverse engineering to understand how the Dockerfile is generated during the build process.

We also believe that writing the schema of the template context is difficult to write by hand. While JSON is leaps and bounds ahead of XML in this regard, we still think it's mostly a data format best handled by having a machine create/read it.

Recommendations/Considerations

1. As pointed out, environmental variables cause more headaches for this process than not. We recommend having a simple interpolation system has a very simple annotation. The annotation system will replace all of the mustache template engine that is currently developed. Our thought is something like the list below for the annotation system (feel free to change):

   1. $varname for simple variables that need to be passed. Vars should be letters and numbers only to create a boundary and allow them to stand out more in the document. That way $nodeVersion-alpine only gets interpolated for the $nodeVersion.
   2. $listname[] for a list of items that should be passed
   3. $!varname or $!listname[] for a required field
   4. After a list is defined, we need a !! to denote the end of the interpolation.
   5. If a list is just a list of primitive values then $1 will denote it's value.
   6. If a list is an object properties can be accessed via $prop_name within the list.

   This annotation system allows for removal of the schema as well, as it's not part of the actual docker file. It also follow rules similar to GraphQL which is used heavily in the repository, so developers should feel relatively familiar and comfortable with it. In the end the docker file would look something like this:

   FROM node:$!nodeVersion-alpine as base  
     
   RUN apk --no-cache --virtual build-dependencies add \  
       bash \  
       git \  
       openssh \  
       make \  
       g++  
     
   WORKDIR /project  
     
   # Install deps in its own step, making rebuilds faster  
   # when just the Web3API schema & implementation files change  
   COPY package.json .  
   RUN yarn  
     
   # Copy all manifest files  
   $web3apiManifests[]  
   COPY $1 .  
   !!
     
   # Copy all source files  
   $!include[]  
   COPY $1 $1 
   !!
   
   $web3apiModules[]  
   COPY $dir $dir   
   !!
   
   $web3apiModules[]  
   # Build the module at $dir  
   RUN ./node_modules/.bin/asc $dir/w3/entry.ts \  
       --path ./node_modules \  
       --outFile ./build/$name.wasm \  
       --use abort=$dir/w3/entry/w3Abort \  
       --optimize --debug --importMemory \  
       --runtime stub
   !!

   Another alternative to this format to simplify the file would be to remove the need for an end indicator for lists - so no !!. This would allow for a single line only, but since most of these are bash commands using the RUN module you can simply concatenate the commands with &&, or use the same list variables twice. Example below:

   FROM node:$!nodeVersion-alpine as base  
     
   RUN apk --no-cache --virtual build-dependencies add \  
       bash \  
       git \  
       openssh \  
       make \  
       g++  
     
   WORKDIR /project  
     
   # Install deps in its own step, making rebuilds faster  
   # when just the Web3API schema & implementation files change  
   COPY package.json .  
   RUN yarn  
     
   # Copy all manifest files  
   $web3apiManifests[]  
   COPY $1 .  
     
   # Copy all source files  
   $!include[]  
   COPY $1 $1
   
   # Run some command on all includes to copy to another folder
   $!include[]
   RUN cp $1 /usr/bin/$1 && chmod /usr/bin/$1 
   
   $web3apiModules[]  
   COPY $dir $dir   
   
   $web3apiModules[]  
   # Build the module at $dir  
   RUN ./node_modules/.bin/asc $dir/w3/entry.ts \  
       --path ./node_modules \  
       --outFile ./build/$name.wasm \  
       --use abort=$dir/w3/entry/w3Abort \  
       --optimize --debug --importMemory \  
       --runtime stub

2. Since we can now define whether or not something is required within the actual Dockerfile template, we don't see the need for the additional schema file. The types seems not so important, and would be easier for the developer to identify any issues that occur by viewing the interpolated Dockerfile.

   If necessary we could at minimum remove the web3api.build.json file, and implement needed functionality in the YAML manifest file instead. This removes the need for writing JSON by hand for the developers to YAML which is preferred in our opinion, and also removes the additional file. But again, unless types are really an important factor here it doesn't seem like there is much benefit from doing it with the annotation system implemented.

   A good example of this is the Swagger implementation. This would allow us to add a property to this file called schema which specifies whether something is required and not and the types. Below is an example:

     ...
   schema:
     SomeObject:
       type: object
       required:
         - name
         - totalOrders
       properties:
       name:
         type: string
       totalOrders:
         type: integer
     ...

[cbrzn]

I think this can be closed and (probably) re-ask for this with the new changes introduced in #986