Support for YAML and Helm chart static analysis

.github/scripts/setup_build_ubuntu-20.04.sh

@@ -3,4 +3,4 @@
 # Install required packages for CodeCompass build
 sudo apt-get install -y git cmake make g++ libboost-all-dev llvm-10-dev clang-10 \
  libclang-10-dev odb libodb-dev thrift-compiler libthrift-dev default-jdk libssl-dev \
- libgraphviz-dev libmagic-dev libgit2-dev ctags doxygen libgtest-dev npm libldap2-dev
+ libgraphviz-dev libmagic-dev libgit2-dev ctags doxygen libgtest-dev npm libldap2-dev libyaml-cpp-dev

.github/scripts/setup_runtime_ubuntu-20.04.sh

@@ -4,4 +4,4 @@
 sudo apt-get install -y git cmake make g++ graphviz \
  libboost-filesystem1.71.0 libboost-log1.71.0 libboost-program-options1.71.0 \
  libllvm10 clang-10 libclang1-10 libthrift-0.13.0 default-jre libssl1.1 libmagic1 \
- libgit2-28 ctags googletest libldap-2.4-2
+ libgit2-28 ctags googletest libldap-2.4-2 libyaml-cpp0.6

.github/workflows/ci.yml

@@ -256,4 +256,4 @@ jobs:
  if: ${{ github.repository == 'Ericsson/CodeCompass' && (github.ref_name == 'master' || startsWith(github.ref_name, 'release/') == true) }}
  uses: ./.github/workflows/tarball.yml
  secrets:
- GITLAB_TRIGGER_TOKEN: ${{ secrets.GITLAB_TRIGGER_TOKEN }}
+ GITLAB_TRIGGER_TOKEN: ${{ secrets.GITLAB_TRIGGER_TOKEN }}

doc/helm_documentation.md

@@ -0,0 +1,161 @@
+# Helm chart static analysis plugin
+## Developers' documentation
+
+The microservice plugin serves the purpose of analyzing Helm charts and storing data about the microservices and their relations in the database.
+
+The plugin uses the _yaml-cpp_ library to parse YAML nodes. 
+The analysis is executed file-by-file.
+**This plugin heavily relies on the rules and conventions of Helm and Kubernetes.**
+
+## YAML file analysis steps
+
+The microservice parser receives one or more root directories of Helm charts. 
+The directory is traversed in a recursive manner. 
+The plugin is capable of multi-threaded analysis, where each thread receives a file for analysis.
+
+### File purpose analysis
+
+First the parser checks the purpose of the YAML file in analysis.
+The following "purposes" (i.e. file types) are checked:
+
+- _Helm charts_: the actual chart files which contains the name and other metadata of the microservice.
+ Within this category, _integration charts_ and _subcharts_ are distinguished. 
+ Processing is somewhat different based on the subtype (see details below).
+ Files named _Chart.yaml_ or _Chart.yml_ are classified into this category.
+
+- _Helm values_: files which contain vital information of microservice relations and default values (e.g. for environment variables).
+ Files named _values.yaml_ or _values.yml_ are classified into this category.
+ These files are collected in a container for further analysis.
+
+- _Helm templates_: files that define microservice relations and metadata, such as ConfigMaps, Secrets, Certificates etc.
+ Files that are inside a directory named _templates_ are classified into this category.
+ These files are collected in a container for further analysis.
+
+- _Docker compose files_: files named _compose.yml, compose.yaml, docker_compose.yml, or docker_compose.yaml_ are classified into this category.
+
+- _CI files_: any file that does not belong to any category above and contains the substring _"ci"_, is classified as a continuous integration file.
+
+_Note:_ At this point, the plugin is more specifically aimed at the analysis of Helm charts.
+However, it can be used for generic YAML parsing as well, so the non-Helm types are left in.
+
+### YAML node analysis
+
+After the file type check, the keys and values in the file are analyzed in a recursive manner.
+_yaml-cpp_ distinguishes the following node types: scalar, sequence, map, and null and undefined.
+These types appear within the `SymbolType` field. 
+The nodes are processed according to their type.
+
+Generally, every node is broken down to the smallest, scalar node which is persisted in the database.
+However, a scalar holds the type of node in which it was included: e.g. if a scalar is originally a key in a map,
+its `AstType` will be `MAP`. Maps and sequences need to be parsed in slightly different ways, 
+because their iterators are not interchangeable. In the end, all nodes are broken down to scalars,
+which are then persisted in the database with their containing type.
+
+In order to provide better syntax highlight, a `SymbolType` field holds the place of a node:
+`Key`, `Value`, `NestedKey`, `NestedValue`, or `Other`.
+
+The location of nodes within files is calculated by the `Mark()` method of _yaml_cpp_,
+and the content of a node is extracted with the `Dump()` method.
+
+### Exception handling
+
+If any error happens during file analysis, the error will be saved as a `BuildLog` object. 
+The erroneous file will remain in _Not parsed_ status.
+
+Errors usually happen in Helm charts if the user did not execute the `helm template` command on the charts before parsing,
+or when there are syntax errors in the file.
+
+### Chart analysis
+
+As mentioned above, Helm charts should be handled differently if they are integration charts or subcharts.
+
+#### Integration charts
+
+If the file is an integration chart - a chart which describes the entire software, not just a component -,
+tha parser check if there is a _dependencies_ key in the file.
+If not, the service that the file describes is registered in the database.
+If there is, the listed microservices are registered in the database. 
+If a listed service contains an _alias_ key, its value should be persisted as the name of the service.
+Otherwise, the value of the obligatory _name_ key is the name of the service.
+
+_Note: this approach might be refined in the future to adapt to more possible architecture description formats._
+
+#### Subcharts
+
+If a _Chart.yaml_ file is contained by a _charts_ directory, it should be considered a subchart
+(i.e. a component in a microservice-based software).
+The described microservice should be persisted in the database.
+
+_Note: in case of parsing multiple projects, keep in mind that a full-value microservice (i.e.
+a microservice which is not defined in a subchart, but not necessarily an integration service)
+can define a component in within another project. Thus it can be a subchart in on project, and
+an integration chart on its own at the same time._
+
+### Template file analysis
+
+This analysis is implemented in the `TemplateAnalyzer` class.
+
+#### Dependencies
+
+After basic YAML parsing, the previously collected template files are analyzed to collect
+microservice dependencies.
+The content of each template file is investigated.
+Any template object should be persisted in the database as a `HelmTemplate` object with
+the file that it is defined in.
+Based on the content and its format, the analysis branches off in the following directions:
+
+- _Service dependency:_ if the value of the `kind` key in the template file is _Service_. 
+ This type defines additional services that derive from the original service.
+ In this case, the new service is persisted in the database as an _EXTERNAL_ service, and an edge
+ is created between the two services. The defined service depends on the defining service.
+- _Mount dependencies:_ these types are usually collected within big deployment files. 
+ The dependencies listed as the value of a _volumes_ key should be analyzed.
+ Please note that there can be multiple _volumes_ keys in a deployment file.
+ We consider two subtypes:
+ - _ConfigMaps_: these contain lots of important information that is needed for K8S pod definition.
+ - _Secrets_: these contain confidential information within services.
+ Both types can be identified by a `kind` key in the list.
+- _Certificate dependencies:_ while the previous types could be identified by a `kind` key
+ within a file, a certificate can be defined by several template types.
+ An approximate heuristics is to check the value of the `kind` key for containing the
+ "Certificate" or the "InternalUserCA" substring. (_WARNING:_ naming conventions may vary
+ in every project.) Certificate dependencies define further secrets which should also
+ be persisted in the database.
+
+#### Resources
+
+All template files are checked to see if they contain resource usage information.
+The following resources are collected:
+
+- _CPU:_ listed with `resources` and `requests` keys. This resource is calculated in
+ the number of CPU cores which can be a fraction.
+- _Memory:_ listed with `resources` and `requests` keys. Calculated in gigabytes.
+- _Storage:_ listed within a `volumeClaimTemplates` list, with a `storage` key. 
+ Calculated in gigabytes.
+
+### Analysis of _values_ files
+
+This analysis is implemented in the `ValueAnalyzer` class.
+
+After basic YAML parsing, the previously collected _values.yaml_ files are analyzed to find
+further connection points based on references. 
+Values files otherwise contain default values which might be overridden during deployment.
+
+Every values file is checked for references for microservices.
+If the analyzer finds a reference, a new `HelmTemplate` object is persisted in the database,
+and a new edge is defined between the microservices.
+
+## Frontend functionality
+
+The plugin is capable of providing the following functionality:
+
+- _Syntax highlight:_ the YAML nodes are colored according to the symbol type.
+- _InfoTree:_ some very basic metadata is available of the nodes (name, symbol type, value type).
+- _Microservice navigator:_ there is an accordion menu on the left side in which the detected
+ microservices are listed. The diagrams are available by right clicking the services.
+- _Diagrams:_
+ - Dependent services
+ - Config maps
+ - Secrets
+ - Resource usage
+

doc/usage.md

@@ -74,7 +74,7 @@ For full documentation see:
  (`-E SQL_ASCII` flag is recommended!)
 - [Start PostgreSQL database](https://www.postgresql.org/docs/12/app-postgres.html)
 
-## 1. Generate compilation database
+## 1/a. Generate compilation database
 If you want to parse a C++ project, you have to create a [compilation database
 file](http://clang.llvm.org/docs/JSONCompilationDatabase.html).
 
@@ -103,6 +103,19 @@ installation. The first command line argument is the output file name and the
 second argument is the build command which compiles your project. This can be a
 simple compiler invocation or starting a build system.
 
+## 1/b. Prepare Helm charts
+If you want to parse a Helm chart, and the chart files contain template files,
+you need to run the `helm template` command on the
+charts first, in order to bring the chart to a pure YAML format.
+
+1. Run `helm template` on the chart. Save the output of the command in a separate directory,
+ e.g. if you run the command from the root of the chart, 
+ `helm template <chart name> . --output-dir=./output`
+2. Copy all files and directories from the chart to the output directory (except the original
+ template files), in the original directory order: Chart.yaml, values.yaml files, files
+ directories, etc. This is needed because of the recursive directory traversal.
+3. Parse the project as described in below, with the output directory as input.
+
 ## 2. Parse the project
 For parsing a project with CodeCompass, the following command has to be emitted:
 

docker/dev/Dockerfile

@@ -25,6 +25,7 @@ RUN set -x && apt-get update -qq \
  libmagic-dev \
  libsqlite3-dev \
  libssl-dev \
+ libyaml-cpp-dev \
  llvm-10 clang-10 llvm-10-dev libclang-10-dev \
  npm \
  thrift-compiler libthrift-dev \

docker/runtime/Dockerfile

@@ -71,6 +71,7 @@ RUN set -x && apt-get update -qq && \
  libldap-2.4-2 \
  libmagic-dev \
  libthrift-dev \
+ ibyaml-cpp-dev \
  ctags \
  tini && \
  apt-get clean && \

plugins/helm/CMakeLists.txt

@@ -0,0 +1,7 @@
+find_package(yaml-cpp REQUIRED)
+
+add_subdirectory(model)
+add_subdirectory(parser)
+add_subdirectory(service)
+
+install_webplugin(webgui)

plugins/helm/model/CMakeLists.txt

@@ -0,0 +1,15 @@
+set(ODB_SOURCES
+ include/model/helmtemplate.h
+ include/model/microservice.h
+ include/model/microserviceedge.h
+ include/model/msresource.h
+ include/model/yamlastnode.h
+ include/model/yamlfile.h
+ include/model/yamlcontent.h)
+
+generate_odb_files("${ODB_SOURCES}")
+
+add_odb_library(yamlmodel ${ODB_CXX_SOURCES})
+add_dependencies(yamlmodel model)
+
+install_sql()

plugins/helm/model/include/model/helmtemplate.h

@@ -0,0 +1,63 @@
+#ifndef CC_MODEL_HELM_H
+#define CC_MODEL_HELM_H
+
+#include <string>
+
+#include <odb/core.hxx>
+#include <odb/lazy-ptr.hxx>
+#include <odb/nullable.hxx>
+
+#include <model/file.h>
+#include <model/microservice.h>
+
+namespace cc
+{
+namespace model
+{
+
+typedef uint64_t HelmTemplateId;
+
+#pragma db object
+struct HelmTemplate
+{
+ enum class DependencyType
+ {
+ SERVICE,
+ MOUNT,
+ CERTIFICATE,
+ RESOURCE,
+ OTHER
+ };
+
+ #pragma db id
+ HelmTemplateId id;
+
+ #pragma db not_null
+ FileId file;
+
+ std::string name;
+
+ #pragma db not_null
+ DependencyType dependencyType;
+
+ #pragma db not_null
+ std::string kind;
+
+ #pragma db not_null
+ MicroserviceId depends;
+
+ bool operator==(HelmTemplate& rhs);
+};
+
+inline std::uint64_t createIdentifier(const HelmTemplate& helm_)
+{
+ return util::fnvHash(
+ helm_.name +
+ helm_.kind +
+ std::to_string(helm_.depends) +
+ std::to_string(helm_.file));
+}
+}
+}
+
+#endif // CC_MODEL_HELM_H

plugins/helm/model/include/model/microservice.h

@@ -0,0 +1,47 @@
+#ifndef CC_MODEL_MICROSERVICE_H
+#define CC_MODEL_MICROSERVICE_H
+
+#include <odb/core.hxx>
+#include <odb/lazy-ptr.hxx>
+#include <odb/nullable.hxx>
+
+#include "model/file.h"
+
+#include "util/hash.h"
+
+namespace cc
+{
+namespace model
+{
+
+typedef std::uint64_t MicroserviceId;
+
+#pragma db object
+struct Microservice
+{
+ enum class ServiceType
+ {
+ INTERNAL,
+ EXTERNAL
+ };
+
+ #pragma db id
+ MicroserviceId serviceId;
+
+ #pragma db not_null
+ std::string name;
+
+ FileId file;
+
+ ServiceType type;
+};
+
+inline std::uint64_t createIdentifier(const Microservice& service_)
+{
+ return util::fnvHash(
+ service_.name);
+}
+}
+}
+
+#endif // CC_MODEL_MICROSERVICE_H