Tree Sitter Multi Codeview Generator aims to generate combined multi-code view graphs that can be used with various types of machine learning models (sequence models, graph neural networks, etc).
comex is a rebuild of Tree Sitter Multi Codeview Generator for easier invocation as a Python package. This rebuild also includes a cli interface. Currently, comex generates codeviews for Java and C#, for both method-level and file-level code snippets. comex can be used to generate over comex is designed to be easily extendable to various programming languages. This is primarliy because we use tree-sitter for parsing, a highly efficient incremental parser that supports over
If you wish to learn more about the approach taken, here are some conference talks and publications:
- ASE 2023 demonstration: COMEX: A Tool for Generating Customized Source Code Representations
- ICSE 2023 tutorial: The Landscape of Source Code Representation Learning in AI-Driven Software Engineering Tasks
Comex played a critical role in developing CodeSAM, a novel framework for infusing multiple code-views into transformer-based models. CodeSAM builds on tools like comex to create structured code representations (e.g., AST, CFG, DFG), enabling fine-tuning of language models like CodeBERT. Experimental results show that by using this technique, we can improve downstream performance when compared to SLMs like GraphCodeBERT and CodeBERT on ML for SE tasks by utilizing individual code-views or a combination of code-views during fine-tuning.
You can find more details about CodeSAM in the following pre-print:
If you use Comex in your research, please cite our work by using the following BibTeX entry:
@inproceedings{das2023comex,
title={COMEX: A Tool for Generating Customized Source Code Representations},
author={Das, Debeshee and Mathews, Noble Saji and Mathai, Alex and Tamilselvam, Srikanth and Sedamaki, Kranthi and Chimalakonda, Sridhar and Kumar, Atul},
booktitle={2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)},
pages={2054--2057},
year={2023},
organization={IEEE}
}
comex is published on the Python Registry and can be easily installed via pip:
pip install comexNote: You would need to install GraphViz(dot) so that the graph visualizations are generated
To setup comex for development using the source code in your python environment:
pip install -r requirements-dev.txtThis performs an editable install, meaning that comex would be available throughout your environment (particularly relevant if you use conda or something of the sort). This means now you can interact and import from comex just like any other package while remaining standalone but also reflecting any code side updates without any other manual steps
This is the recommended way to get started with comex as it is the most user friendly
The attributes and options supported by the CLI are well documented and can be viewed by running:
comex --helpFor example, to generate a combined CFG and DFG graph for a java file, you can run:
comex --lang "java" --code-file ./test.java --graphs "cfg,dfg"The comex package can be used by importing required drivers as follows:
from comex.codeviews.combined_graph.combined_driver import CombinedDriver
CombinedDriver(
src_language=lang,
src_code=code,
output_file="output.json",
graph_format=output,
codeviews=codeviews
)In most cases the required combination can be obtained via the combined_driver module as shown above.
src_language: denotes one of the supported languaged hence currently "java" or "cs"
src_code: denotes the source code to be parsed
output_file: denotes the output file to which the generated graph is written
graph_format: denotes the format of the output graph. Currently supported formats are "dot" and "json". To generate both pass "all"
codeviews: refers to the configuration passed for each codeview
While comex provides method-level and file-level support for both Java and C#, it's important to note the following limitations and known issues:
-
No Inter-file Analysis Support: The tool currently does not support codeviews that involve interactions between multiple Java files. It is designed to generate codeviews for individual Java files only.
-
Syntax Errors in Code: Despite supporting non-compileable code, to ensure accurate codeviews, the input Java code must be free of syntax errors. Code with syntax errors may not be correctly parsed and displayed in the generated codeviews.
-
Limited Support for Function Call Arguments: The tool does not provide proper support for when a function call is passed as an argument to another function call in Java code. The resulting codeview might not accurately represent the intended behavior in such cases.
In addition to the limitations mentioned for Java, the tool has the following limitations specific to C#:
-
No Support for Lambda Functions and Arrow Expressions: The tool does not support codeviews involving lambda functions and arrow expressions in C#. The generated codeviews may not accurately represent these language features.
-
No Support for Compiler Directives: Compiler directives, such as pragma directives, are not supported by the tool. Code involving such directives may not be properly displayed in the generated codeviews.
-
Incomplete Operator Declaration Support: The tool may have limited support for operator declarations in C#. Certain constraints and edge cases related to operator overloading might not be fully captured in the generated codeviews.
-
Limited Support for Inheritance and Abstraction: The tool's support for inheritance and abstraction in C# is not fully comprehensive. Codeviews involving complex inheritance hierarchies or advanced abstraction patterns may not be accurately represented.
Please note that while we continuously work to improve the tool and address these limitations, the current implementation may not be perfect. We appreciate your understanding and encourage you to provide feedback and report any issues you encounter, as this helps us enhance the tool's capabilities.
Combined simple AST+CFG+DFG for a simple Java program that finds the maximum among 2 numbers:
Below we present more examples of input code snippets and generated codeviews for both Java and C#.
CLI Command:
comex --lang "java" --code-file sample/example.java --graphs "cfg,dfg"Java Code Snippet:
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
public class Main {
// static String INPUT = "5\n3 2 2 4 1\n1 2 2 2 1";
static String INPUT = "";
public static void main(String[] args) {
InputStream is = INPUT.isEmpty() ? System.in : new ByteArrayInputStream(INPUT.getBytes());
Scanner scanner = new Scanner(is);
final int n = scanner.nextInt();
List<Position> positionList = new ArrayList<>(n);
for (int i = 0; i < n; i++) {
positionList.add(
new Position(
scanner.nextInt(),
scanner.nextInt(),
scanner.nextInt()
)
);
}
System.out.println(solve(positionList) ? "Yes" : "No");
}
static class Position {
int t;
int x;
int y;
public Position(int t, int x, int y) {
this.t = t;
this.x = x;
this.y = y;
}
}
static boolean solve(List<Position> positionList) {
Position currentPosition = new Position(0, 0, 0);
for (int i = 0; i < positionList.size(); i++) {
Position nextPosition = positionList.get(i);
if (!possibleMove(currentPosition.t, nextPosition.t, currentPosition.x, nextPosition.x, currentPosition.y, nextPosition.y)) {
return false;
}
currentPosition = nextPosition;
}
return true;
}
static boolean possibleMove(int t1, int t2, int x1, int x2, int y1, int y2) {
int tDiff = t2 - t1;
int absX = Math.abs(x1 - x2);
int absY = Math.abs(y1 - y2);
if (absX + absY <= tDiff) {
if (tDiff % 2 == (absX + absY) % 2) {
return true;
}
}
return false;
}
}Generated Codeview:
CLI Command:
comex --lang "cs" --code-file sample/example.cs --graphs "cfg,dfg"C# Code Snippet:
public class DFG_A2 {
public void main(string[] args) {
BufferedReader br = new BufferedReader(new InputStreamReader(System.in)); // {}
String str = br.attribute1; // {3}
str = br.attribute2.method1(); // {3}
br.attribute1 = br.attribute2; // {3,5}
br.method2(br.attribute1, br.attribute2); // {3,5,6}
BufferedReader br2 = br; // {5,6,7}
br.method3(); // {5,6,7}
int j = br2.attribute1.method2(3,4); // {8}
}
}Generated Codeview:
More examples and results can be found in the tests/data directory
The code is structured in the following way:
- For each code-view, first the source code is parsed using the tree-sitter parser and then the various code-views are generated. In the tree_parser directory, the Parser and ParserDriver is implemented with various funcitonalities commonly required by all code-views. Language-specific features are further developed in the language-specific parsers also placed in this directory.
- The codeviews directory contains the core logic for the various codeviews. Each codeview has a driver class and a codeview class, which is further inherited and extended by language in case of code-views that require language-specific implementation.
- The cli.py file is the CLI implementation. The drivers can also be directly imported and used like a python package. It is responsible for parsing the source code and generating the codeviews.
The repo is setup to automatically perform CI tests on making pulls to main and development branches. To test locally:
Run specific test
- Say you wish to run
test_cfgfunction - Drop the
'[...]'part to run all tests in a file- formatted as [extension-filename]
- no-cov prevents coverage report from being printed
pytest -k 'test_cfg[cs-test7]' --no-covRun all tests and get coverage report
pytestAnalyze the deviation report given by deepdiff by using the verbose output.
This will help quickly figure out difference from the gold file
pytest -k 'test_cfg[cs-test7]' --no-cov -vvMake sure to bump the version in setup.cfg.
Then run the following commands:
rm -rf build dist
python setup.py sdist bdist_wheelThen upload it to PyPI using twine (pip install twine if not installed):
twine upload dist/*This tool was developed for research purposes as a part of the OSCP Project. Efficient representation of source code is essential for various software engineering tasks using AI pipelines such as code translation, code search and code clone detection. Code Representation aims at extracting the both syntactic and semantic features of source code and representing them by a vector which can be readily used for the downstream tasks. Multiple works exist that attempt to encode the code as sequential data to easily leverage state of art NN models like transformers. But it leads to a loss of information. Graphs are a natural representation for the code but very few works(MVG-AAAI’22) have tried to represent the different code features obtained from different code views like Program Dependency Graph, Data Flow Graph etc. as a multi-view graph. In this work, we want to explore more code views and its relevance to different code tasks as well as leverage transformers model for the multi-code view graphs. We believe such a work will help to
- Establish influence of specific code views for common tasks
- Demonstrate how graphs can combined with transformers
- Create re-usable models
This tool is based on the ongoing joint research effort between IBM and Risha Lab at IIT Tirupati to explore the effects of different code representations on code based tasks involving: