srclib-java requires:
- Oracle JDK 8 or OpenJDK 8
- Maven (in some cases we may execute
mvn generate-sources
) - Gradle (if project to be indexed doesn't come with Gradle wrapper)
tools.jar
that comes with the Oracle 8 JDK and OpenJDK 8 are both buggy
as of November 7th, 2014. com.sun.source.util.JavacTask.analyze()
throws a null pointer exception on some repositories. See
https://bugs.openjdk.java.net/browse/JDK-8062359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel
for a description of the problem and a workaround. Build instructions for
OpenJDK 8 are here: http://openjdk.java.net/projects/build-infra/guide.html.
You'll probably be able to use your system JDK without this patch on most projects; only follow those steps if you see NPEs in the javac API.
You need to have the following tools available to build Java toolchain:
- make
- JDK 8
- git
srclib-java can be built and registered with the following commands:
make
mkdir -p $SRCLIBPATH/sourcegraph.com/sourcegraph/
ln -s $PWD $SRCLIBPATH/sourcegraph.com/sourcegraph/srclib-java
or you may run srclib toolchain install java
which will clone the latest version to $SRCLIBPATH/sourcegraph.com/sourcegraph/srclib-java
and build it.
Run git submodule update --init
the first time to fetch the submodule test
cases in testdata/case
.
make test
- Test in program mode
make test-gen
- Generate new test data in program mode
srclib-java requires Java 8 to build and run. To allow it to run on systems that don't have Java 8 installed or activated by default, it can be bundled with a Java 8 JRE.
Currently this process requires tedious manual effort; fixing it is noted as a TODO.
The bundled JREs should be present in the bundled-jdk
directory. The
.bin/srclib-java
script checks for their existence and uses the
bundled JRE for the current platform, if it exists.
Running make bundled-jdk
downloads and unarchives the bundled JREs
from S3. If you are producing a new srclib-java release, then you MUST
make sure the srclib-java toolchain bundle contains these bundled JREs.
If you need to update the versions of the bundled JREs or add some for
other platforms, edit the .bin/srclib-java
script and the
Makefile
. These files contain hardcoded lists of platforms and JDK
versions that you'll need to update.
Please ensure you have specified file encoding in your .gradle or pom.xml build file, otherwise srclib-java will use platform encoding to compile files and compilation may fail when your files using charset different from the system's one.
We do not support annotations (yet), so if your code relies on annotation processor such as lombok there will be compilation errors and some parts of your code wouldn't be graphed.
If some parts of your code is generated using third party tool or plugin, we may fail to graph your code properly.
If your project is Maven-based, we are aware about existence of org.antlr:antlr4-maven-plugin and will try to generate java files from ANTLR grammar but please make sure that your directory structure matches java package names, i.e. don't put generated classes from foo.bar.baz package to src/generated, use src/generated/foo/bar/baz. Also please make sure that your ANTLR-based subproject compiles by running mvn -f SUBPROJECT-DIR/pom.xml generate-sources
command.
If your project is Android-based, we are trying to generate R.java, BuildConfig.java and AIDL-based java files using:
- with Maven:
mvn -f SUBPROJECT-DIR/pom.xml generate-sources
- with Gradle: run
:PROJECT:generateDebugSources
or:PROJECT:assembleDebug
before extracting meta information from Gradle build files.
If your project is Gradle-based, we'll run all not-standard (see Java Plugin for the list of tasks we considering "standard") tasks compile
task depends on before extracting meta information from Gradle build files. These tasks may produce generated source files, but they may also compile some parts of your code.
Sorry, but we don't support (yet) if part of your code is written in Groovy, Closure, or Scala or contains ajc files.
There may be compilation errors if your project contains empty java files.
If your file A.java contains two classes A and B, there might be compilation errors while trying to graph your source. It's a good practice to keep single top-level class per java file.
We expect that your project may be compiled out of the box using latest Gradle, Maven, and Java 8. If there are some build pre-requisites (such as some expected environment variables, hardcoded file locations, pre-installed libraries and/or tools, presence of some configuration files), we may fail to graph your source code properly. Please try to make sure that
- If your project is Gradle-based (but not Android-based) then
gradlew compileJava
works in clean environment - If your project is Gradle-based (and Android-based) then
gradlew assembleDebug
works in clean environment - If your project is Maven-based then
mvn compile
works in clean environment - If your project is Ant-based then
ant
works in clean environment
If few of your modules have class with the same name, we may be unable to resolve which one to use when graphing your source code.
When scanning for source units, we are processing ALL pom.xml and gradle build files in your repository. Some of them might be remnants of old build process and we might fail to process them if they were abandonned. Sometimes there may be both pom.xml and .gradle files in the same directory which may define the same group / artifact pair and we can't tell which one will be taken into account when graphing your source code. It's possible that we'll pick the wrong one.
Please try to support most recent version of Maven (it's currently 3.3.9). Apache Maven 3.x should be fine in most cases.
When graphing your Maven-based project, we are able to identify the following Maven plugins:
com.simpligility.maven.plugins::android-maven-plugin
- we'll runmvn -f pom.xml generate-sources
to generate source filesorg.apache.maven.plugins::maven-compiler-plugin
- we'll try to extract source code encoding and source code version from plugin's configurationcom.jayway.maven.plugins.android.generation2::android-maven-plugin
- we'll runmvn -f pom.xml generate-sources
to generate source filescom.jayway.maven.plugins.android.generation2::maven-android-plugin
- we'll runmvn -f pom.xml generate-sources
to generate source filesorg.codehaus.mojo::build-helper-maven-plugin
- we'll add extra source directories based on plugin's configurationorg.antlr::antlr4-maven-plugin
- we'll runmvn -f pom.xml generate-sources
to generate source files
If you think we should add a support for more plugins that may have impact on source code generation or compilation, please let us know.
We are trying to be as close as possible to the way Maven gathers dependencies, resolves artifacts, and compiles your code but there may be some differences.
- When Maven module B depends on module A, Maven will compile A and pass A's output to B classpath while we'll pass A's source directories to B sourcepath.
- We are using separate Maven repository directory
.m2
per repository, thus firstsrc scan
will try to download all artifacts from remote repositories which may take some time. If your project depends from pre-installed provided artifacts, we may be unable to fetch them. - We are trying to avoid running Maven goals or compiling project at the
src scan
phase (unless we know that project has generated source code for sure) - We suggest that your project should be compilable at any level - better resuts may be obtained if you can run
mvn -f PATH-TO/pom.xml compile
without any errors for every pom.xml file in your repository. - When scanning for source units, we'll try to process ALL pom.xml files in your repository so please try to keep only good ones there.
If you compiling your project using Gradle vX please add gradlew
Gradle wrapper as suggested here: https://docs.gradle.org/current/userguide/gradle_wrapper.html otherwise please expect that we'll try to use the latest Gradle version. Don't forget to add and check in gradle/wrapper
wrapper's directory (some don't).
- When we are extracting meta information from Gradle script files, we'll try to run first all not-standard tasks
compile
task depends on (see Java Plugin for the list of tasks we considering "standard"). Such tasks may generate source code, repack jar files using jarjar or do something else in order to prepare your project for compilation. Unfortunately we can't identify what particular task does thus we may run some tasks that weren't needed. For Android-based projects we'll run eithergenerateDebugSources
orassembleDebug
(which one we found first) to generate java files so please make sure that these tasks may be run out of the box and won't fail, otherwise we'll be unable to properly extract project information. - When scanning for java files to graph, we taking into account only two sourcesets -
main
andtest
. - When scanning for java files to graph, we taking into account only the following configurations -
compile
,testCompile
,provided
,compileOnly
- If your project contains
gradlew
and/orgradlew.bat
please- make sure that you provided gradle launch script for both Unix and Windows
- make sure that you ship gradle-wrapper.jar with your project, otherwise we won't be able to run
gradlew
- We are using separate gradle user home per repository -
.gradle-srclib
, so first run ofsrc scan
may take some time while all artifacts being downloaded. - Please make sure that you can run without errors
gradlew
in each project's directory. If you can't - so do we.
Please see instructions here
Please see instructions here
srclib-java
provides basic support for projects supposed to be built with
Apache Ant. Currently we support the following Ant tasks and types
- property
- basename
- condition
- dirname
- import
- loadproperties
- xmlproperty
- and
- antversion
- contains
- equals
- filesmatch
- hasfreespace
- hasmethod
- isfailure
- isfalse
- isfileselected
- islastmodified
- isreference
- isset
- issigned
- istrue
- matches
- not
- or
- os
- parsersupports
- resourcecontains
- resourceexists
- resourcesmatch
- typefound
- xor
- available
- uptodate
- fileset
- path
- get
- mkdir
- javacc
- macrodef
srclib-java
extracts source files, classpath, and bootstrap classpath from
javac
tasks (which also may be enclosed into macrodef
).
srclib-java
tries to resolve external dependencies by using "search by SHA1"
feature of Maven Central
srclib-java
uses bundled JavaCC 6.1.2.
srclib-java
does not support Apache Ivysrclib-java
does not support custom Ant tasks- there will be compilation errors when project contains code based on Ant classes because Ant automatically includes Ant jar files into classpath while we don't.
- some Ant tasks may not be executed and some may fail because of requirements not met. Such requirements may include missing properties, missing directories or missing files which may appear as a result of custom Ant tasks or actions we don't support yet.
- there may be errors when repository containing Ant-based project provides
build properties file in the form of template (for example,
build.properties.sample
). We suggest to providebuild.properties
with default values and loadlocal.build.properies
if there is such a file. - all source files and classpath entries of all
javac
elements encountered inbuild.xml
file would be merged into the single source unit. Onebuild.xml
=== one source unit. - there may be compilation issues when repository contains few sub-projects that may depend on each other. We won't be able to resolve dependencies between sub-projects.