Skip to content
This repository has been archived by the owner on May 29, 2018. It is now read-only.

Defer slow parts of scanning #27

Open
sqs opened this issue Dec 20, 2014 · 0 comments
Open

Defer slow parts of scanning #27

sqs opened this issue Dec 20, 2014 · 0 comments

Comments

@sqs
Copy link
Member

sqs commented Dec 20, 2014

The scan phase should be fast and seldom fail because it is not parallelized and does not have separate logs for each package. It is also possible for Java source units detected by the scanner to be ignored (e.g., using SkipDirs in the Srcfile); in that case it is undesirable to incur the wait to download all of an ignored module's deps.

(In this case, "fast" means "takes hundreds of milliseconds and doesn't hit the network.")

Also, I can't find anywhere that I wrote this down, but I initially had in mind that scanning would always be fast and some other design decisions in srclib assume that it is fast. For example, to make srclib usable for realtime editing, it needs to be able to run the scan phase whenever the project's configuration changes, which could be often (e.g., a file is added or deleted). If scan is slow, then we'd either have to wait a long time to re-scan each time the project configuration changes, or we would have to move more scan-related logic into the grapher and other tools (e.g., the grapher might assume responsibility for knowing that src/**/*.java are the Java files). The former is obviously not good and the latter means that the grapher now has to be entangled with the build system (a big goal of the separation was to avoid having to write a Maven grapher, a Gradle grapher, etc.).

Currently the srclib-java toolchain fetches all of the Maven/Gradle deps in the scan phase. We should defer that work to the graph and depresolve phases. (There needs to be some way to share work between those 2, since the graph phase almost always needs to resolve deps.)

@sqs sqs self-assigned this Dec 20, 2014
@sqs sqs removed their assignment May 12, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant