Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to pull and index dependency data from the Maven M2 directory to remove the need for building the version of source you want to analyze #10

Open
Lambeaux opened this issue Jul 7, 2020 · 1 comment
Assignees
Labels
capabilities/data-sources Concerns input formats and normalization priority/high Used to denote an issue that should stand out from among the rest. type/feature New feature or request that's satisfying some kind of functional requirement
Milestone

Comments

@Lambeaux
Copy link
Contributor

Lambeaux commented Jul 7, 2020

Issue summary & value proposition

One key piece of insight is how dependency graphs change across released versions or as part of a pull request. By moving indexing to the M2 instead of the cloned repository:

  • Curious engineers can keep iterating on their current task and do not need to context switch, redundantly clone, or redundantly build a version of source they want to analyze.
  • The feedback loop is much faster and takes advantage of existing artifacts.
  • The feedback loop can move to any continuous integration environment, which typically works with and passes an M2 across build stages, opening the possibility to catch problematic dependencies early.
  • Indexing could be targeted at a remote M2 location, not necessarily a local one; quality controllers such as architects or program managers can more readily enforce the invariants that the team has agreed upon, or find out where they are troublesome and adjust their strategies.

Proposed solution

This is the next crucial feature that will make osg-eyes more useful to more engineers. Several things need to happen:

  • Decouple from source code directory assumptions that are part of the (index-repo ...) command.
  • Rename concept of "Locale" to "Locator"; make the "m2" locator.
  • Modularize the information model across poms, manifests, blueprints, and features.
  • Introduce the concept of a "Connector" for gluing all the pieces together and keeping the collapsible hierarchy intact for the set of core transforms that provide the usefulness and insight.

While still a valid filter syntax, the canonical data model used to represent nodes is changing from just <repo>/<bundle-symbolic-name> to full maven coordinates.

  • While navigating a repo, the manifest was the source of truth.
  • While navigating a repo, the root directory was specified to start the process.
  • While navigating a repo, the directory structure was a good enough search target.
  • Navigating an M2 will require the pom (or whatever build config is used) to be the source of truth.
  • Navigating an M2 will require a root pom / artifact to be specified to start the process.
  • Navigating an M2 will require the pom to inform the traversal on where to look next.
  • The concept of "Connectors" will be functions that can discover additional data with respect to the pom's location.
  • This should scale to most build reactor models assuming appropriate specification of the project.

(Optional) Standalone but related improvements worth doing if repo indexing is here to stay:

  • Pull basic metadata from poms so that the updated data models and code can still be used while indexing with the repo strategy.
  • (Related) Pull basic metadata from poms so repo locators can be labeled and versioned within the index.

(Optional) (Related) Support index metadata

  • Label and version repo locators with the index so that the context of the operation is preserved.
@Lambeaux Lambeaux added type/feature New feature or request that's satisfying some kind of functional requirement priority/high Used to denote an issue that should stand out from among the rest. labels Jul 7, 2020
@Lambeaux Lambeaux added this to the 0.2 milestone Dec 3, 2020
@Lambeaux Lambeaux self-assigned this Dec 3, 2020
@Lambeaux
Copy link
Contributor Author

Lambeaux commented Dec 3, 2020

Bumping this in priority. Instead of making the full UI jump, will consider tackling that later and will wrap up the maven functionality to be used on the same JVM just like how the CLI works today. That should be sufficient for reporting on DDF's dependencies and backlogging next steps.

Indexing example proved successful on the mvn-indexing branch. Going to move it to a new branch, nuke the UI commits, and rework things so Clojure queries my custom indexer instead of the filesystem directly.

No need to create connectors or manually traverse an m2 as of yet. Index Creators will be the solution for now.

@Lambeaux Lambeaux added the capabilities/data-sources Concerns input formats and normalization label Dec 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
capabilities/data-sources Concerns input formats and normalization priority/high Used to denote an issue that should stand out from among the rest. type/feature New feature or request that's satisfying some kind of functional requirement
Projects
None yet
Development

No branches or pull requests

1 participant