v0.4.0 - Smooth out configuration parsing and add retries

This small release focused more on the internal side of the config management. Now the storage implementation itself provides it's own way of setting up and it will then be picked up in the crawler config as a constant variant. Based on this constant the individual configuration setups are called inside of the main function.

This adds a new key to the crawler configuration, simply called Storage or setup with the $STORAGE_BACKEND environment variable.

There is one known issue with the current setup around displaying help messages from the configuration of the storage itself, I'll come back to that at a later point.

Also I did spend some time smoothing out the project streaming in my GitLab client, it will now check if it is properly authenticated before starting to stream anything. Retries are added to this check as well and will serve as the foundation for a future implementation of retries in the entire GitLab client.

Changelog

• a9aa80b Streamline configuration for storage and it's parsing
• f2a9e86 Add an authentication check before trying to stream projects

v0.3.0 - Split storage implementation

This release prepares the codebase for a few things, one multiple storage backends are now a possibility through the new Storage interface that's available. Also this is some groundwork for the upcoming testsuite for the codebase.

The Storage interface expects two functions:

CreateProjectNode(projectPath string) error
CreateIncludeEdge(include IncludeEdge) error

And the IncludeEdge type is just the metadata pulled out of GitLab packed into a type:

type IncludeEdge struct {
	SourceProject string
	TargetProject string
	Ref           string
	Files         []string
}

Configuration of the storage is still handled by the overall config package, that whole mechanism is subject to change as I'm not very happy with it - The interface for configuration of the final binary should be fairly stable I'm only planning to switch up some internal plumbing.

Changelog

c261df2 Add storage interface and neo4j implementation
b2058f4 Prevent empty linkheaders entering the parsing functions
d358828 Fix updateProjectInGraph to attempt to write includes after single failure

v0.2.0

Changelog

e79b86b Rename GitlabRPS to GitlabMaxRPS
3a170ab fix rate limiter
86560dc add gitlab rate limiter

v0.1.1

Changelog

298a36d Move init func to main

v0.1.0

Changelog

8d69fad Initial Commit