Problems with detecting updates of Golang dependencies in private repositories

[michaellihs]

Package manage/ecosystem
go_modules

Manifest contents prior to update

module go.my-company.com/our-service

go 1.13

require (
	...
	go.my-company.com/private-dependency.git v1.0.2
	...
)

Updated dependency
from go.my-company.com/private-dependency.git v1.0.2 to go.my-company.com/private-dependency.git v1.1.0

What you expected to see, versus what you actually saw

I'd expect a pull request for go.my-company.com/private-dependency.git v1.1.0

What I actually get is an error message like this

/usr/local/bundle/gems/dependabot-go_modules-0.113.28/lib/dependabot/go_modules/file_parser.rb:145:in `handle_parser_error': go: go.my-company.com/private-dependency.git@v1.0.2: reading go.my-company.com/private-dependency.git/go.mod at revision v1.0.2: unknown revision v1.0.2 (Dependabot::DependencyFileNotResolvable)
	from /usr/local/bundle/gems/dependabot-go_modules-0.113.28/lib/dependabot/go_modules/file_parser.rb:90:in `block (2 levels) in module_info'
	from /usr/local/bundle/gems/dependabot-common-0.113.28/lib/dependabot/shared_helpers.rb:143:in `with_git_configured'
	from /usr/local/bundle/gems/dependabot-go_modules-0.113.28/lib/dependabot/go_modules/file_parser.rb:71:in `block in module_info'
	from /usr/local/bundle/gems/dependabot-common-0.113.28/lib/dependabot/shared_helpers.rb:37:in `block (2 levels) in in_a_temporary_directory'
	from /usr/local/bundle/gems/dependabot-common-0.113.28/lib/dependabot/shared_helpers.rb:37:in `chdir'
	from /usr/local/bundle/gems/dependabot-common-0.113.28/lib/dependabot/shared_helpers.rb:37:in `block in in_a_temporary_directory'
	from /usr/local/lib/ruby/2.6.0/tmpdir.rb:93:in `mktmpdir'
	from /usr/local/bundle/gems/dependabot-common-0.113.28/lib/dependabot/shared_helpers.rb:34:in `in_a_temporary_directory'
	from /usr/local/bundle/gems/dependabot-go_modules-0.113.28/lib/dependabot/go_modules/file_parser.rb:70:in `module_info'
	from /usr/local/bundle/gems/dependabot-go_modules-0.113.28/lib/dependabot/go_modules/file_parser.rb:20:in `parse'
	from ./update.rb:95:in `<main>'

Images of the diff or a link to the PR, issue or logs
not applicable

Background

We use

git config --global url."https://OAuth:${SYSTEM_ACCESSTOKEN}@dev.visualstudio.com/org/project/_git/".insteadOf "git://go.my-company.com/"

to redirect go mod dependency requests to git://go.my-company.com, which works for all dependency-related operations with go mod but seems to raise issues with the dependabot-core/go_modules/helpers/install-dir/go_modules/bin/helper utility.

If I run

echo '{"function":"getUpdatedVersion","args":{"dependency":{"name":"go.my-company.com/private-dependency.git","version":"v1.0.2","indirect":false}}}' | /Users/mimi/workspace/github/dependabot-core/go_modules/helpers/install-dir/go_modules/bin/helper

on my laptop, things work as expected and I see

{"result":"v1.1.0"}

If I run it within dependabot-core, I either get the error mentioned above, or I get

/Users/mimi/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/dependabot-common-0.113.28/lib/dependabot/shared_helpers.rb:112:in `run_helper_subprocess': git ls-remote -q git://go.my-company.com/private-dependency in /Users/mimi/workspace/go/pkg/mod/cache/vcs/104cd3930c025d3880a366f9222d12c72e76daf950d49cd5534b9fbe2f3bf3c1: exit status 128: (Dependabot::SharedHelpers::HelperSubprocessFailed)
	fatal: unable to look up go.my-company.com (port 9418) (nodename nor servname provided, or not known)

so it seems like the

[url "https://OAuth:TOKEN@dev.visualstudio.com/org/project/_git/"]
        insteadOf = git://go.my-company.com/

in my .gitconfig is not used when running dependabot.

A git ls-remote -q git://go.my-company.com/private-dependency shows the refs as expected.

Thanks for your assistance!

[mikeparker]

I'd be interested to know if private repositories are supported in gomod dependabot.

[michaellihs]

I'd be interested to know if private repositories are supported in gomod dependabot.

I guess, this depends on the git configuration on the host that runs dependabot. If ssh keys / basic auth credentials are in place, I'd assume, it works. On my local machine, dependabot can access the private repo with the configuration shown above.

[alexruf]

I guess, this depends on the git configuration on the host that runs dependabot. If ssh keys / basic auth credentials are in place, I'd assume, it works. On my local machine, dependabot can access the private repo with the configuration shown above.

That's also how I think it works. At least when I update my Git config so the URLs point to my private repos, Go Modules is able to resolve them correctly.

When running Dependabot in a container and not directly on the host, it should also work to overwrite certain Git config parameters by setting the environment variable GIT_CONFIG_PARAMETERS.

[Okeanos]

A solution for this would be really helpful.

Neither using GIT_CONFIG_PARAMETERS='url."https://OAuth:${SYSTEM_ACCESSTOKEN}@dev.visualstudio.com/org/project/_git/".insteadOf="git://go.my-company.com/"'to actually try to force resolution via the correct URL nor using dependency-ignore settings actually works.

The former (Git config) apparently doesn't work because the resolution chain goes through the following steps:

1. file_parser -> git_source
2. path_converter -> git_url_for_path
3. importresolver -> VCSRemoteForImport

In the VCSRemoteForImport it'll finally just use the http(s) url based on the go.mod package name – no way here to inject a rewrite because the underlying go process (which would use Git and hence the Git config) isn't used.

For the latter (dependency-ignore) it isn't even possible to prevent private (rewritten) packages from being parsed and evaluated in the first place to allow at least public dependency to be checked. See #1959 for additional details.

There is also no recovery option as far as I can tell, i.e. let Dependabot fail for a private/rewritten go module but just log it and continue with the remaining dependencies. Dependabot will just fail and not recover at all. This makes Dependabot fully unusable in some use cases.