PR's are getting opened that revert a bunch of deps #4536

jeffwidman · 2021-12-15T20:45:30Z

Seeing something seriously wrong with Dependabot PR's that just got opened in some private repos.

This is for Golang code.

It's trying to bump some basic open source libs such as:

Bump github.com/spf13/viper from 1.10.0 to 1.10.1
Bump google.golang.org/grpc from 1.42.0 to 1.43.0

However, the go.mod file changes are modifying a bunch of libraries that are completely unrelated... even worse, they're reverting those versions to really old versions.

The job ids:

244841423
245085086

The text was updated successfully, but these errors were encountered:

jeffwidman · 2021-12-15T20:49:22Z

Right before this, @dtomcej also spotted an error about out of disk space on job 245009475

Maybe related to the issue mentioned in #4346

Maybe that is causing this? IDK, that seems very weird that this would partially work and not completely blow up...

dtomcej · 2021-12-15T20:54:50Z

In addition, we are seeing attempts to upgrade to package versions that are higher than the latest, and therefore do not even exist.

jurre · 2021-12-15T21:04:25Z

Thanks for reporting! We're investigating what's happening

mctofu · 2021-12-15T23:07:51Z

@jeffwidman are you able to reproduce this with the dry run script on the repo?

mx-psi · 2021-12-20T16:47:48Z

This may be related to #4322, which also had some random dependencies being changed and where we also saw "out of memory errors" (we haven't experienced this again, but we are seeing "out of memory errors" again in the last days)

dtomcej · 2021-12-20T18:16:35Z

@mx-psi it is very possible as the repos have large numbers of related dependencies

jurre · 2021-12-21T17:17:02Z

@dtomcej are you able to reproduce the problem on a public repo? I'm trying to find a way to figure out what's going on

dtomcej · 2021-12-21T22:22:23Z

@jurre I am currently running the dry_run script against one of the affected repos. Will post the results here when they are ready.

dtomcej · 2021-12-21T22:42:44Z

Hello @jurre I have compiled some results that hopefully may shed some light on the issue. I ran the dry_run script on my workstation, without any CPU/RAM/Disk limits, and returned the same results! I take this to mean that it is not an issue of resources, as there were no limits in my testing.

Sanitized results are as follows:

[dependabot-core-dev] ~/dependabot-core $ bin/dry-run.rb go_modules private/repository
=> cloning into /home/dependabot/dependabot-core/tmp/private/repository
=> parsing dependency files
=> updating 27 dependencies: (...)

(...)

=== github.com/private/I (0.53.0)
 => checking for updates 8/27
 => latest available version is 0.54.0
 => latest allowed version is 0.54.0
 => requirements to unlock: own
 => requirements update strategy: 
 => updating github.com/private/I from 0.53.0 to 0.54.0
 => handled error whilst updating github.com/private/I: git_dependencies_not_reachable {:"dependency-urls"=>["github.com/private/I v0.54.0\ngo get"]}

=== github.com/private/M (0.38.0)
 => checking for updates 9/27
 => latest available version is 0.38.0
 => latest allowed version is 0.38.0
    (no update needed as it's already up-to-date)

(...)

=== github.com/private/P (0.29.0)
 => checking for updates 10/27
 => latest available version is 0.30.0
 => latest allowed version is 0.30.0
 => requirements to unlock: own
 => requirements update strategy: 
 => updating github.com/private/P from 0.29.0 to 0.30.0

    ± go.mod
    ~~~
    13,15c13,15
    < 	github.com/private/I v0.53.0
    < 	github.com/private/M v0.38.0
    < 	github.com/private/P v0.29.0
    ---
    > 	github.com/private/I v0.9.0
    > 	github.com/private/M v0.30.0
    > 	github.com/private/P v0.30.0
    119c119
    ---

The tool recognizes that module private/M is up to date at v0.38.0 then proceeds to directly downgrade it to v0.30 a few lines later. The same thing happens for private/I, which the tool recognizes needs an upgrade from v0.53.0 to v0.54.0, and then directly proceeds to downgrade it to v0.9.0 a few lines later.

mctofu · 2021-12-21T23:48:50Z

@dtomcej: Thanks! Something odd is definitely going on but it's great that you can reproduce it. Since we don't have access to this repo would you be able to run these steps that Dependabot would perform in the dev container and see if any of them result in the unexpected changes? If so it would be worth repeating them outside the dev container as well.

git clone private/repository & cd into it
GOPRIVATE=* go get -d github.com/private/P@v0.30.0
GOPRIVATE=* go get -d
GOPRIVATE=* go mod tidy
GOPRIVATE=* go mod vendor (if vendoring)

Another way to inspect what Dependabot is doing here would be to used the debugger. You could insert a line containing byebug to

dependabot-core/go_modules/lib/dependabot/go_modules/file_updater/go_mod_updater.rb

Line 86 in b78e5d7

original_go_mod = File.read("go.mod")

and then run the dry-run script as bin/dry-run.rb go_modules private/repository --dep github.com/private/P. That would allow you to step through what's going on.

dtomcej · 2021-12-22T17:33:34Z

Hello @mctofu, those goprivate commands worked without any issues, and they correctly updated the required module, and nothing else.

I tried with private/P and private/I, both of which needed updating, and both of them updated independently correctly, only touching the required deps.

dtomcej · 2021-12-22T17:34:30Z

Could you give me a more detailed prcedure to add byebug to the updater file? I would like to try that route too.

Not sure what would need to be added.

mctofu · 2021-12-23T01:04:58Z

@dtomcej: I pushed up a branch that adds some debugging statements during the update process that will hopefully guide us to where it's going wrong: https://github.com/dependabot/dependabot-core/compare/mctofu/debug-go-mod-update. If you switch to that branch and redo the dry run we should see some additional output like:

output with debug

[dependabot-core-dev] ~/dependabot-core $ bin/dry-run.rb go_modules mctofu/pushgate --dep github.com/aws/aws-lambda-go
=> cloning into /home/dependabot/dependabot-core/tmp/mctofu/pushgate
=> parsing dependency files
=> updating 1 dependencies: github.com/aws/aws-lambda-go

=== github.com/aws/aws-lambda-go (1.20.0)
 => checking for updates 1/1
 => latest available version is 1.27.1
 => latest allowed version is 1.27.1
 => requirements to unlock: own
 => requirements update strategy: 
 => updating github.com/aws/aws-lambda-go from 1.20.0 to 1.27.1
** original go.mod:
module github.com/mctofu/pushgate

go 1.14

require (
        github.com/aws/aws-lambda-go v1.20.0
        github.com/gregdel/pushover v0.0.0-20200416074932-c8ad547caed4
        github.com/stretchr/testify v1.6.1
)
** substitute go.mod:
module github.com/mctofu/pushgate

go 1.14

require (
        github.com/aws/aws-lambda-go v1.20.0
        github.com/gregdel/pushover v0.0.0-20200416074932-c8ad547caed4
        github.com/stretchr/testify v1.6.1
)
** package: true
** run go get -d github.com/aws/aws-lambda-go@v1.27.1
** updated dep go.mod:
module github.com/mctofu/pushgate

go 1.14

require (
        github.com/aws/aws-lambda-go v1.27.1
        github.com/gregdel/pushover v0.0.0-20200416074932-c8ad547caed4
        github.com/stretchr/testify v1.6.1
)
** package: true
** run go get -d
** updated go.mod:
module github.com/mctofu/pushgate

go 1.14

require (
        github.com/aws/aws-lambda-go v1.27.1
        github.com/gregdel/pushover v0.0.0-20200416074932-c8ad547caed4
        github.com/stretchr/testify v1.6.1
)
** run go mod tidy -e
** tidy go.mod:
module github.com/mctofu/pushgate

go 1.14

require (
        github.com/aws/aws-lambda-go v1.27.1
        github.com/gregdel/pushover v0.0.0-20200416074932-c8ad547caed4
        github.com/stretchr/testify v1.6.1
)
** vendor go.mod:
module github.com/mctofu/pushgate

go 1.14

require (
        github.com/aws/aws-lambda-go v1.27.1
        github.com/gregdel/pushover v0.0.0-20200416074932-c8ad547caed4
        github.com/stretchr/testify v1.6.1
)
** version adjusted go.mod:
module github.com/mctofu/pushgate

go 1.14

require (
        github.com/aws/aws-lambda-go v1.27.1
        github.com/gregdel/pushover v0.0.0-20200416074932-c8ad547caed4
        github.com/stretchr/testify v1.6.1
)

    ± go.mod
    ~~~
    6c6
    <   github.com/aws/aws-lambda-go v1.20.0
    ---
    >   github.com/aws/aws-lambda-go v1.27.1
    ~~~

    ± go.sum
    ~~~
    2,3c2,3
    < github.com/aws/aws-lambda-go v1.20.0 h1:ZSweJx/Hy9BoIDXKBEh16vbHH0t0dehnF8MKpMiOWc0=
    < github.com/aws/aws-lambda-go v1.20.0/go.mod h1:jJmlefzPfGnckuHdXX7/80O3BvUUi12XOkbv4w9SGLU=
    ---
    > github.com/aws/aws-lambda-go v1.27.1 h1:MAH6hbrsktcSr/gGQKLvHeJPeoOoaspJqh+O4g05bpA=
    > github.com/aws/aws-lambda-go v1.27.1/go.mod h1:jJmlefzPfGnckuHdXX7/80O3BvUUi12XOkbv4w9SGLU=
    6d5
    < github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
    16d14
    < github.com/stretchr/objx v0.1.0 h1:4G4v2dO3VZwixGIRoQ5Lfboy6nUhCyYzaqnIAPPhYs4=
    23d20
    < gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
    25d21
    < gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
    ~~~

That will be easier than using byebug but if you want to give that a try too you'll add the byebug statement wherever you'd like to start debugging.

        def update_files # rubocop:disable Metrics/AbcSize, Metrics/PerceivedComplexity
          in_repo_path do
            # Map paths in local replace directives to path hashes
            original_go_mod = File.read("go.mod")
            byebug
            puts "** original go.mod:\n#{original_go_mod}"
            original_manifest = parse_manifest
            original_go_sum = File.read("go.sum") if File.exist?("go.sum")

Then when you run the dry-run script you'll get an interactive terminal:

[dependabot-core-dev] ~/dependabot-core $ bin/dry-run.rb go_modules mctofu/pushgate --dep github.com/aws/aws-lambda-go
=> cloning into /home/dependabot/dependabot-core/tmp/mctofu/pushgate
=> parsing dependency files
=> updating 1 dependencies: github.com/aws/aws-lambda-go

=== github.com/aws/aws-lambda-go (1.20.0)
 => checking for updates 1/1
 => latest available version is 1.27.1
 => latest allowed version is 1.27.1
 => requirements to unlock: own
 => requirements update strategy: 
 => updating github.com/aws/aws-lambda-go from 1.20.0 to 1.27.1

[83, 92] in /home/dependabot/dependabot-core/go_modules/lib/dependabot/go_modules/file_updater/go_mod_updater.rb
   83:         def update_files # rubocop:disable Metrics/AbcSize, Metrics/PerceivedComplexity
   84:           in_repo_path do
   85:             # Map paths in local replace directives to path hashes
   86:             original_go_mod = File.read("go.mod")
   87:             byebug
=> 88:             puts "** original go.mod:\n#{original_go_mod}"
   89:             original_manifest = parse_manifest
   90:             original_go_sum = File.read("go.sum") if File.exist?("go.sum")
   91:  
   92: 
(byebug)

From there you can evaluate statements like

(byebug) File.exist?("go.sum")
true

Or use n to step to the next line, s to step in or c to continue as some basic commands.

dtomcej · 2021-12-29T22:02:12Z

Hello @mctofu, I think we are getting somewhere:

When using your branch, it appears that the versions get broken after the go get -d github.com/private/P@v0.30.0 step:

[dependabot-core-dev] ~/dependabot-core $ bin/dry-run.rb go_modules private/repository --dep github.com/private/P
=> cloning into /home/dependabot/dependabot-core/tmp/private/repository
=> parsing dependency files
=> updating 1 dependencies: github.com/private/P

=== github.com/private/P (0.29.0)
 => checking for updates 1/1
 => latest available version is 0.30.0
 => latest allowed version is 0.30.0
 => requirements to unlock: own
 => requirements update strategy: 
 => updating github.com/private/P from 0.29.0 to 0.30.0
** original go.mod:
module github.com/private/repository

go 1.17

require (
	(...)
	github.com/private/I v0.53.0
	github.com/private/M v0.38.0
	github.com/private/P v0.29.0
	(...)
)

** substitute go.mod:
module github.com/private/repository

go 1.17

require (
	(...)
	github.com/private/I v0.53.0
	github.com/private/M v0.38.0
	github.com/private/P v0.29.0
	(...)
)

** package: false
** run go get -d github.com/private/P@v0.30.0
** updated dep go.mod:
module github.com/private/repository

go 1.17

require (
	(...)
	github.com/private/I v0.9.0
	github.com/private/M v0.30.0
	github.com/private/P v0.30.0
	(...)
)

** package: false
** run go get -d
** updated go.mod:
module github.com/private/repository

go 1.17

require (
	(...)
	github.com/private/I v0.9.0
	github.com/private/M v0.30.0
	github.com/private/P v0.30.0
	(...)
)

(...)

** version adjusted go.mod:
module github.com/private/repository

go 1.17

require (
	(...)
	github.com/private/I v0.9.0
	github.com/private/M v0.30.0
	github.com/private/P v0.30.0
	(...)
)

    ± go.mod
    ~~~
    13,15c13,15
    < 	github.com/private/I v0.53.0
    < 	github.com/private/M v0.38.0
    < 	github.com/private/P v0.29.0
    ---
    > 	github.com/private/I v0.9.0
    > 	github.com/private/M v0.30.0
    > 	github.com/private/P v0.30.0
    ~~~

	(...)

mctofu · 2021-12-29T23:25:45Z

Thanks @dtomcej! That definitely narrows it down but is also very strange 🤔!

Because ** package: false is in the output that tells me Dependabot is creating a dummy go file in the working dir before running go get

dependabot-core/go_modules/lib/dependabot/go_modules/file_updater/go_mod_updater.rb

Line 184 in c5389f6

File.write(tmp_go_file, "package dummypkg\n") unless package

I'm not familiar with why that's necessary but some next steps to try could be:

Removing that line so the file isn't created
Redo the manual steps in PR's are getting opened that revert a bunch of deps #4536 (comment) but manually create a dummy.go containing package dummypkg and see if that reproduces the issue.

If that doesn't seem to be the culprit the next thing we could try is switching back to an older version of go. That is configured here:

dependabot-core/Dockerfile

Lines 170 to 171 in c5389f6

    
           ARG GOLANG_VERSION=1.17.5 
        
           ARG GOLANG_CHECKSUM=bd78114b0d441b029c8fe0341f4910370925a4d270a6a590668840675b0c653e

Some previous versions: 50b95a0, 1cba38d, 093592f, af35f1b

After changing it you need to relaunch the dev shell with bin/docker-dev-shell --rebuild and then you'll be able to verify it updated successfully by running go version in the shell.

dtomcej · 2021-12-30T20:47:37Z

I think I am making more progress here. After fighting with our corp SSO, I was able to get a lot closer to reproducing the issue in a raw format:

On my workstation (outside the dev container), in our repo, this is the output:

% go version
go version go1.17.5 darwin/amd64
% GOPRIVATE=* go get -v -d github.com/private/P@v0.30.0
go get: upgraded github.com/private/P v0.29.0 => v0.30.0
go get: upgraded github.com/spf13/viper v1.10.0 => v1.10.1
go get: upgraded google.golang.org/grpc v1.42.0 => v1.43.0

In the dev container, in our same repo, this is the output:

$ go version
go version go1.17.5 linux/amd64
$ GOPRIVATE=* go get -v -d github.com/private/P@v0.30.0
get "google.golang.org/grpc": found meta tag vcs.metaImport{Prefix:"google.golang.org/grpc", VCS:"git", RepoRoot:"https://github.com/grpc/grpc-go"} at //google.golang.org/grpc?go-get=1
go get: downgraded github.com/private/I v0.53.0 => v0.9.0
go get: downgraded github.com/private/M v0.38.0 => v0.30.0
go get: upgraded github.com/private/P v0.29.0 => v0.30.0
go get: upgraded github.com/spf13/viper v1.10.0 => v1.10.1
go get: upgraded google.golang.org/grpc v1.42.0 => v1.43.0

mctofu · 2021-12-30T22:21:07Z

@dtomcej: Wow, that's interesting! Curious if you'd get the same results using the golang:1.17.5 docker image.

dtomcej · 2021-12-30T23:24:10Z

@mctofu Turns out we get the exact identical results with the golang:1.17.5 image as we did with the dependabot dev image, so it doesn't seem to be an issue with dependabot at all :(

mctofu · 2021-12-31T01:45:24Z

Would you mind trying with older versions of the golang image to determine if this is a problem that's been recently introduced? We may want to roll Dependabot back to an older version temporarily if it's a bug and knowing which version causes it would help track down the issue.

Thanks for your help in debugging this!

dtomcej · 2022-01-03T22:59:06Z

Hello @mctofu, Have tested with all versions of 1.17, and it does not appear to be related to any of the minor version updates.

It appears that the issue stems from how go handles updating dependencies using go get in combination with go 1.17 lazy module loading, and the go module cache.

Running:

go clean -modcache && \
GOPRIVATE=* go get -v -d github.com/private/P@v0.30.0 && \
GOPRIVATE=* go mod tidy - compat=1.17 && \
git diff go.mod

returns the incorrect versions noted above, whether it is on my local workstation, or in any of the containers.

However:

go clean -modcache && \
GOPRIVATE=* go mod download && \
GOPRIVATE=* go get -v -d github.com/private/P@v0.30.0 && \
GOPRIVATE=* go mod tidy - compat=1.17 && \
git diff go.mod

Returns the correct versions. It appears from what I can tell that the go get -d in the presence of no module cache overwrites and downgrades private/I and private/P despite being direct dependencies, and when the cache is primed, it does not overwrite.

dtomcej · 2022-01-03T23:02:57Z

I will see if I can open a PR to get a patch working, and test it against our repo for confirmation.

mctofu · 2022-02-25T20:13:15Z

@dtomcej Did you have any success in determining what the issue was in go get -d? We could add the go mod download step as a workaround but if there is a fix possible to the go tooling that would be preferred.

dtomcej · 2022-02-28T20:08:00Z

Hello @mctofu unfortunately my tests gave inconsistent results. Since go 1.18 changes this a bit, and is so close, I am waiting to test with 1.18 and see whether that gives my tests more repeatable outcomes.

abdulapopoola · 2024-03-04T21:25:15Z

Closing out as stale

jeffwidman added the T: bug 🐞 Something isn't working label Dec 15, 2021

mctofu added the L: go:modules Golang modules label Dec 16, 2021

jeffwidman mentioned this issue Jan 11, 2022

Incorrect parsing of version into url hostname #4625

Closed

jsoref mentioned this issue Jun 8, 2022

Dependabot downgraded github.com/cert-manager/cert-manager v1.8.0 => v0.7.2 as a result of upgrading k8s.io/component-base cert-manager/approver-policy#71

Closed

This comment was marked as off-topic.

Sign in to view

janisz mentioned this issue Sep 19, 2022

Bump cloud.google.com/go/storage from 1.24.0 to 1.25.0 stackrox/stackrox#2810

Closed

jeffwidman mentioned this issue Sep 20, 2022

Dependabot opened a PR to update one dependency, but also downgraded another dependency. #5757

Open

1 task

This comment was marked as off-topic.

Sign in to view

jeffwidman mentioned this issue Dec 1, 2022

Consider not validating all modules for go updates #6243

Closed

jeffwidman mentioned this issue Feb 7, 2023

Dependabot removes the indirect Go dependencies that are in use since Dec 2022 #6479

Open

abdulapopoola closed this as not planned Won't fix, can't repro, duplicate, stale Mar 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PR's are getting opened that revert a bunch of deps #4536

PR's are getting opened that revert a bunch of deps #4536

jeffwidman commented Dec 15, 2021 •

edited

Loading

jeffwidman commented Dec 15, 2021 •

edited

Loading

dtomcej commented Dec 15, 2021

jurre commented Dec 15, 2021

mctofu commented Dec 15, 2021

mx-psi commented Dec 20, 2021

dtomcej commented Dec 20, 2021

jurre commented Dec 21, 2021

dtomcej commented Dec 21, 2021

dtomcej commented Dec 21, 2021

mctofu commented Dec 21, 2021

dtomcej commented Dec 22, 2021

dtomcej commented Dec 22, 2021

mctofu commented Dec 23, 2021

dtomcej commented Dec 29, 2021

mctofu commented Dec 29, 2021

dtomcej commented Dec 30, 2021

mctofu commented Dec 30, 2021

dtomcej commented Dec 30, 2021

mctofu commented Dec 31, 2021

dtomcej commented Jan 3, 2022

dtomcej commented Jan 3, 2022

mctofu commented Feb 25, 2022

dtomcej commented Feb 28, 2022

This comment was marked as off-topic.

This comment was marked as off-topic.

abdulapopoola commented Mar 4, 2024

PR's are getting opened that revert a bunch of deps #4536

PR's are getting opened that revert a bunch of deps #4536

Comments

jeffwidman commented Dec 15, 2021 • edited Loading

jeffwidman commented Dec 15, 2021 • edited Loading

dtomcej commented Dec 15, 2021

jurre commented Dec 15, 2021

mctofu commented Dec 15, 2021

mx-psi commented Dec 20, 2021

dtomcej commented Dec 20, 2021

jurre commented Dec 21, 2021

dtomcej commented Dec 21, 2021

dtomcej commented Dec 21, 2021

mctofu commented Dec 21, 2021

dtomcej commented Dec 22, 2021

dtomcej commented Dec 22, 2021

mctofu commented Dec 23, 2021

dtomcej commented Dec 29, 2021

mctofu commented Dec 29, 2021

dtomcej commented Dec 30, 2021

mctofu commented Dec 30, 2021

dtomcej commented Dec 30, 2021

mctofu commented Dec 31, 2021

dtomcej commented Jan 3, 2022

dtomcej commented Jan 3, 2022

mctofu commented Feb 25, 2022

dtomcej commented Feb 28, 2022

This comment was marked as off-topic.

This comment was marked as off-topic.

abdulapopoola commented Mar 4, 2024

jeffwidman commented Dec 15, 2021 •

edited

Loading

jeffwidman commented Dec 15, 2021 •

edited

Loading