-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime error: invalid memory address or nil pointer dereference #1852
Comments
I believe this is the same issue I am having, which is reported here: #1804 This is really disrupting our workflow and it is unclear what we can do about it. |
Having same issue, using TF Cloud as backend |
I have had the same issue. #1931 doesn't really fix it... updating didn't change anything. |
@marceloboeira Hmm is it the same stack trace? There could be another nil pointer hiding somewhere that I can't reproduce. |
Yeap:
|
@sapslaj I realised that this started to happen after we've updated gitlab recently, but interestingly enough it doesn't happen to all repos with terraform we have. |
Ahh yes. I was testing with GitHub instead of GitLab so this might be a different issue unrelated to what I fixed then. My bad for linking my PR to this issue without taking a closer look. It looks like it's happening somewhere in PullIsMergable. I wonder if something changed with the GitLab update? It looks like Atlantis is on the latest |
Yes, so we kind of found a way to overpass this bug by creating an empty pipeline on the repo. It seems that on line 208: statuses, _, err := g.Client.Commits.GetCommitStatuses(mr.ProjectID, mr.HeadPipeline.SHA, nil) the I didn't have much time to check what could be an alternative here, maybe using However, it could be other dependencies of isPipelineSkipped := mr.HeadPipeline.Status == "skipped" |
A more complete log + stack trace overview:
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:97
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
github.com/runatlantis/atlantis/server/events/command_runner.go:252 github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics
github.com/runatlantis/atlantis/server/events/command_runner.go:377
runtime.gopanic
runtime/panic.go:1038
runtime.panicmem
runtime/panic.go:221
runtime.sigpanic
runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
github.com/runatlantis/atlantis/server/events/command_runner.go:252 |
In this particular example, `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty. This seems to be caused by the lack of permission to update the commit status. ``` runtime.gopanic runtime/panic.go:1038 runtime.panicmem runtime/panic.go:221 runtime.sigpanic runtime/signal_unix.go:735 github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand github.com/runatlantis/atlantis/server/events/command_runner.go:252 ``` The least invasive solution is to simply use the commit-hash from pull, and guess that the pipeline was "skipped" unless the HeadPipeline is there. The outcome is: When mr.HeadPipeline is present: - use the commit hash and status from the HeadPipeline When mr.HeadPipeline is NOT present: - use the commit hash from pull request struct - assume the pipeline was "skipped" In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable. More info: - runatlantis#1852
In this particular example, `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty. This seems to be caused by the lack of permission to update the commit status. ``` runtime.gopanic runtime/panic.go:1038 runtime.panicmem runtime/panic.go:221 runtime.sigpanic runtime/signal_unix.go:735 github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand github.com/runatlantis/atlantis/server/events/command_runner.go:252 ``` The least invasive solution is to simply use the commit-hash from pull, and guess that the pipeline was "skipped" unless the HeadPipeline is there. The outcome is: When mr.HeadPipeline is present: - use the commit hash and status from the HeadPipeline When mr.HeadPipeline is NOT present: - use the commit hash from pull request struct - assume the pipeline was "skipped" In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable. More info: - runatlantis#1852
After a lot of debugging, it seems that the root cause is a broken permission token + gitlab updating. In resume, we have a user for Atlantis with a token created a long time ago. Over gitlab updates, this token doesn't work the same way a new token, created with the same permissions. atlantis/server/events/vcs/gitlab_client.go Lines 195 to 199 in 55f5d81
This code doesn't error, yet it results on a broken The atlantis/server/events/vcs/gitlab_client.go Lines 208 to 211 in 55f5d81
Running locally, I could simply replace the code to run atlantis/server/events/vcs/gitlab_client.go Line 221 in 55f5d81
In this case, the pipeline is skipped so we could simply: isPipelineSkipped := mr.HeadPipeline == nil || mr.HeadPipeline.Status == "skipped" I have opened #1981 as a way to go around this and avoid the runtime panic... |
For people coming into this issue, if the stack trace is the same, you might've updated GitLab recently which is one of the possible causes for this issue. Solution: create a new token for atlantis with the |
The error on atlantis:
after some troubleshooting using the same gitlab project, branch and commit id with two separate merge request i got two distinct results calling the GitLab API: https://gitlab.com/api/v4/projects/:id/merge_requests/:iid for the field that is used by the go-gitlab module to retrieve the value for
and
Which means there is not issue with the token used because i'm using the same one and i got the two results, one with data on head_pipeline field and the other one with This discards that there is an issue on the module used to access the Gitlab API (github.com/xanzy/go-gitlab) that is doing the expected work or issues on permission access caused by the token. IF the row 208 requires the SHA of the commit for the head_pipeline, maybe that value can be found under |
I observed the same issue with our gitlab instance after updating atlantis to 0.19.2. I downgraded back to 0.17.4, the version we had before the update, and everything currently works again. So while its almost definitely an issue with the gitlab api, older versions do seem to be fine with the api issue. |
is this still happening with |
@jamengual yes, the patch I wrote wasn't accepted so if it wasn't fixed in the gitlab client it can (and does) still happen. We recently updated gitlab on our side and started getting this error again. |
In my case, the problem disappeared after changing the role of the access token from "reporter" to "developer". I don't know if it has anything to do with the error, but I hope it helps someone! |
@c0da for us it was always developer, but somehow we had to renew tokens for every gitlab update... |
This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.' |
following this workaround helped me out
|
For me removing the |
I'm running into the same issue using atlantis v0.23.3, and gitlab version 15.9. I've rotated access tokens, removed the .terraform.lock.hcl file, attempted the gitlab-ci.yml workaround - all with no success. Here is the stack trace I'm receiving:
Any other suggestions? |
it looks like as @marceloboeira said even updating the library did not fixed anything and it broke more than just this.....we will evaluate if this + other fixes are needed and report back but we currently do not have anyone actively working on this. |
@jamengual I used to have a "safety check" PR open for this: Let me know if it makes sense to rebase/reopen that one. |
This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.' |
From what I've gathered, Atlantis is not compatible with GitLab until gitlab fixes their API response. |
This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.' |
Seeing the same issue on Github enterprise server 3.8.x. In my case looks like it happens if the PR does not meet the mergable requirements and the user trying to run
|
Is there a timeline when this will be fixed in the gitlab-client api of Atlantis? |
In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty. This seems to be caused by the lack of permission to update the commit status. ```go runtime.gopanic runtime/panic.go:1038 runtime.panicmem runtime/panic.go:221 runtime.sigpanic runtime/signal_unix.go:735 github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand github.com/runatlantis/atlantis/server/events/command_runner.go:252 ``` The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there. The outcome is: When mr.HeadPipeline is present: - use the commit hash and status from the HeadPipeline When mr.HeadPipeline is NOT present: - use the commit hash from pull request struct - assume the pipeline was "skipped" In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable. More info: - runatlantis#1852
In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty. This seems to be caused by the lack of permission to update the commit status. ```go runtime.gopanic runtime/panic.go:1038 runtime.panicmem runtime/panic.go:221 runtime.sigpanic runtime/signal_unix.go:735 github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand github.com/runatlantis/atlantis/server/events/command_runner.go:252 ``` The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there. The outcome is: When mr.HeadPipeline is present: - use the commit hash and status from the HeadPipeline When mr.HeadPipeline is NOT present: - use the commit hash from pull request struct - assume the pipeline was "skipped" In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable. More info: - runatlantis#1852
In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty. This seems to be caused by the lack of permission to update the commit status. ```go runtime.gopanic runtime/panic.go:1038 runtime.panicmem runtime/panic.go:221 runtime.sigpanic runtime/signal_unix.go:735 github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand github.com/runatlantis/atlantis/server/events/command_runner.go:252 ``` The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there. The outcome is: When mr.HeadPipeline is present: - use the commit hash and status from the HeadPipeline When mr.HeadPipeline is NOT present: - use the commit hash from pull request struct - assume the pipeline was "skipped" In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable. More info: - runatlantis#1852
In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty. This seems to be caused by the lack of permission to update the commit status. ```go runtime.gopanic runtime/panic.go:1038 runtime.panicmem runtime/panic.go:221 runtime.sigpanic runtime/signal_unix.go:735 github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand github.com/runatlantis/atlantis/server/events/command_runner.go:252 ``` The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there. The outcome is: When mr.HeadPipeline is present: - use the commit hash and status from the HeadPipeline When mr.HeadPipeline is NOT present: - use the commit hash from pull request struct - assume the pipeline was "skipped" In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable. More info: - runatlantis#1852
…ty (#3428) In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty. This seems to be caused by the lack of permission to update the commit status. ```go runtime.gopanic runtime/panic.go:1038 runtime.panicmem runtime/panic.go:221 runtime.sigpanic runtime/signal_unix.go:735 github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand github.com/runatlantis/atlantis/server/events/command_runner.go:252 ``` The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there. The outcome is: When mr.HeadPipeline is present: - use the commit hash and status from the HeadPipeline When mr.HeadPipeline is NOT present: - use the commit hash from pull request struct - assume the pipeline was "skipped" In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable. More info: - #1852
fwiw, i ran into this issue on atlantis I'm not sure precisely what fixed it -- maybe all the things! I changed the three things I found in this thread:
I'm back on the road for now. Thank you to all who shared their experiences. |
…ty (runatlantis#3428) In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty. This seems to be caused by the lack of permission to update the commit status. ```go runtime.gopanic runtime/panic.go:1038 runtime.panicmem runtime/panic.go:221 runtime.sigpanic runtime/signal_unix.go:735 github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand github.com/runatlantis/atlantis/server/events/command_runner.go:252 ``` The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there. The outcome is: When mr.HeadPipeline is present: - use the commit hash and status from the HeadPipeline When mr.HeadPipeline is NOT present: - use the commit hash from pull request struct - assume the pipeline was "skipped" In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable. More info: - runatlantis#1852
…ty (runatlantis#3428) In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty. This seems to be caused by the lack of permission to update the commit status. ```go runtime.gopanic runtime/panic.go:1038 runtime.panicmem runtime/panic.go:221 runtime.sigpanic runtime/signal_unix.go:735 github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand github.com/runatlantis/atlantis/server/events/command_runner.go:252 ``` The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there. The outcome is: When mr.HeadPipeline is present: - use the commit hash and status from the HeadPipeline When mr.HeadPipeline is NOT present: - use the commit hash from pull request struct - assume the pipeline was "skipped" In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable. More info: - runatlantis#1852
Hi,
I am trying to integrate atlantis (v0.17.4) with gitlab (free self managed ver. 11.9.6).
It is working fine till autoplan
Comment output:atlantis apply -p preprod
🚮 To delete this plan click ***
🔁 To plan this project again, comment:
atlantis plan -p preprod
⏩ To apply all unapplied plans from this pull request, comment:
atlantis apply
🚮 To delete all plans and locks for the PR, comment:
atlantis unlock
But fails when I do "atlantis apply -p preprod":
Error: goroutine panic. This is a bug.
Following is the request body sent by gitlab
Installation repoConfig
atlantis.yaml at repo's root
Thanks for any help!
The text was updated successfully, but these errors were encountered: