Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Projects created automatically are broken #738

Closed
felipecrs opened this issue Feb 23, 2023 · 22 comments
Closed

Projects created automatically are broken #738

felipecrs opened this issue Feb 23, 2023 · 22 comments
Labels
bug Something isn't working invalid This doesn't seem right

Comments

@felipecrs
Copy link

Describe the bug
If I run mvn sonar:sonar with the sonar.pullrequest parameters for a project that does not already exist in Sonarqube, the project gets created automatically.

However, in that case, the project that gets created in Sonarqube is apparently broken because when I try to open it from the UI this shows up:

image

Digging a bit further, this issue is apparently related to the branch name of the project. I think the project does not get created by the plugin with the correct branch name set as the main branch.

If I create the project manually first in SonarQube, and set its main branch to the correct branch, then mvn sonar:sonar works as expected.

To Reproduce

  1. Run mvn sonar:sonar with the sonar.pullrequest parameters for a project that does not exist already in Sonarqube.
  2. After the analysis runs, go to Sonarqube and try to open the project from its UI. It should fail with the same error as the screenshot of earlier.

Expected behavior
To have the project created with the target branch sonar.pullrequest.base set as the default branch.

Screenshots
See above.

Software Versions

  • SonarQube Version: 9.9 (build 65466)
  • Plugin Version: 1.14.0

Additional context
This causes the Sonar Gerrit Jenkins plugin to fail for some weird reasons like:

SonarQube analysis 'AYaAYSId_x56GYGzmtJ8' failed with message: Could not find target branch 'null' in project.

The Sonarqube logs shows things like:

ERROR ce[AYaAYSId_x56GYGzmtJ8][o.s.c.t.s.ComputationStepExecutor] Execution of listener failed
java.lang.IllegalStateException: Branch has not been set
java.lang.IllegalStateException: Could not find target branch 'null' in project
ERROR web[AYZ/fJ+hFk6S4/ySAAUs][o.s.s.w.WebServiceEngine] Fail to process request http://<host>/sonarqube/api/project_pull_requests/list?project=<project>%3Aspring-app-parent
java.lang.NullPointerException: Pull request data should be available for branch type PULL_REQUEST
@felipecrs felipecrs added the bug Something isn't working label Feb 23, 2023
@mc1arke
Copy link
Owner

mc1arke commented Feb 24, 2023

I don't think this can work the way you want. Sonarqube needs to have the analysis of the target branch in place before it can perform a Pull Request analysis, so dynamically creating a project during pull request submission isn't something that would feasibly allow this since the target branch couldn't already exist in Sonarqube and Sonarqube would therefore find no baseline to work against. The error messages could probably do with some clarification here, but I'm closing this as invalid given Sonarqube core can't support this way of working

@mc1arke mc1arke closed this as not planned Won't fix, can't repro, duplicate, stale Feb 24, 2023
@mc1arke mc1arke added the invalid This doesn't seem right label Feb 24, 2023
@felipecrs
Copy link
Author

Well, shouldn't at least the scan fail more gracefully in this case?

For example, instead of having the project created brokenly in Sonarqube, requiring some administrator to delete the broken project and recreate it manually, it could simply fail during the first scan saying "please create the project in Sonarqube before issuing the first scan".

@felipecrs
Copy link
Author

Thanks for the answer by the way.

@felipecrs
Copy link
Author

felipecrs commented Feb 24, 2023

Sonarqube needs to have the analysis of the target branch in place before it can perform a Pull Request analysis

I don't think this is the case, not according to my tests. As long as the branch is also fetched by git during the pull request analysis, the Sonarqube analyses it just fine. I did a few tests for it (without having Sonarqube scanned the branch prior to the PR analysis).

so dynamically creating a project during pull request submission isn't something that would feasibly allow this since the target branch couldn't already exist in Sonarqube

But this is exactly what happens today: the project gets created automatically but without the branch, and I'm reporting that, as part of the automatic creation of the project, I'd say the branch had to be created as well.

@MDullak
Copy link

MDullak commented Feb 27, 2023

Hi,

I upgraded today from 8.9 LTS with plugin 1.8.1 to LTS with plugin 1.14.0.
Everything looked OK up to the point where I now have the same issue: same screen + same log message.
My branches were OK, unless something got broken during the migration (which I am not aware of).

Now I can not do anything. Not even manually go to sonarqube.xxx.com/project/branches?id=Project_Name to see what I have and what I am missing.

I am running branch to which the PRs were merging now (our main development branch) and will report if that fixed the problem.

@MDullak
Copy link

MDullak commented Feb 27, 2023

So the moment that analysis was scheduled (it is still in progress) everything is back OK.
Strange thing is that main branch was already analyzed according to log.
We will see what will happen once current analysis in completed.

@Wihrt
Copy link

Wihrt commented Feb 28, 2023

I have looked into this problem.

For the following explanation, we will run an analysis for a non-existant branch with directly a merge request.

My analysis is the following :

  • Analysis ▶️ Fails because the target branch of the merge request does not exists. It is totally normal.
  • Database ▶️ Writes in database a row with the merge request and the missing info. It is strange
  • Web ▶️ Throws an error because of the merge request with the missing info. It is normal.

The brutal solution would be to destroy the project, then re-create it properly.

The clean solution is to remove only the errors.
To do that, you would need the database and the API.

Database
Run the following Query :
SELECT project_uuid, kee FROM public.project_branches WHERE branch_type = 'PULL_REQUEST' and merge_branch_uuid is null

You can run a small query to get the project key. It will be needed for the API

API
Run the following command:
curl -u <user>:<password> https://sonar/api/project_pull_requests/delete?project=<project_key>&pullRequest=<pullRequestId>

Once you have done that, your project is now available.

If you are interested, I have developed a small Python script to perform that :)

@MDullak
Copy link

MDullak commented Feb 28, 2023

Hi, I got it back up and running by manually performing analysis of the target branch (my main branch).
This resolved the issue and I have all the data visible now.
Maybe I should have opened new issue not commenting on this one since my issue was not on a new project, it was after upgrade.

@felipecrs
Copy link
Author

Anyway, it's not necessary to perform an analysis of the target branch. The only requirement is that the target branch exists in SonarQube (it does not need to be analyzed).

This is why I think a pull request execution should also create the target branch automatically.

@mc1arke
Copy link
Owner

mc1arke commented Mar 1, 2023

The only requirement is that the target branch exists in SonarQube (it does not need to be analyzed).

You can't have a branch in Sonarqube without an analysis, and having a branch without an up-to-date analysis would render the PR analysis as invalid since the baseline wouldn't have all the findings to perform a comparison against

This is why I think a pull request execution should also create the target branch automatically.

This isn't something that would be reasonable to try and support since it would give potentially misleading results. The scanner component that submits details for analysis also does not support submitting more than the currently checked-out branch, so this operation wouldn't be feasible.

@felipecrs
Copy link
Author

felipecrs commented Mar 1, 2023

You can't have a branch in Sonarqube without an analysis, and having a branch without an up-to-date analysis would render the PR analysis as invalid since the baseline wouldn't have all the findings to perform a comparison against

According to my tests you are wrong in this statement. However, let's assume you are right.

So, someone is trying to add the sonarqube scan to their project. The way how the scan is introduced in the CI workflow has to be through a PR, then, after PR is merged, Sonarqube is finally able to scan the branch for the first time and produce meaningful results for the following PR scans.

However, with the current behavior, the user would be stopped to even merge the PR to begin with.

What's your proposal to that?

In my opinion, even if you were right about the scan producing misleading results, it should be acceptable as a first time scan (otherwise the person can't even merge the PR which adds the scan and thus the scan will never run for the branch).

The scanner component that submits details for analysis also does not support submitting more than the currently checked-out branch, so this operation wouldn't be feasible.

But I'm not talking about publishing an analysis of the branch while in pull request mode. I'm talking about creating the branch for the project (without publishing any analysis for the branch), that's a different thing.

@mustafaglr
Copy link

Does your branch has special charachters? If there is could you try to put your branch name in single quotes?
(E.g. -Dsonar.pullrequest.branch='hotfix/1.1')

@felipecrs
Copy link
Author

Mine hasn't (branch name is "develop").

@mjaggard
Copy link

Database Run the following Query : SELECT project_uuid, kee FROM public.project_branches WHERE branch_type = 'PULL_REQUEST' and merge_branch_uuid is null

You can run a small query to get the project key. It will be needed for the API

API Run the following command: curl -u <user>:<password> https://sonar/api/project_pull_requests/delete?project=<project_key>&pullRequest=<pullRequestId>

Once you have done that, your project is now available.

Doing this means that I now get no results in the SQL query but the project is still unavailable

@Wihrt
Copy link

Wihrt commented Jun 26, 2023

Database Run the following Query : SELECT project_uuid, kee FROM public.project_branches WHERE branch_type = 'PULL_REQUEST' and merge_branch_uuid is null
You can run a small query to get the project key. It will be needed for the API
API Run the following command: curl -u <user>:<password> https://sonar/api/project_pull_requests/delete?project=<project_key>&pullRequest=<pullRequestId>
Once you have done that, your project is now available.

Doing this means that I now get no results in the SQL query but the project is still unavailable

Honestly, I have tested this procedure in SonarQube 9.8.
Since I don't know on which version you are running your Sonarqube.
Quite difficult to help you without other informations ...

@felipecrs
Copy link
Author

felipecrs commented Jun 26, 2023

In my case, it's SonarQube 9.9.1 LTS.

@mjaggard
Copy link

9.9.1.69595 for me. Any idea what else I can try to look into?

@mjaggard
Copy link

@Wihrt apologies that did work for me. I have no idea why the SQL request didn't return the problem after the first time I tried. Maybe I misread or changed the command somehow.

Anyway, I did find one problem which was that you have to POST to that URL in order for the delete to work. For example:
curl -vvv -d '' -u api_key: 'https://sonarhost/api/project_pull_requests/delete?project=projectkey&pullRequest=1336'

@mjaggard
Copy link

@mc1arke I think there is a bug here. If you create a pull request from one branch to another that is not scanned, it completely crashes the whole project in Sonarqube. In my case I created a new branch from a colleagues' slightly longer-running branch and it broke everyone's use of Sonar. I'm happy for my PR to not work but it shouldn't break the whole project

mc1arke added a commit that referenced this issue Jan 1, 2024
The scanner currently skips validation of a target branch if a Pull
Request is used to create a new project, so the resulting project fails
to load in front-end due to the Pull Request API treating the data on
that pull request as invalid. This is being overcome by validating that
a target branch exists for all Pull Request submissions and rejecting
the scan submission if the target branch is not found in Sonarqube.

Additionally, there's a delay between a Pull Request being recorded in
the database by the server component as a result of the call from the
scanner, and the Compute Engine recording the Pull Request details
(source, target, title etc.) against the branch. During this time the
Pull Request treats that Pull Request as invalid and throws an error,
meaning the project cannot be loaded through the UI, or the Pull
Requests listed through the API. As the Pull Request response fields
filled from the Pull Request data are not mandatory, those fields are
now only being completed if the Pull Request data is set on the branch
DTO rather than throwing an exception if the data isn't set.
mc1arke added a commit that referenced this issue Jan 1, 2024
The scanner currently skips validation of a target branch if a Pull
Request is used to create a new project, so the resulting project fails
to load in front-end due to the Pull Request API treating the data on
that pull request as invalid. This is being overcome by validating that
a target branch exists for all Pull Request submissions and rejecting
the scan submission if the target branch is not found in Sonarqube.

Additionally, there's a delay between a Pull Request being recorded in
the database by the server component as a result of the call from the
scanner, and the Compute Engine recording the Pull Request details
(source, target, title etc.) against the branch. During this time the
Pull Request treats that Pull Request as invalid and throws an error,
meaning the project cannot be loaded through the UI, or the Pull
Requests listed through the API. As the Pull Request response fields
filled from the Pull Request data are not mandatory, those fields are
now only being completed if the Pull Request data is set on the branch
DTO rather than throwing an exception if the data isn't set.
@mwallace72
Copy link

Would it be possible to get this change for v1.14, to provide this for Sonarqube 9.9 LTS? I'm not opposed to making the PR, assuming it's just the same code as the fix in v1.17, but this would be my first contribution so wanted to ask first.

@felipecrs
Copy link
Author

That would be super nice for me as well.

rafaelissaomiyagawa pushed a commit to rafaelissaomiyagawa/sonarqube-community-branch-plugin that referenced this issue Mar 15, 2024
@felipecrs
Copy link
Author

@mc1arke is it very hard to release a newer version of the plugin for SonarQube 9.9 LTS?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working invalid This doesn't seem right
Projects
None yet
Development

No branches or pull requests

7 participants