Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sq 8 2+azure devops #202

Closed
wants to merge 6 commits into from
Closed

Conversation

jcuzzi
Copy link
Contributor

@jcuzzi jcuzzi commented Jun 30, 2020

Manual merge of @Iloer's Azure DevOps PR into @mc1arke's sq-8_2-support branch (with some additions to get it all working).

  • Add support for Azure DevOps PR Decoration

Notes:

  1. The following should be set in the scanner properties:
    sonar.pullrequest.vsts.instanceUrl=https://[organization].visualstudio.com/
    sonar.pullrequest.vsts.project=[Project]
    sonar.pullrequest.vsts.repository=[Repository]

  2. Ensure that the following option is unchecked in the Azure DevOps Branch Policy configuration, otherwise an exception will be thrown:
    "Reset status whenever there are new changes"

mc1arke and others added 4 commits June 7, 2020 15:22
Sonarqube 8.2 introduced a new `ProjectDto` with associated changes in `ProjectAlmSettingsDao` to require this new class in various methods, as well as changes in `ComponentFinder` to allow retrieval of instances of a project. This change makes use of these new classes and methods to allow the Web Services for setting up ALM bindings to operate.

As this class and the associated methods did not exist in previous versions of Sonarqube, this change breaks backwards compatibility, so means the plugin will now only support Sonarqube 8.2.

Sonarqube 8.2 also provides the ability to set the optional URL parameter on a Gitlab project so that scans run outside of Gitlab CI operate properly, so the additional parameter is now included in the appopriate WebServices and the Gitlab decorator.
@twaddell
Copy link

twaddell commented Jul 2, 2020

I can confirm this build works well in sonarqube version 8.3.1.34397
I've tested with a couple of PRs and comments were added in Azure Devops. I will leave this build running on our server and report back if any issues arise.

@btshft
Copy link

btshft commented Jul 2, 2020

UPD: I figured out what the problem was. Target branch should have been analyzed before creating the PR. Problem solved.

I get the following error on version 8.3.1.34397 with on-premise Azure DevOps. Any ideas?

==============================================================================
Task         : Publish Quality Gate Result
Description  : Publish SonarQube's Quality Gate result on the Azure DevOps build result, to be used after the actual analysis.
Version      : 4.8.1
Author       : sonarsource
Help         : Version: 4.8.1. [More Information](http://redirect.sonarsource.com/doc/install-configure-scanner-tfs-ts.html)
==============================================================================
##[error][SQ] Task failed with status FAILED, Error message: Could not find target branch 'null' in project

Sonar Properties:

{
  "sonar.host.url": "http://sonarqube.dev.local/",
  "sonar.login": "*******",
  "sonar.projectKey": "*******",
  "sonar.projectName": "*******",
  "sonar.projectVersion": "1.0.0+7ddb00c1e6b4de01bc65fb6a0647b30e858d20f6",
  "sonar.pullrequest.key": "661",
  "sonar.pullrequest.base": "develop",
  "sonar.pullrequest.branch": "azure-pipelines-pull-requests",
  "sonar.pullrequest.provider": "vsts",
  "sonar.pullrequest.vsts.instanceUrl": "https://azure.dev.local/tfs/DefaultCollection/",
  "sonar.pullrequest.vsts.project": "*******",
  "sonar.pullrequest.vsts.repository": "*******",
  "sonar.scanner.metadataFilePath": "/azp/agent/_work/_temp/sonar/20200703.10/321f1354-82cf-599f-2c3f-9959d0134674/report-task.txt"
}
Compute Engine logs
2020.07.03 08:06:06 INFO  ce[][o.s.c.t.CeWorkerImpl] Execute task | project=****** | type=REPORT | pullRequest=661 | id=AXMTthCS8fQE63MSzW1Z | submitter=dev-admin
2020.07.03 08:06:06 DEBUG ce[AXMTthCS8fQE63MSzW1Z][o.s.c.t.p.s.ExtractReportStep] Analysis report is 146 KB uncompressed
2020.07.03 08:06:06 INFO  ce[AXMTthCS8fQE63MSzW1Z][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=41ms
2020.07.03 08:06:06 INFO  ce[AXMTthCS8fQE63MSzW1Z][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=4ms
2020.07.03 08:06:06 INFO  ce[AXMTthCS8fQE63MSzW1Z][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=0ms
2020.07.03 08:06:06 INFO  ce[AXMTthCS8fQE63MSzW1Z][o.s.c.t.s.ComputationStepExecutor] Execute DB migrations for current project | status=SUCCESS | time=1ms
2020.07.03 08:06:06 INFO  ce[AXMTthCS8fQE63MSzW1Z][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
2020.07.03 08:06:06 INFO  ce[AXMTthCS8fQE63MSzW1Z][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=FAILED | time=7ms
2020.07.03 08:06:06 ERROR ce[AXMTthCS8fQE63MSzW1Z][o.s.c.t.s.ComputationStepExecutor] Execution of listener failed
java.lang.IllegalStateException: Branch has not been set
	at com.google.common.base.Preconditions.checkState(Preconditions.java:508)
	at org.sonar.ce.task.projectanalysis.analysis.AnalysisMetadataHolderImpl.getBranch(AnalysisMetadataHolderImpl.java:164)
	at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.createBranch(PostProjectAnalysisTasksExecutor.java:232)
	at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.createProjectAnalysis(PostProjectAnalysisTasksExecutor.java:176)
	at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.finished(PostProjectAnalysisTasksExecutor.java:107)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeListener(ComputationStepExecutor.java:91)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:63)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:209)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:191)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:158)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:133)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:85)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
2020.07.03 08:06:06 ERROR ce[AXMTthCS8fQE63MSzW1Z][o.s.c.t.CeWorkerImpl] Failed to execute task AXMTthCS8fQE63MSzW1Z
java.lang.IllegalStateException: Could not find target branch 'null' in project
	at com.github.mc1arke.sonarqube.plugin.ce.CommunityBranchLoaderDelegate.createPullRequest(CommunityBranchLoaderDelegate.java:98)
	at com.github.mc1arke.sonarqube.plugin.ce.CommunityBranchLoaderDelegate.load(CommunityBranchLoaderDelegate.java:78)
	at com.github.mc1arke.sonarqube.plugin.ce.CommunityBranchLoaderDelegate.load(CommunityBranchLoaderDelegate.java:50)
	at org.sonar.ce.task.projectanalysis.component.BranchLoader.load(BranchLoader.java:44)
	at org.sonar.ce.task.projectanalysis.step.LoadReportAnalysisMetadataHolderStep.execute(LoadReportAnalysisMetadataHolderStep.java:90)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:209)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:191)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:158)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:133)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:85)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
2020.07.03 08:06:06 INFO  ce[AXMTthCS8fQE63MSzW1Z][o.s.c.t.CeWorkerImpl] Executed task | project=****** | type=REPORT | pullRequest=661 | id=AXMTthCS8fQE63MSzW1Z | submitter=dev-admin | status=FAILED | time=107ms

@btshft
Copy link

btshft commented Jul 6, 2020

Now I can confirm, this build works fine with SonarQube 8.3.1.34397 and Azure on-premise with REST API 5.1-preview.
The only thing I had to rebuild the plugin with the Azure REST API version 5.1-preview. So it would be nice to move version to plugin settings instead of hardcode constant.

@alekssako
Copy link

Throwing error on ce.log :

Caused by: java.net.URISyntaxException: Illegal character in path at index 53: http://xxxxxxxxxxxxxx/tfs/xxxxxxxxx/DEVOPS IMPLEMENTATION/_apis/git/repositories/xxxxxxxxxxx/pullRequests/4004/statuses?api-version=5.1-preview.1
        at java.base/java.net.URI$Parser.fail(URI.java:2913)
        at java.base/java.net.URI$Parser.checkChars(URI.java:3084)
        at java.base/java.net.URI$Parser.parseHierarchical(URI.java:3166)
        at java.base/java.net.URI$Parser.parse(URI.java:3114)
        at java.base/java.net.URI.<init>(URI.java:600)
        at java.base/java.net.URI.create(URI.java:881)
        ... 24 common frames omitted

I think should fix spaces on project names or repository names. If having spaces it throws errors.

@jcuzzi
Copy link
Contributor Author

jcuzzi commented Jul 8, 2020

@alekssako f203253 allows for spaces in the URI - it should encode the full URI before sending the request.

Thanks for the reviews.

@alekssako
Copy link

alekssako commented Jul 9, 2020

@alekssako f203253 allows for spaces in the URI - it should encode the full URI before sending the request.

Thanks for the reviews.

@jcuzzi Keep it up with the good work you`ve done.

I found out that in sonarqube "See the PR" link is referencing to the API instead of the PR itself on the azure devops.

Annotation 2020-07-09 102250

Can't find it in the code, but its something like :
http://xxxxxxx/tfs/xxxxxx/ProjectName/_apis/git/repositories/RepoName/pullRequests/4004 this is referencing to the api,

Instead it should be something like :
http://xxxxxxx/tfs/xxxxxx/ProjectName/_git/RepoName/pullrequest/4004

Copy link
Owner

@mc1arke mc1arke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. I've given it an initial review although haven't tried executing it yet so may come back with further comments.

A few points not necessarily covered in my comments:

  • Formatting: please ensure opening curly braces are part of the preceding line rather than being on their own line. Your change has some of these correct but seems to switch for some classes
  • Please squash your commits and ensure your commit message explains why you've done things you have (i.e. if there's anything you want people to understand why you've done something then put it in the commit message)
  • Your decorator class needs to be immutable for thread safety. The rest of the classes in the plugin also aim for immutability for consistency, so please aim for this too.

If you have any questions about any of my comments or disagree with any of them then please shout!

LOGGER.info("Found " + properties.size() + " scanner properties...");

for (Map.Entry<String,String> entry : properties.entrySet())
LOGGER.debug("Key = " + entry.getKey() + ", Value = " + entry.getValue());
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please put curly braces around this to make it clear what the limits of the for loop are

LOGGER.info("Found " + properties.size() + " scanner properties...");

for (Map.Entry<String,String> entry : properties.entrySet())
LOGGER.debug("Key = " + entry.getKey() + ", Value = " + entry.getValue());
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use String.format rather than String concatenation please?

.getIssues();
/*.stream()
.filter(i -> OPEN_ISSUE_STATUSES.contains(i.getIssue().getStatus()))
.collect(Collectors.toList());*/
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this code relevant to something? I suspect you do actually need to filter, but if you don't then please remove commented out code

.count() > 0 ) {

if(!issue.getIssue().getStatus().equals(Issue.STATUS_OPEN)
&& azureThread.getStatus().equals(CommentThreadStatus.ACTIVE)) {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use == and != for enum comparison

private Boolean deleted;
@JsonProperty("_links")
private ReferenceLinks links;
public CommentThread(){};
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this constructor used? In theory, all the fields above it should be final, but this constructor prevents that.

Comment on lines +49 to +57
projectConfiguration.get(GitlabServerPullRequestDecorator.PULLREQUEST_GITLAB_INSTANCE_URL).ifPresent(v -> sensorContext
.addContextProperty(GitlabServerPullRequestDecorator.PULLREQUEST_GITLAB_INSTANCE_URL, v));
projectConfiguration.get(GitlabServerPullRequestDecorator.PULLREQUEST_GITLAB_PROJECT_ID).ifPresent(v -> sensorContext
.addContextProperty(GitlabServerPullRequestDecorator.PULLREQUEST_GITLAB_PROJECT_ID, v));
projectConfiguration.get(GitlabServerPullRequestDecorator.PULLREQUEST_GITLAB_PROJECT_URL).ifPresent(v -> sensorContext
.addContextProperty(GitlabServerPullRequestDecorator.PULLREQUEST_GITLAB_PROJECT_URL, v));
projectConfiguration.get(GitlabServerPullRequestDecorator.PULLREQUEST_GITLAB_PIPELINE_ID).ifPresent(v -> sensorContext
.addContextProperty(GitlabServerPullRequestDecorator.PULLREQUEST_GITLAB_PIPELINE_ID, v));

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have these been added? They seem unrelated to the intent of your change

.addContextProperty(AzureDevOpsServerPullRequestDecorator.PULLREQUEST_AZUREDEVOPS_PULLREQUEST_ID, v));


Optional.ofNullable(system2.property(AzureDevOpsServerPullRequestDecorator.PULLREQUEST_AZUREDEVOPS_INSTANCE_URL)).ifPresent(
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these variables not be the ones that are injected by Azure DevOps, rather than custom-named ones?

Comment on lines +125 to +128
public Map<String,String> getScannerProperties() {
return scannerContext.getProperties();
}

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only seems to have been introduced to allow logging of all the properties, which doesn't seem like a good use case. The logging should either happen in the PostAnalysisTask that triggers decoration, or be removed.

Comment on lines +56 to +58
private String authorizationHeader;
private static final Logger LOGGER = Loggers.get(AzureDevOpsServerPullRequestDecorator.class);
public static final String API_VERSION_PREFIX = "?api-version=";
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please ensure you're following the Java Language Specification field ordering:
public static final
private static final
private final
private

Comment on lines +60 to +66
private String apiVersion = "6.0-preview.1";
private String azureUrl = "";
private String baseBranch = "";
private String branch = "";
private String pullRequestId = "";
private String azureRepositoryName = "";
private String azureProjectId = "";
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should all be local variables that get passed to any required methods, otherwise it will cause a race condition if multiple decorators are executed at the same time

@mc1arke
Copy link
Owner

mc1arke commented Jul 17, 2020

Sorry, I've just merged the 8.2 support branch which has auto-closed your MR. Could you open a new one pointing at master please?

@jcuzzi
Copy link
Contributor Author

jcuzzi commented Jul 21, 2020

Yeah, once I have time to make the requested changes, I'll create a new PR to master.

@bambuca bambuca mentioned this pull request Jul 22, 2020
@jcuzzi
Copy link
Contributor Author

jcuzzi commented Jul 24, 2020

@alekssako f203253 allows for spaces in the URI - it should encode the full URI before sending the request.
Thanks for the reviews.

@jcuzzi Keep it up with the good work you`ve done.

I found out that in sonarqube "See the PR" link is referencing to the API instead of the PR itself on the azure devops.

Annotation 2020-07-09 102250

Can't find it in the code, but its something like :
http://xxxxxxx/tfs/xxxxxx/ProjectName/_apis/git/repositories/RepoName/pullRequests/4004 this is referencing to the api,

Instead it should be something like :
http://xxxxxxx/tfs/xxxxxx/ProjectName/_git/RepoName/pullrequest/4004

This should be fixed in #218

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants