Azure Devops support #719

mcdafydd · 2019-07-26T20:08:47Z

This pull request adds support for using Azure Devops as a VCS provider.

- remove ADHostname - add ADOrg and ADProject

cmd/server.go: - Add default empty strings for Azure Devops webhook basic user/pass server/user_config.go: - Rename AzureDevopsBasicUser to AzureDevopsWebhookBasicUser to signify difference from the API call credentials server/server.go: - fix azuredevopsClient var - add Azure Devops to []supportedVCSHosts - add AzureDevopsRequestValidator for basic auth webhook support server/azuredevops_request_validator*.go: - add charset utf-8 to support content-types for webhooks - start adding validator tests server/events_controller.go*: - change azuredevops webhook header to Request-Id - move Validate() and Basic Auth webhook support to go-azuredevops - add pull and comment event functions

- remove unneeded NewAzureDevopsClient() code - change to using const refs - add GetPullRequest()

- Pull request events - Work item comment events

- Add missing headRepo set to AzureDevops event parser - Strip 'refs/heads' from branch values sent to models.PullRequest - Fix calls to NewClient() to use org, not username - Add notes about use of /'s in repo FullName - Force use of 'dev.azure.com' for VCSHost.Hostname value - NewAzureDevopsClient(): Rename 'account' to 'username' to match docs - NewAzureDevopsClient(): Rename unused 'hostname' to 'org'

- Add bluemonday to strip HTML tags from comments - Rename all WorkItemEvent refs to WorkItemCommentedEvent - Rename pullEvent to event to match types - Minor comment clean-ups - Support multiple pull request links in a work item

- Fixes logic error in PullIsMergeable() - Replace GetRemoteURL() with GetWebURL() - returns what we want

- Remove project from NewRepo() calls - Add new function SplitAzureDevopsRepoFullName

Co-Authored-By: Luke Kysow <1034429+lkysow@users.noreply.github.com>

lkysow · 2019-10-02T21:35:06Z

Can you help explain why users would rather link and comment on a work item for each pull request rather than comment directly on the PR?

mcdafydd · 2019-10-03T02:34:54Z

Can you help explain why users would rather link and comment on a work item for each pull request rather than comment directly on the PR?

I was following the idea in the Microsoft documentation here that recommends driving development from the work item, which seemed fine at the time given the lack of the PR comment webhook.

In our environment today, putting comments directly on the PR and more closely linking the Terraform plan with its execution makes sense. I can't think of a great reason we would need to run atlantis commands from a work item rather than a PR, but I imagine we will remain interested in linking our pull requests to a work item for tracking.

I'm happy to defer to more experienced teams doing this and adopt the recommended guidance. I started adding support for the new event payload to go-azuredevops. I don't suspect it'll take too much more work to finish it.

Thanks!

mcdafydd · 2019-10-04T13:35:08Z

I deployed a new branch from my fork just now azure-devops-pr-comment. I'll test it out a bit. The pull request comments are threaded. I'm creating a new thread for each response from atlantis for now, rather than sending the atlantis response into the same thread with the command, since I don't have an easy way to pass the thread and parent comment ID from the webhook payload over to the vcs.CreateComment() function. Also, this code only supports Resource Version 2.0 of the webhook payload (which seems to be the default).

The branch would still need some code removed presuming we move forward with this method for responding to comments.

Edit: removing quoting

lkysow · 2019-10-04T17:42:21Z

👍 I like going this route.

- Update docs - Remove work item commented event code

mcdafydd · 2019-10-05T17:11:14Z

@dedamico @naru014 @jpreese If you're interested in testing the changes, I'd love any feedback.
All work item comment connectivity is gone, so interaction is now solely through pull request creation, updates, and comments, similar to other VCS providers.

I ran into the issue yesterday where upon PR creation I immediately got back a comment saying the workspace was already locked. It was caused by a master branch policy that automatically adds reviewers. So when the PR was created, a "pullrequest.created" event was generated, and then about 1 second later a "pullrequest.updated" event was sent when the reviewers were added.

Is this a situation we want to improve in code at all or just document?

lkysow · 2019-10-07T19:27:27Z

Is this a situation we want to improve in code at all or just document?

Can we find out that what changed was just new reviewers? We should only act on pull request update events that are about pushing a new commit (or force pushing). We should detect this before it gets to the stage where it ends up triggering locking.

mcdafydd · 2019-10-08T01:43:19Z

Can we find out that what changed was just new reviewers? We should only act on pull request update events that are about pushing a new commit (or force pushing). We should detect this before it gets to the stage where it ends up triggering locking.

Ok, sounds good. Yes, I'm sure this is what triggered it. Looking at the webhook payloads, I think the primary way we can detect it is in the message text "changed the reviewer list". There's no clear field for this particular type of pullrequest.updated event type.

Here are the relevant log snippets. First, the PR opening webhook in the ADO event history:

Sent at: Saturday, October 5, 2019 12:41:20 AM
Message
Dev created pull request 11871 (PR title) in REPO

Atlantis receives it:

2019/10/05 00:41:20+0000 [INFO] middleware.go:38 server: POST /events – from 20.36.242.132:5952
2019/10/05 00:41:20+0000 [DBUG] events_controller.go:130 server: Handling AzureDevops post
2019/10/05 00:41:20+0000 [DBUG] events_controller.go:226 server: Request valid
2019/10/05 00:41:20+0000 [DBUG] events_controller.go:242 server: Handling as pull request event
2019/10/05 00:41:20+0000 [INFO] events_controller.go:547 server: Identified event as type "opened"
2019/10/05 00:41:20+0000 [INFO] events_controller.go:347 server: Executing autoplan
2019/10/05 00:41:20+0000 [INFO] middleware.go:42 server: POST /events – respond HTTP 200

Then the second PR updated event history log:

Sent at: Saturday, October 5, 2019 12:41:21 AM
Message
Dev changed the reviewer list for pull request 11871 (PR title) in REPO

and receipt:

2019/10/05 00:41:21+0000 [INFO] middleware.go:38 server: POST /events – from 20.36.242.132:5953
2019/10/05 00:41:21+0000 [DBUG] events_controller.go:130 server: Handling AzureDevops post
2019/10/05 00:41:21+0000 [DBUG] events_controller.go:226 server: Request valid
2019/10/05 00:41:21+0000 [DBUG] events_controller.go:242 server: Handling as pull request event
2019/10/05 00:41:21+0000 [INFO] events_controller.go:547 server: Identified event as type "updated"
2019/10/05 00:41:21+0000 [INFO] events_controller.go:347 server: Executing autoplan
2019/10/05 00:41:21+0000 [INFO] middleware.go:42 server: POST /events – respond HTTP 200
2019/10/05 00:41:21+0000 [DBUG] project_command_builder.go:102 org/project/REPO#11871: Got workspace lock
2019/10/05 00:41:21+0000 [WARN] project_command_builder.go:99 org/project/REPO#11871: Workspace was locked
2019/10/05 00:41:21+0000 [DBUG] project_command_builder.go:110 org/project/REPO#11871: 5 files were modified in this pull request
2019/10/05 00:41:21+0000 [INFO] working_dir.go:123 org/project/REPO#11871: Creating dir "/home/user/.atlantis/repos/org/project/REPO/11871/default"
2019/10/05 00:41:22+0000 [EROR] command_runner.go:410 org/project/REPO#11871: The default workspace is currently locked by another command that is running for this pull request–wait until the previous command is complete and try again

lkysow · 2019-10-08T19:18:01Z

Looks like you're right. We can do a string.Contains to look for a message that the source branch was updated. Or we could require users specify the webhook change type to only be source branch updated. However I like not requiring users to set this and us just detecting the right types for them.

mcdafydd · 2019-10-08T21:21:27Z

I just saw this feature too :). We would also need "Votes score changed" events for vcs.PullIsApproved(). I'm down to make it easier for users so will add something to match the event type and submit it soon.

mcdafydd · 2019-10-09T13:25:04Z

There were at least six valid pull request updated message strings that we would've wanted to accept, so I just ignored the one message we don't want for now.

yradsmikham · 2019-10-23T20:00:45Z

Can we get an update on this? Would love to see it merge soon! Thanks!

lkysow · 2019-10-25T20:50:43Z

runatlantis.io/docs/access-credentials.md

+- Create a Personal access token by following [https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops)
+- Label the password "atlantis"
+- The minimum scopes required for this token are:
+  - Code (Read)


Needs Code (Read & Write), I think for writing pr comments

lkysow · 2019-10-26T00:38:00Z

server/events/models/models.go

@@ -71,7 +74,20 @@ func NewRepo(vcsHostType VCSHostType, repoFullName string, cloneURL string, vcsU
 		return Repo{}, errors.New("cloneURL can't be empty")
 	}

-	if !strings.HasSuffix(cloneURL, ".git") {
+	if vcsHostType == AzureDevops {


This was failing for me because my repo had a - in it. The regex could be fixed but is this necessary? Can a malformed repo name get in here?

lkysow · 2019-10-26T00:59:02Z

server/events/event_parser.go

+		err = errors.New("lastMergeSourceCommit.commitID is null")
+		return pullModel, baseRepo, headRepo, err
+	}
+	url := pull.GetURL()


We need to iterate through the links and pull out the web url. THis URL is the json url. CAn be done later though

{ "resource": { "comment": { "_links": { "pullRequests": { "href": "https://dev.azure.com/lkysow/_apis/git/pullRequests/1" <<<< need this } } },

lkysow · 2019-10-26T01:00:40Z

server/locks_controller.go

+		project = ""
+	}
+	viewData.RepoOwner = owner
+	viewData.RepoProject = project


This isn't actually used in the UI right now.

lkysow

I think I can safely say that this is the best PR ever submitted to the Atlantis repo. Thanks for all your tireless work and bearing through the long review.

David McPike and others added 30 commits April 1, 2019 20:49

initial changes for azure devops support

fe25896

initial azuredevops client - copy from bitbucketcloud

ce380ec

continue adding skeleton for azure devops support

cfcf59d

merge from head

1946dc0

continue azure devops support

03bfefe

add project memeber to Repo struct, NewRepo, and tests

5a1eb59

update NewRepo calls with project argument

c5eb119

azure devops should ignore appending .git to clone url

08d47de

- switch to go-azuredevops package

a17e565

- remove ADHostname - add ADOrg and ADProject

add azure devops cmd flags

2c83238

Merge remote-tracking branch 'upstream/master' into azure-devops

262e31b

replace azuredevops secret for basic auth

811a2a6

change go-azuredevops package constraint location

40593c7

add project parameter to bitbucketcloud NewRepo()

b7ff1c2

initial commit of azuredevops client

a058a66

Updates to azuredevops_client.go

4bffa57

- remove unneeded NewAzureDevopsClient() code - change to using const refs - add GetPullRequest()

add Azure Devops pull getter support

4d11f80

Initial support for azure devops events

5cbcf25

- Pull request events - Work item comment events

Fixes repo fullName for Azure Devops

87e88e3

update readme

8c27c12

fix CreateStatus() if TargetURL is empty

10abea0

Merge remote-tracking branch 'upstream/master' into azure-devops

10f210c

Azure Devops support

cbb28ec

- Add bluemonday to strip HTML tags from comments - Rename all WorkItemEvent refs to WorkItemCommentedEvent - Rename pullEvent to event to match types - Minor comment clean-ups - Support multiple pull request links in a work item

Fixes steps that use cloneURL for Azure Devops

36a849d

Azure Devops support

bf1ccc4

- Fixes logic error in PullIsMergeable() - Replace GetRemoteURL() with GetWebURL() - returns what we want

Azure Devops fork updates

7f1fd23

- Remove project from NewRepo() calls - Add new function SplitAzureDevopsRepoFullName

Merge remote-tracking branch 'upstream/master' into azure-devops

dd9041f

update readme

5d7d3c0

mcdafydd and others added 2 commits October 2, 2019 13:01

Update runatlantis.io/docs/deployment.md

d64be55

Co-Authored-By: Luke Kysow <1034429+lkysow@users.noreply.github.com>

Resolve documentation comments

95234bf

Change Azure Devops from Work Item to PR comment

3b5a5b6

mcdafydd added 2 commits October 5, 2019 00:17

Complete migration to PR comment event

9424893

- Update docs - Remove work item commented event code

Update mocks

92e5126

mcdafydd added 2 commits October 9, 2019 08:07

Ignore PR updated change reviewer events

7c42914

Clean up log messages

a8685e0

Fix azuredevops PullIsApproved logic

822928c

lkysow reviewed Oct 25, 2019

View reviewed changes

lkysow reviewed Oct 26, 2019

View reviewed changes

lkysow approved these changes Oct 26, 2019

View reviewed changes

lkysow merged commit 774b6da into runatlantis:master Oct 26, 2019

lkysow mentioned this pull request Nov 12, 2019

Code Commit #483

Closed

amasover mentioned this pull request Mar 13, 2020

Azure DevOps - Atlantis autoplan is triggered again when pull request is approved #946

Closed

mcdafydd deleted the azure-devops branch March 21, 2020 13:45

jpreese mentioned this pull request Mar 23, 2020

Azure DevOps Service Hook Source? mcdafydd/atlantis#3

Closed

andyshinn mentioned this pull request May 1, 2020

Hide previous command comments #1022

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure Devops support #719

Azure Devops support #719

mcdafydd commented Jul 26, 2019

lkysow commented Oct 2, 2019

mcdafydd commented Oct 3, 2019

mcdafydd commented Oct 4, 2019 •

edited

Loading

lkysow commented Oct 4, 2019

mcdafydd commented Oct 5, 2019

lkysow commented Oct 7, 2019 •

edited

Loading

mcdafydd commented Oct 8, 2019

lkysow commented Oct 8, 2019

mcdafydd commented Oct 8, 2019

mcdafydd commented Oct 9, 2019

yradsmikham commented Oct 23, 2019

lkysow Oct 25, 2019

lkysow Oct 26, 2019

lkysow Oct 26, 2019

lkysow Oct 26, 2019

lkysow left a comment

Azure Devops support #719

Azure Devops support #719

Conversation

mcdafydd commented Jul 26, 2019

lkysow commented Oct 2, 2019

mcdafydd commented Oct 3, 2019

mcdafydd commented Oct 4, 2019 • edited Loading

lkysow commented Oct 4, 2019

mcdafydd commented Oct 5, 2019

lkysow commented Oct 7, 2019 • edited Loading

mcdafydd commented Oct 8, 2019

lkysow commented Oct 8, 2019

mcdafydd commented Oct 8, 2019

mcdafydd commented Oct 9, 2019

yradsmikham commented Oct 23, 2019

lkysow Oct 25, 2019

Choose a reason for hiding this comment

lkysow Oct 26, 2019

Choose a reason for hiding this comment

lkysow Oct 26, 2019

Choose a reason for hiding this comment

lkysow Oct 26, 2019

Choose a reason for hiding this comment

lkysow left a comment

Choose a reason for hiding this comment

mcdafydd commented Oct 4, 2019 •

edited

Loading

lkysow commented Oct 7, 2019 •

edited

Loading