Skip to content

Commit

Permalink
Add simulation API (#725)
Browse files Browse the repository at this point in the history
This adds a new POST route api/simulate/:owner/:repo/:number. This route
requires a GitHub bearer token with access to read the the underlying
pull request.

When called, policy-bot will re-evaluate the policy of the referenced pr
and return a simulated result back to the caller. The result does NOT
get written back to the pull request as a STATUS.

A request body can be included specifying a number of options which can
modify how the simulation runs. These should be relatively easy to
extend with additional options in the future.

Today this supports:

* Simulating new comments.
* Simulating new reviews.
* Simulating ignoring existing reviews and comments.
* Simulating a change of the base branch.

The route can also be used without any options to trigger policy-bot to
re-evaluate the pull request as is.

The returned simulated result does not currently contain all of the
fields of the base result but it should be easy to extend as needed.
  • Loading branch information
atatkin authored Mar 14, 2024
1 parent 19d0813 commit a793ea3
Show file tree
Hide file tree
Showing 8 changed files with 969 additions and 6 deletions.
79 changes: 79 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ UI to view the detailed approval status of any pull request.
- [Approval Policies](#approval-policies)
- [Disapproval Policy](#disapproval-policy)
- [Testing and Debugging Policies](#testing-and-debugging-policies)
- [Simulation API](#simulation-api)
- [Caveats and Notes](#caveats-and-notes)
- [Disapproval is Disabled by Default](#disapproval-is-disabled-by-default)
- [Interactions with GitHub Reviews](#interactions-with-github-reviews)
Expand Down Expand Up @@ -549,6 +550,84 @@ $ rcode=$(curl https://policybot.domain/api/validate -XPUT -T path/to/policy.yml
$ if [[ "${rcode}" -gt 299 ]]; then cat /tmp/response && exit 1; fi
```

#### Simulation API

It can be useful to simulate how Policy Bot would evaluate a pull request if certain conditions were changed. For example: adding a review from a specific user or group, or adjusting the base branch.

An API endpoint exists at `api/simulate/:org/:repo/:prNumber` to simiulate the result of a pull request. Simulations using this endpoint will NOT write the result back to the pull request status check and will instead return the result.

This API requires a GitHub token be passed as a bearer token. The token must have the ability to read the pull request the simulation is being run against.

The API can be used as such:

```sh
$ curl https://policybot.domain/api/simulate/:org/:repo/:number -H 'authorization: Bearer <token>' -H 'content-type: application/json' -X POST -d '<data>'
```

Currently the data payload can be configured with a few options:

Ignore any comments from specific users, team members, org members or with specific permissions
```json
{
"ignore_comments":{
"users":["ignored-user"],
"teams":["ignored-team"],
"organizations":["ignored-org"],
"permissions":["admin"]
}
}
```

Ignore any reviews from specific users, team members, org members or with specific permissions
```json
{
"ignore_reviews":{
"users":["ignored-user"],
"teams":["ignored-team"],
"organizations":["ignored-org"],
"permissions":["admin"]
}
}
```

Simulate the pull request as if the following comments from the following users had also been added
```json
{
"add_comments":[
{
"author":"not-ignored-user",
"body":":+1:",
"created_at": "2020-11-30T14:20:28.000+07:00",
"last_edited_at": "2020-11-30T14:20:28.000+07:00"
}
]
}
```

Simulate the pull request as if the following reviews from the following users had also been added
```json
{
"add_reviews":[
{
"author":"not-ignored-user",
"state": "approved",
"body": "test approved review",
"created_at": "2020-11-30T14:20:28.000+07:00",
"last_edited_at": "2020-11-30T14:20:28.000+07:00"
}
]
}
```

Choose a different base branch when simulating the pull request evaluation
```json
{
"base_branch": "test-branch"
}
```

The above can be combined to form more complex simulations. If a Simulation is run without any data being passed, the pull request is evaluated as is.

### Caveats and Notes

There are several additional behaviors that follow from the rules above that
Expand Down
12 changes: 6 additions & 6 deletions policy/common/actor.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,17 +26,17 @@ import (
// team and organization memberships. The set of allowed actors is the union of
// all conditions in this structure.
type Actors struct {
Users []string `yaml:"users"`
Teams []string `yaml:"teams"`
Organizations []string `yaml:"organizations"`
Users []string `yaml:"users" json:"users"`
Teams []string `yaml:"teams" json:"teams"`
Organizations []string `yaml:"organizations" json:"organizations"`

// Deprecated: use Permissions with "admin" or "write"
Admins bool `yaml:"admins"`
WriteCollaborators bool `yaml:"write_collaborators"`
Admins bool `yaml:"admins" json:"-"`
WriteCollaborators bool `yaml:"write_collaborators" json:"-"`

// A list of GitHub collaborator permissions that are allowed. Values may
// be any of "admin", "maintain", "write", "triage", and "read".
Permissions []pull.Permission
Permissions []pull.Permission `yaml:"permissions" json:"permissions"`
}

// IsEmpty returns true if no conditions for actors are defined.
Expand Down
132 changes: 132 additions & 0 deletions policy/simulated/context.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
// Copyright 2018 Palantir Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package simulated

import (
"context"

"github.com/palantir/policy-bot/pull"
)

type Context struct {
pull.Context
ctx context.Context
options Options
}

func NewContext(ctx context.Context, pullContext pull.Context, options Options) *Context {
return &Context{Context: pullContext, options: options}
}

func (c *Context) Comments() ([]*pull.Comment, error) {
comments, err := c.Context.Comments()
if err != nil {
return nil, err
}

comments, err = c.filterIgnoredComments(c.Context, comments)
if err != nil {
return nil, err
}

comments = c.addApprovalComment(comments)
return comments, nil
}

func (c *Context) filterIgnoredComments(prCtx pull.Context, comments []*pull.Comment) ([]*pull.Comment, error) {
if c.options.IgnoreComments == nil {
return comments, nil
}

var filteredComments []*pull.Comment
for _, comment := range comments {
isActor, err := c.options.IgnoreComments.IsActor(c.ctx, prCtx, comment.Author)
if err != nil {
return nil, err
}

if isActor {
continue
}

filteredComments = append(filteredComments, comment)
}

return filteredComments, nil
}

func (c *Context) addApprovalComment(comments []*pull.Comment) []*pull.Comment {
var commentsToAdd []*pull.Comment
for _, comment := range c.options.AddComments {
commentsToAdd = append(commentsToAdd, comment.toPullComment())
}

return append(comments, commentsToAdd...)
}

func (c *Context) Reviews() ([]*pull.Review, error) {
reviews, err := c.Context.Reviews()
if err != nil {
return nil, err
}

reviews, err = c.filterIgnoredReviews(c.Context, reviews)
if err != nil {
return nil, err
}

reviews = c.addApprovalReview(reviews)
return reviews, nil
}

func (c *Context) filterIgnoredReviews(prCtx pull.Context, reviews []*pull.Review) ([]*pull.Review, error) {
if c.options.IgnoreReviews == nil {
return reviews, nil
}

var filteredReviews []*pull.Review
for _, review := range reviews {
isActor, err := c.options.IgnoreReviews.IsActor(c.ctx, prCtx, review.Author)
if err != nil {
return nil, err
}

if isActor {
continue
}

filteredReviews = append(filteredReviews, review)
}

return filteredReviews, nil
}

func (c *Context) addApprovalReview(reviews []*pull.Review) []*pull.Review {
var reviewsToAdd []*pull.Review
for _, review := range c.options.AddReviews {
reviewsToAdd = append(reviewsToAdd, review.toPullReview())
}

return append(reviews, reviewsToAdd...)
}

func (c *Context) Branches() (string, string) {
base, head := c.Context.Branches()
if c.options.BaseBranch != "" {
return c.options.BaseBranch, head
}

return base, head
}
Loading

0 comments on commit a793ea3

Please sign in to comment.