Notification resources clutter up pipeline UI

[vito]

Feature Request

What challenge are you facing?

Configuring a Slack notification resource or other similar resources that "augment" the pipeline is verbose and clutters up the pipeline UI with resources that don't actually influence pipeline semantics.

A Modest Proposal

We should find a pattern for using these resources that reduces the amount of copy/paste pipeline code, and also allows us to express them more meaningfully (or just hide them) from the pipeline UI.

The initial focus of this is in the Slack notification resource, and other fire-and-forget resources that have no bearing to the build plan they're tacked on to.

Maybe we could support something like this:

decorators:
- name: slack-notify
  on_success: 
    put: slack
    params:
      channel: '#team'
      text: |
        [$BUILD_PIPELINE_NAME/BUILD_JOB_NAME #$BUILD_NAME]($ATC_EXTERNAL_URL/builds/$BUILD_ID) succeeded!

  on_failure:
    put: slack
    params:
      channel: '#team'
      text: |
        [$BUILD_PIPELINE_NAME/BUILD_JOB_NAME #$BUILD_NAME]($ATC_EXTERNAL_URL/builds/$BUILD_ID) failed. :'(

jobs:
- name: job-a
  decorators: [slack-notify]
  plan: [...]

- name: job-b
  decorators: [slack-notify]
  plan: [...]

[ggeorgiev]

That's kind of similar to what I was suggesting in #1016. Of course, this seems way less flexible. Any reason why not to have - say a decorate step that can be triggered at any place in the plan?

[pms1969]

I like the syntax for this... Would these particular decorators then NOT appear in the UI?

[cwinkler]

Looks good to me as long as it can be shown/hidden in the UI.

[vito]

@ggeorgiev I don't see how this is way less flexible - it's a different solution to a different set of problems. This removes the duplication and intermixing of concerns between "augmentations" like notifications and the real meat of the build plan, allowing us to treat them differently. #1016 is calling out a need for reuseable actions that don't quite fit in to the resources model, which I sympathize with, but it doesn't seem to compete with this issue - if anything, it may be used with it later if we do find a pattern for it.

@cwinkler @pms1969 Yep, now that Concourse will know that they're something "special" compared to regular steps in the plan, we can represent them differently in the UI.

[ggeorgiev]

@vito I might be misreading the snipped but it seems to me that decorators are an item in the job out of the plan. This means that concourse will have to decide to run them somehow as part of the plan. This "somehow" is going to be very specific, which will not allow for grouping them in aggregates for example. I have several other scenarios of "decoration" resources that I was hoping to be able to hide that are not going to be able to with the proposed.

For example, I share a lot of images between pipelines. To avoid building the same image multiple times I have this temporary (completely implementation detail) cache image resource. I am not going to be able to hide it with the suggested because it is not a step that is executed at on_success or on_failure.

[petergtz]

@vito Since you mentioned in #1583 that this issue here is about notifications, I'd thought I would link a little utility called Overbook here. It's really just an experiment within this notification space and doesn't solve the UI part, but makes the yaml less verbose, and I thought you might be interested to have a look. We're using it for https://flintstone.ci.cf-app.com/, but I'd be happy to ditch it in favor of something better in the future.

[vito]

@petergtz Great name.

At this point I'm considering having notifications be a first-class concept, and designing it so that it's a) as generic as resources, b) doesn't clutter the web UI, and c) doesn't clutter the pipeline config. Nothing concrete here yet though.

[petergtz]

@vito I've started thinking about this, but would like to verify some of my assumption about the use cases with you:

1. On any broken build, it should notify the person responsible for the breakage.
   In the message there should be a link to the broken Concourse build and ideally a hint, such as a git commit, that caused the breakage. How it identifies root cause, and the person (e.g. the committer) and which medium it uses (Slack, email, etc.) is to be defined.

2. It should be possible to easily define "announcements".
   An announcement could be something like a final release or anything. I.e., here, we don't address a specific person, but a whole group of people who own the pipeline, or might be interested in such updates.

Obviously, we would like to have the above to be configurable (to satisfy a) in your comment), but it should still be trivial to set up common use cases, such as "On broken builds, send a Slack message to a configured Slack channel and tag the committer who made the broken commit."

Does this make sense? Did you have other use cases in mind?

---

(Implementation-wise, it's certainly not trivial to implement this, especially the identification of the person causing the breakage. Because it could be a commit that builds a new release and that release breaks only in the tests job. How to map that back to the committer? I'm wondering if it makes sense to introduce a new optional field that goes along with versions. For git resources, this comes already for free as committer. For other resources it would have to be introduced somehow.)

[JohannesRudolph]

I think the decorators are a great proposal. however, I must admit that I got so frustrated with our current pipeline complexity along the matrix of these dimensions:

• 3 different branches for prod, dev and features
• slightly different deploy steps / gateways for each branch
• different params for artifact drops, esp. object expiration
• different notification policies (e.g. no slack for failed feature branch builds)

So I finally took the plunge this week and started using ejs templates to generate our pipeline. YAML anchors and references got me very far (800 lines of yaml, to generate approx. 2k lines pipeline) but now we're sitting at around 600 lines of a very readable ejs template that's easy to configure and a lot less pain to maintain while modeling our requirements much more flexibly than we ever could using plain yml. Hope I'll get a chance to wrap it up in a blog post soon.

[mattlqx]

I'm hoping to use Concourse to replace my current CI and one of my team's current complaints about what we have implemented with Chef Automate is how difficult it is to find build information. There's some clicking involved but there's a lot of superfluous CI stuff mixed in with the actual build output. With a plain jane Concourse pipeline, the UI is pretty usable and easy to find the output, but once I want to start using Concourse resources to update statuses, on Github PR or Slack or whatever.

It does indeed start getting cluttered fast. With just the PR updates for build status, it quickly drowns out the actual content that devs would be looking for. The redundant display of the get after put doesn't help either. Here's a collapsed example of what I'm looking at right now.

Any progress on being able to take these resources out-of-band from the "real work" of the pipeline?

[vito]

Closing this to focus on concourse/rfcs#10.