[Proposal] Pod Injection Policy #254

jessfraz · 2017-01-05T21:13:21Z

Feature issue kubernetes/enhancements#162

pmorie · 2017-01-11T14:51:02Z

I like the expanded example, looking forward to reading more on the areas we discussed offline that need to be fleshed out.

derekwaynecarr · 2017-01-11T15:23:22Z

I like this concept. I ask that we annotate the resultant resource so we have a record that the user input was modified as a result of the ServiceInjectionPolicy. We do something similar with LimitRange and its a useful reminder to users about why the input they provided was modified. Thoughts?

pmorie · 2017-01-11T15:29:37Z

We do something similar with LimitRange and its a useful reminder to users about why the input they provided was modified. Thoughts?

+1 to that.

jessfraz · 2017-01-11T16:05:27Z

Will update today, sorry got distracted mid update yesterday!

…

On Wed, Jan 11, 2017, 07:29 Paul Morie ***@***.***> wrote: We do something similar with LimitRange and its a useful reminder to users about why the input they provided was modified. Thoughts? +1 to that. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#254 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABYNbKOV3PkZhGpjCOmKFnntmWfKJR_Cks5rRPVlgaJpZM4LcImi> .

derekwaynecarr · 2017-01-12T15:39:41Z

contributors/design-proposals/service-injection-policy.md

+      emptyDir: {}
+```
+
+Pod spec after admission controller:


this is confusing. i thought the SIP would modify resources with Kind: Pod and not any resource that embedded a pod template (whose list is unknowable).

yeah this seems reasonable will update

@derekwaynecarr Is correct.

However, that said, I think another useful example would be a SIP and a ReplicaSet with a pod template whose labels match it, and then the pod that was created, showing that the pod's spec was mutated by the SIP during admission.

duglin · 2017-01-12T15:52:08Z

contributors/design-proposals/service-injection-policy.md

+`Env`, `EnvFrom`, `Volumes`, and `VolumeMounts` apply to the container spec for
+all containers in the pod with the specified matching `LabelSelector`.
+
+*Why all modify all containers in a pod?*


s/Why all/Why/

duglin · 2017-01-12T16:55:54Z

contributors/design-proposals/service-injection-policy.md

+
+type ServiceInjectionPolicySpec struct {
+    Service       string # Reference to the service that is being injected
+    LabelSelector *unversioned.LabelSelector


For consistency with what I'm seeing on https://kubernetes.io/docs/user-guide/labels/ should this be "Selector" instead of "LabelSelector" ?

makes sense

pmorie · 2017-01-12T20:09:54Z

contributors/design-proposals/service-injection-policy.md

+The above described the behavior for explicitly defining what should be
+injected into a pod.
+
+But a helper has been discussed to automatically fill in the


I would expand on this a bit and give a more fleshed out example.

WDYT of s/helper/controller/?

Is this something that would be part of the core kubernetes, or an add-on? If it is part of core k8s we should formalize the api - e.g. annotation and controller behavior.

duglin · 2017-01-13T14:37:41Z

contributors/design-proposals/service-injection-policy.md

+spec:
+  replicas: 2
+  selector:
+    app: website


My k8s newbie-ness might be showing, but if the selector is "app: website" shouldn't there be a label down below with "app: website" ?

pmorie · 2017-01-18T19:18:16Z

contributors/design-proposals/service-injection-policy.md

+      emptyDir: {}
+```
+
+ReplicaSet after admission controller modified pod spec:


This should be a pod created from the replica set, rather than the replica set itself; non-pod resources won't be mutated.

pmorie · 2017-01-18T19:18:54Z

contributors/design-proposals/service-injection-policy.md

+list](https://groups.google.com/forum/#!topic/kubernetes-sig-node/gijxbYC7HT8).
+
+Other solutions include basing the container to inject based off
+matching it's name to another field in the `ServiceInjectionPolicy` spec, but


pmorie · 2017-01-18T19:20:09Z

contributors/design-proposals/service-injection-policy.md

+## Constraints and Assumptions
+
+1.  New mechanisms must be made to work with controllers such as deployments and
+    replicasets that create pods


I think some additional qualification is needed here, and what I had had in mind is that existing controllers that create generations of pods should be made to recreate pods when a new SIP is created that would alter them.

pmorie · 2017-01-18T19:20:33Z

contributors/design-proposals/service-injection-policy.md

+### ServiceInjectionPolicy API object
+
+```go
+type ServiceInjectionPolicy struct {


I think we should spell out:

Validation rules for SIP

Which fields are mutable, if any

pmorie · 2017-01-18T19:22:06Z

contributors/design-proposals/service-injection-policy.md

+
+### AdmissionControl Plugin: ServiceInjectionPolicy
+
+The **ServiceInjectionPolicy** plugin introspects all incoming pod requests and


requests for pod creation, since pods are (for the purposes of SIP) immutable

pmorie · 2017-01-18T19:22:25Z

contributors/design-proposals/service-injection-policy.md

+#### Behavior
+
+This will modify the containers in the pod spec. The supported changes to
+`Env`, `EnvFrom`, `Volumes`, and `VolumeMounts` apply to the container spec for


Volumes are pod spec, not container spec :)

pmorie · 2017-01-18T19:23:10Z

contributors/design-proposals/service-injection-policy.md

+this would not scale well and would cause annoyance with configuration
+management.
+
+The resultant modified spec will be annotated to show that it was modified by


I think in this case it's the entire pod (since the spec alone does not have metadata); we might want to track down whether there are existing annotations used to carry this information.

pmorie · 2017-01-18T21:12:33Z

contributors/design-proposals/service-injection-policy.md

+
+### AdmissionControl Plugin: ServiceInjectionPolicy
+
+The **ServiceInjectionPolicy** plugin introspects all incoming pod creation


We should define the order of precedence when you have multiple SIPs. Seems like it should be based on creation time, oldest applied first.

Ug. Good point. The issue with creation time is that the order can't be easily adjusted. Seems like a good default, but we may need to support something more tunable eventually.

IMHO, we can launch a beta that does a consistent order to help with debugging and stipulate that all SIPs in a namespace must be order agnostic. It might be worth mentioning that we could do some conflict detection using merge keys.

This starts to get into similar areas as kubectl apply - e.g. multiple writers to the same object with potentially conflicting writes

pmorie · 2017-01-18T21:13:02Z

contributors/design-proposals/service-injection-policy.md

+resources you need. This would be similar to the functionality ConfigMaps and
+Secrets.
+
+## Examples


As @duglin requested, an example that shows consuming secrets/configmaps would be great.

pmorie · 2017-01-18T21:16:34Z

contributors/design-proposals/service-injection-policy.md

+
+type ServiceInjectionPolicySpec struct {
+    // Service is a reference to the service being injected (optional, immutable).
+    Service       string


Based on the service-catalog SIG discussion, I think that we can take this field out for now.

@derekwaynecarr we chatted about some use-cases you had related to resource consumption -- are those solidified at all? We are trying to determine whether this is really a generic injection policy and not intrinsically related to services.

pmorie · 2017-01-20T17:20:20Z

contributors/design-proposals/service-injection-policy.md

+## Proposed Changes
+
+### ServiceInjectionPolicy API object
+


Things to add:

Which API group is this in? I have been told by a couple API machinery peeps that extensions is not a great place to continue adding resources. Should we have a new API group for this? cc @deads2k

Is this proposal describing a resource we think is alpha, beta, or GA?

IMO what we are describing noew is alpha

Should we have a new API group for this? cc @deads2k

Yeah. The cost for choosing a bad group the same as putting it in extensions and the version would be wrong, so a separate group would always be preferred. This came for storageclass.

I'd really like to see this as a separate API server and avoid adding new types to the kube core. An alpha API summarized by an alpha summarizer seems very reasonable. The initializers proposal would cover the admission: #132

@deads2k Do you know if initializers will be ready in 1.6?

They will miss 1.6, but the generic api server is usable now and we'll be finishing its split in a couple weeks. If we separate the API servers, we'll be able to have the desired API layout. Knowing that we want to move to initializers, a temporary admission plugin can be built to load the types you need. The additional complication is minimal and it positions the API well going forward.

Just to alleviate concerns about difficulty, here's a pull that creates a complete API server for running a new group: kubernetes/kubernetes#40803 . It is compatible with kubectl: kubectl get serviceinjection

Relying on initializers would mean that we could not have a working prototype until 4 months from now. As I understand it, that is not an acceptable timeline. What alternatives can you suggest to release something on 1.6?

I think a temporary admission plugin:

a temporary admission plugin can be built to load the types you need

Relying on initializers would mean that we could not have a working prototype until 4 months from now. As I understand it, that is not an acceptable timeline. What alternatives can you suggest to release something on 1.6?

A temporary admission plugin can be built with a client that list/watches the resources you need.

pmorie · 2017-01-20T18:34:08Z

contributors/design-proposals/service-injection-policy.md

+
+### Loose coupling between services and their consumers
+
+Using a Service Injection Policy allows pod authors to not have to explicitly


Why is loose coupling desireable?

pmorie · 2017-01-20T18:37:53Z

contributors/design-proposals/service-injection-policy.md

+    Spec ServiceInjectionPolicySpec
+}
+
+type ServiceInjectionPolicySpec struct {


I'm of the mind that we should make this immutable. Mutable fields introduce a ton of corner cases into any piece of the system that deals with this resource. As an example, if we make the resource mutable, does that mean we need to ripple updates to the SIP spec in a service catalog binding into the created SIP? I would rather not deal with that.

Also, it's possible to make an immutable field mutable and impossible to make a mutable field immutable from a backward-compatibility standpoint, so we could start with immutable and make fields mutable in the future if it makes sense to do so.

makes sense

EQ: is this just a documentation thing or is there some kind of tag we add to the resource/properties that allows for our api server to handle/check this for us?

pmorie · 2017-01-20T18:38:44Z

contributors/design-proposals/service-injection-policy.md

+    // VolumeMounts defines the collection of VolumeMount to inject (optional, mutable).
+    VolumeMounts  []VolumeMount
+}
+```


I would do a subsection just devoted to validations

pwittrock · 2017-01-20T18:39:57Z

contributors/design-proposals/service-injection-policy.md

+
+## Constraints and Assumptions
+
+1.  New mechanisms must be made to work with controllers such as deployments and


Existing controllers that create pods should recreate their pods when a new Service Injection Policy is added that would effect them.

I am not sure this is always true. Imagine a new SIP is created that targets all Pods, this would then restart all Pods in the namespace. The impact could be limited by adding a PodDisruptionBudget, but that part of the design would need to be further discussed. May be we should defer this and leave it as a possible future work.

pwittrock · 2017-01-20T18:41:06Z

contributors/design-proposals/service-injection-policy.md

+
+## Use Cases
+
+1.  As a user, I want to be able to describe a way that pods should be injected


This sentence is a mouthful ;)

pwittrock · 2017-01-20T18:42:00Z

contributors/design-proposals/service-injection-policy.md

+
+### Loose coupling between services and their consumers
+
+Using a Service Injection Policy allows pod authors to not have to explicitly


s/pod/pod or podtemplate/

pmorie · 2017-02-22T22:04:13Z

I totally get the ServiceBroker idea. The part of this that rubs me wrong is the implicitness of it. If I want to use my CloudSpanner instance, I should really say so. It shouldn't just happen magically.

If you don't want to use it, it's entirely optional. If you don't use it, you'll have to reference everything in every pod spec you want to consume it in.

Signed-off-by: Jess Frazelle <acidburn@google.com>

jessfraz · 2017-02-22T22:17:31Z

everything has been updated PTAL, thanks!

pwittrock · 2017-02-23T00:23:33Z

@thockin

I totally get the ServiceBroker idea. The part of this that rubs me wrong is the implicitness of it. If I want to use my CloudSpanner instance, I should really say so. It shouldn't just happen magically.

+1 #176 Is a proposal to offer something more along these lines.

pmorie · 2017-02-23T06:10:52Z

@thockin

I totally get the ServiceBroker idea. The part of this that rubs me wrong is the implicitness of it. If I want to use my CloudSpanner instance, I should really say so. It shouldn't just happen magically.

In the pre-kubernetes versions of OpenShift, we had a feature for our image-analog that was a more limited version of PIP. We found that people really like not having to worry about writing links explicitly and just having them injected. As Brian has articulated very well, having implicit mechanisms reduces the complexity of descriptors users write a great deal.

I swear that we have had this discussion before and I won you over. I cannot find it now.

Going back to your concern:

If I want to use my CloudSpanner instance, I should really say so. It shouldn't just happen magically.

We can have an escape hatch annotation that you can place on a pod to exempt it from being acted on by PIP. That way, if you have a workload you absolutely positively do not want to be subject to this type of mutation, you can express that.

I do and don't agree with the spooky at a distance analogy. Yes, it is action at a distance, but so is everything any admission controller does. For example, the service account token volume is basically a subset of this idea.

@bgrant0607 @smarterclayton @thockin I would really like to have this proposal accepted so that we can publish something alpha in 1.6. Does adding the escape hatch annotation make it more palatable? This is a very important part of the story for service catalog and we have invested quite a bit of time on the concept. I would like to have folks begin using this at 1.6 so we can incorporate feedback into a more mature version in the 1.7 timeframe.

thockin · 2017-02-23T06:22:44Z

I withdraw any objection. I still find it weird but I ack the design and I don't have the bandwidth to rebut much further :)

For example, the service account token volume is basically a subset of

this idea. I don't see it as similar at all. ServiceAccount token is injected into the pod spec. It's obvious, upon inspection, why that volume is mounted at runtime. If I understand correctly this is not mutating the pod specs, just the runtime behavior. In my ideal world, I would be able to see this information in the pod spec. But see first sentence. :)

…

On Wed, Feb 22, 2017 at 10:10 PM, Paul Morie ***@***.***> wrote: @thockin <https://github.com/thockin> I totally get the ServiceBroker idea. The part of this that rubs me wrong is the implicitness of it. If I want to use my CloudSpanner instance, I should really say so. It shouldn't just happen magically. In the pre-kubernetes versions of OpenShift, we had a feature for our image-analog that was a more limited version of PIP. We found that people *really* like not having to worry about writing links explicitly and just having them injected. As Brian has articulated very well, having implicit mechanisms reduces the complexity of descriptors users write a great deal. I swear that we have had this discussion before and I won you over. I cannot find it now. Going back to your concern: If I want to use my CloudSpanner instance, I should really say so. It shouldn't just happen magically. We can have an escape hatch annotation that you can place on a pod to exempt it from being acted on by PIP. That way, if you have a workload you absolutely positively do not want to be subject to this type of mutation, you can express that. I do and don't agree with the spooky at a distance analogy. Yes, it is action at a distance, but so is everything any admission controller does. For example, the service account token volume is basically a subset of this idea. @bgrant0607 <https://github.com/bgrant0607> @smarterclayton <https://github.com/smarterclayton> @thockin <https://github.com/thockin> I would really like to have this proposal accepted so that we can publish something alpha in 1.6. Does adding the escape hatch annotation make it more palatable? This is a very important part of the story for service catalog and we have invested quite a bit of time on the concept. I would like to have folks begin using this at 1.6 so we can incorporate feedback into a more mature version in the 1.7 timeframe. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#254 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVIHZ37_DqBrkvIiS51gCkifQtHfwks5rfSLwgaJpZM4LcImi> .

derekwaynecarr · 2017-02-23T06:29:33Z

@thockin - this does mutate the resultant pod spec upon creation and records via annotation that it did it.

pmorie · 2017-02-23T06:34:12Z

If I understand correctly this is not mutating the pod specs,
just the runtime behavior.

It does mutate - see the examples section. I am going to merge - we can update the proposal to refine articulation of the motivation if necessary.

Thanks a lot for this work @jessfraz !

erictune · 2017-02-23T16:33:27Z

Please don't put it in apps. apps is already at v1beta1. Adding a v1alpha1 is going backwards.

…

On Wed, Feb 22, 2017 at 11:25 PM, Brian Grant ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In contributors/design-proposals/pod-injection-policy.md <#254 (comment)>: > +1. Database Administrator provisions a MySQL service for their cluster. +2. Database Administrator creates secrets for the cluster containing the + database name, username, and password. +3. Database Administrator creates a `PodInjectionPolicy` defining the database + port as an enviornment variable, as well as the secrets. See + [Examples](#examples) below for various examples. +4. Developer of an application can now label their service with the specified + `Selector` the Database Administrator tells them, and consume the MySQL + database without needing to know any of the details from step 2 and 3. + +## Proposed Changes + +### PodInjectionPolicy API object + +This resource is alpha. The policy itself is immutable. The API group will be +added to `apps` and the version is `v1alpha1`. I don't have a better suggestion right now, but almost certainly I'm going to ask this to be moved to a different group, since I envision it at a different layer of the system than the other APIs in the apps group. https://docs.google.com/document/d/1XkjVm4bOeiVkj- Xt1LgoGiqWsBfNozJ51dyI-ljzt1o/edit — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#254 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHuudhFhdMrU4mwTIyYpXv0azQxK95xMks5rfTRSgaJpZM4LcImi> .

pmorie · 2017-02-23T16:38:38Z

@erictune How about an 'injection' group? I get that we don't want a ton of single-resource groups, but when there's no fitting group, it seems like we should create a new one.

erictune · 2017-02-23T16:52:37Z

Some thoughts on naming:

The word "Policy" makes me think about something that says what you cannot do. This is more about setting values you don't know you need to set. "Environment" and "Defaults" seem like more appropriate terms to use.
"PodInjection" sounds like it injects pods, which it doesn't.
What about "PodDefaults" or "PodEnvironment".

jessfraz · 2017-02-23T17:09:30Z

Defaults kinda makes it seem like it would be a "default" tho

bgrant0607 · 2017-02-23T17:57:27Z

I agree with @erictune that ServiceInjection made more sense than PodInjection. I called this configurable defaulting in kubernetes/kubernetes#17097 -- it's selector-scoped default settings for pods. How about PodSettings, PodProperties, PodAttributes, or the like?

As for group, I agree with @pmorie that we do already have too many groups. Why does ImagePolicy have its own group? I asked PodDisruptionBudget to use a "policy" group with the expectation that other policies would go there.

We could create a "settings" group for APIs like PIP, LimitRange, and MetadataPolicy that modify other resources in a straightforward way (as opposed to autoscaling, which is reactive/predictive).

jessfraz · 2017-02-23T22:08:51Z

so should we vote?

@pmorie

Automatic merge from submit-queue (batch tested with PRs 41931, 39821, 41841, 42197, 42195) Admission Controller: Add Pod Preset Based off the proposal in kubernetes/community#254 cc @pmorie @pwittrock TODO: - [ ] tests **What this PR does / why we need it**: Implements the Pod Injection Policy admission controller **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes # **Special notes for your reviewer**: **Release note**: ```release-note Added new Api `PodPreset` to enable defining cross-cutting injection of Volumes and Environment into Pods. ```

tamalsaha · 2017-03-07T01:07:01Z

@jessfraz , is there any chance that this will also allow adding side car container on the fly in future?

pmorie · 2017-03-07T02:34:45Z

@tamalsaha containers and init containers are definitely on the table

tamalsaha · 2017-03-07T02:41:43Z

@pmorie good to know. Any chance that will make in 1.6? Also will the extensible admission controller feature make in 1.6?

pwittrock · 2017-03-07T04:50:36Z

@tamalsaha Definitely not in 1.6.

[Proposal] Pod Injection Policy

…s-secondary-for-1.7-release Propose Caleb Miles as member of 1.7 release team

[Proposal] Pod Injection Policy

…ubernetes#254) starting from January 1st 2020 I move to another project

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 5, 2017

jessfraz mentioned this pull request Jan 6, 2017

[Service Catalog]: Pod Preset kubernetes/enhancements#162

Closed

23 tasks

pmorie self-assigned this Jan 10, 2017

jessfraz force-pushed the sip branch from a9b411d to bc444f4 Compare January 10, 2017 17:47

jessfraz force-pushed the sip branch 2 times, most recently from b348ed2 to 5fe1296 Compare January 11, 2017 21:24

derekwaynecarr reviewed Jan 12, 2017

View reviewed changes

duglin reviewed Jan 12, 2017

View reviewed changes

pmorie reviewed Jan 12, 2017

View reviewed changes

duglin reviewed Jan 13, 2017

View reviewed changes

duglin mentioned this pull request Jan 16, 2017

Add support for Pod-Preset - aka Pod Injection Policy kubernetes-retired/service-catalog#266

Closed

jessfraz force-pushed the sip branch 2 times, most recently from cf19bf3 to 1410a81 Compare January 18, 2017 19:13

pmorie reviewed Jan 18, 2017

View reviewed changes

jessfraz force-pushed the sip branch 3 times, most recently from 73537f2 to 653908b Compare January 18, 2017 19:49

pmorie reviewed Jan 18, 2017

View reviewed changes

jessfraz force-pushed the sip branch 3 times, most recently from 034243d to cc7d5bc Compare January 19, 2017 22:18

pmorie reviewed Jan 20, 2017

View reviewed changes

pwittrock reviewed Jan 20, 2017

View reviewed changes

updates to pod injection policy proposal

a612a22

Signed-off-by: Jess Frazelle <acidburn@google.com>

jessfraz force-pushed the sip branch from 4c1445c to a612a22 Compare February 22, 2017 22:17

jessfraz mentioned this pull request Feb 22, 2017

Admission Controller: Add Pod Preset kubernetes/kubernetes#41931

Merged

1 task

pmorie merged commit 2b56d8b into kubernetes:master Feb 23, 2017

jessfraz deleted the sip branch February 23, 2017 06:34

jessfraz mentioned this pull request Feb 23, 2017

rename pod injection #396

Merged

andraxylia mentioned this pull request Mar 30, 2017

Admission Controller: Enhance PodPreset to inject containers and init containers kubernetes/kubernetes#43874

Closed

ruebenramirez pushed a commit to ruebenramirez/community that referenced this pull request Apr 22, 2017

Merge pull request kubernetes#254 from jessfraz/sip

b79ba4e

[Proposal] Pod Injection Policy

shyamjvs pushed a commit to shyamjvs/community that referenced this pull request Sep 22, 2017

Merge pull request kubernetes#254 from calebamiles/propose-calebamile…

51560b3

…s-secondary-for-1.7-release Propose Caleb Miles as member of 1.7 release team

jorgemoralespou mentioned this pull request Oct 2, 2018

R&D - component configuration semantics redhat-developer/odo#675

Closed

MadhavJivrajani pushed a commit to MadhavJivrajani/community that referenced this pull request Nov 30, 2021

Merge pull request kubernetes#254 from jessfraz/sip

b970ff7

[Proposal] Pod Injection Policy

danehans pushed a commit to danehans/community that referenced this pull request Jul 18, 2023

remove myself (vadimeisenbergibm) from Istio developers/maintainers (k…

6233ae2

…ubernetes#254) starting from January 1st 2020 I move to another project


		### AdmissionControl Plugin: ServiceInjectionPolicy

		The ServiceInjectionPolicy plugin introspects all incoming pod requests and


		### Loose coupling between services and their consumers

		Using a Service Injection Policy allows pod authors to not have to explicitly


		## Constraints and Assumptions

		1. New mechanisms must be made to work with controllers such as deployments and


		## Use Cases

		1. As a user, I want to be able to describe a way that pods should be injected

[Proposal] Pod Injection Policy #254

[Proposal] Pod Injection Policy #254

Conversation

jessfraz commented Jan 5, 2017 • edited Loading

pmorie commented Jan 11, 2017

derekwaynecarr commented Jan 11, 2017

pmorie commented Jan 11, 2017

jessfraz commented Jan 11, 2017 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmorie commented Feb 22, 2017

jessfraz commented Feb 22, 2017

pwittrock commented Feb 23, 2017

pmorie commented Feb 23, 2017

thockin commented Feb 23, 2017 via email

derekwaynecarr commented Feb 23, 2017

pmorie commented Feb 23, 2017

erictune commented Feb 23, 2017 via email

pmorie commented Feb 23, 2017

erictune commented Feb 23, 2017

jessfraz commented Feb 23, 2017

bgrant0607 commented Feb 23, 2017

jessfraz commented Feb 23, 2017

tamalsaha commented Mar 7, 2017

pmorie commented Mar 7, 2017

tamalsaha commented Mar 7, 2017

pwittrock commented Mar 7, 2017

jessfraz commented Jan 5, 2017 •

edited

Loading