-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
An aspect system for the construct tree #1136
Comments
cc @RomainMuller @eladb @moofish32 for opinions |
[+ @clareliguori @sam-goodwin] A few things come to mind: I love the game-system analogy. I think it's apt. You mention that the CDK is a bit different since it's organized as a hierarchy, but game worlds are also organized as a hierarchy! Anyway - totally relevant. I am wondering how this interplays with what we called "Aspects" in #282. I think there's a connection but there are also a few use cases there that I am not sure this addresses? Should we merge those up? My main feedback about the API design is that I think there's a use case to apply an aspect without explicit support from the target (i.e. I'd like to apply an aspect that traverses the tree and calculates the cost of stack). So maybe instead of I wonder if this removes the need to maintain a list of
Laziness: I like to think about this as "attach" instead of "apply", and definitely vote for the declarative model (any order-dependent model will not hold water in my mind). The reactive solution sounds shaky... I think a phased approach would work better - perhaps we devise a phase model where aspects say in which phase they expect to be applied (currently we have one phase: "synthesis", but maybe we needed two). Maybe that's where this is connected to #282... |
The advantage of the reactive method is that it will be invariant against ordering even if the visitors were to add new children. Any non-reactive system wouldn't be |
First, from the tags aspect I think this makes a lot more sense. I've really struggled with tags without aspects. For some reason as I designed tags I thought we wanted to avoid putting more intelligence on I do think we are going to give up some things.
I agree with @eladb, that the reactive model is shaky right now. For example, removing a tag is quite difficult with the reactive model we have today. Now I might also argue that use case should not happen very often, but I just don't have any data for it either way. The concept of phases does make sense to me, but I don't have any good feel for what we will have besides The other part of tags that seems to get ugly in code is what to do about initialization. Once we allow two entry points we have to resolve conflicts. For example, if a parent has a tag added after the L1 is initialized, which tag takes precedence? The reverse is also possible. L1s also support a few input formats, I haven't seen a situation where CDK alters the L1 types, but life would be easier for CDK if tag inputs were standardized. ASG will likely be a snow flake, but that's fine for just one special case. I'm overall in favor of this design, but I definitely expect a few users that will disagree. There are definitely plenty of valuable use cases, including pre-deploy security checks that have been discussed in other threads and can be addressed here. |
I think the phased approach might be a good compromise. I realize it won’t solve the problem in the general case but I think it should suffice. ——- I was also wondering about CloudFormation rendering- is this really an aspect? I am not sure! |
I think any behavior that we want to apply to a swath of constructs (but only the ones that support it) is an aspect. Or end goal was to be cloud agnostic, so we should be able to mix AWS and GCP constructs in one tree, for example. What is the fact that some constructs render to CFN and others don't, if not an aspect? But more general than CFN vs GCP, some constructs "synthesize themselves" and others don't. An aspect (command) could totally be "synthesize yourself to this directory". |
If we want to do a definite aspect ordering, I feel it should be top-down so that aspects that apply in-place mutations (such as tags) applied lower in the tree override higher-level aspects without extra work. |
But synthesizing an output is not "applying a behavior to a construct". It actually expects a result. In a sense, it's a synchronous operation and not just "attaching" a behavior to the tree. I feel those are things are not the same.
Isn't this a per-aspect behavior (and I even think that for tagging we should support both)? |
I guess if we don't do "constructs have aspect objects" but feature detecting by means of "constructs have public members that we test against", this all works happily and fine in JavaScript, but how will this translate to JSII languages?
For the former, can we feature-detect public methods in Java from JavaScript? For the latter, can someone in Java feature-detect a public method in JavaScript? I think the answer to say "both of these can only be written in JavaScript" is not good enough. |
Yes, I suppose you're right. We do need to decide on a visiting order though, top/down or bottom/up. I was trying to think of reasons to prefer one over the other. |
Sure. I guess I was thinking of the mechanism (tree walking and applying some code to every construct) moreso that the naming and timing. The generic walking is still a useful feature to have, to implement these. Maybe the answer is that aspects are implemented using the tree walker? |
Yes, tree walking is useful, but I am not sure that interesting as a shared utility. Given a node with children, implementing a simple DFS/BFS is quite trivial. Without jsii supporting lambdas, a generic tree walker would require too much ceremony in my mind, but I don't object to it. Bottom line, I think we should leave synthesis out of this story for now besides aiding in defining the application phases. |
I was avoiding asking this question, but I do want to ask it now that the conversation has shifted.
I'll be honest I don't really like the idea of it as an aspect because I want clear control of Security Groups. |
I was surprised in Typescript that public static members of a child class are not available in the parent during |
Are there any use-cases for aspects running during or after synthesis? It seems to me that aspects all run prior to synthesis, perhaps the term 'phase' is overloaded here.
It's not clear to me what the objections to reactive application of aspects are. How is removing a tag complicated by a reactive model? What does it mean to be declarative in this context? If you can both add and remove a tag then order and scope matters, and I don't see how phases help us here. If we guarantee a predictable order of application for existing and future children with a reactive model, is that not the simplest behavior to reason about? |
I think we should move for implementing this, even though we haven't quite specced out all the use cases it could be used for.
The aspect system I'm proposing is composed of 2 things:
Right now, it can be used to simplify the implementation of Tags, and my spidey senses are saying we can find other uses for it in the future. |
I'm going to try to answer a few questions from above and propose a slight modification of this proposal with some specifics about tags.
I think the shaky part that @sam-goodwin is asking me about is really captured here. If
I would like these to both result in the same situation that PrecedenceWhat happens when tags disagree. The example above is a simple use case. In the first design I tried to provide complete set of controls to the developer for precedence and I think that created an overcomplicated solution. Three options:
In my opinion full control is too complex and not necessary for the 80 % use case. I currently took more of a full control option and I think we should eliminate it. This means removing The simplified rules for precedence then become:
Closest to me is the construct has the highest priority and in a cascading fashion delegates that priority to it's nearest parent. If the construct has a tag set multiple times the last one in wins. I'm voting to not enable further user control of precedence, but I'm open to hearing use cases. RemovalRemoval in the current implementation has a special RemoveTagProps I think we should kill that and consolidate to just one interface with two aspects Features being removed
I don't think these issues are deal breakers most developers can either write unit tests or synth templates and inspect to understand what is going to happen. EditMany of these ideas came from a conversation with @rix0rrr. So if you like them, they were his. If you don't, yell at me 😀 |
Alright, back from vacation and I have a few more questions/decisions to talk about. Aspect API/**
* Aspects that can be applied to Constructs.
*/
export interface IAspect {
readonly type: string;
visit(node: Construct): void;
} The main change here is that I am handing the Alternatively I did consider making myConstruct.apply(new SetTag('MyKey', 'MyValue'));
# versus
myConstruct.apply(new SetTag(myParent, 'MyPathId', { key: 'MyKey', value: 'MyValue' })); PropagatePropagate Propagate to Resources Only
This sounds ok, but my next question is should the tag propagate until it hits a resource or the leaf node? Obviously in our current implementation this should almost always have a
This situation has some interesting results regarding our current VPC and Subnet L2s.
For Propagate to Resources Only the tag appears on VPC and IGW. For Propagate to at least one Resource the situation results in the same behavior as Now we can say these constructs are unique, and we can override the Which propagation strategy do you prefer from above? Or is there another option here? If we do nothing Resource Public API changeWhen we flip tag propagation and resolution to happen at synthesis we have to open up the resource public API a bit, specifically properties need to be modified. /**
* Represents a CloudFormation resource.
*/
export class Resource extends Referenceable {
// snip
/**
* AWS resource properties.
*
* This object is rendered via a call to "renderProperties(this.properties)".
*/
public readonly properties: any;
// snip I don't see any way around this if we are resolving during synthesis. I considered using overrides, but that seems like a horrible idea since overrides should not normally be needed. 👎 The other option is make Synthesis TimeI have two options to discuss here.
I prefer the second option because this means during testing you don't have to manually trigger I have a good portion of this coded and working, but I think these key decisions should be discussed. I plan to clean up the original PR tonight and push up some code to help this discussion. Your feedback is greatly needed on the topics above. |
Welcome back! Same for me :).
Agreed this is correct. Before we had
Not sure you need it for that reason. I'm still favoring order implied in the visiting mechanism, so the work that is done in
Agreed. Also, I think I would favor construct.apply(new Tag('key', 'value'[, options]));
construct.apply(new RemoveTag('key')); I'm optimizing the 99% case of applying tags and never removing any, over naming consistency between the classes.
Which makes me think this is actually not an useful feature to support. If But in that case, you might as well have set Or is "Set tags the PRIMARY RESOURCE of this compound construct, and not any of the others" ? Seems like
I'm not sure why this is true. The logic you have proposed in the PR (a (generated) Resource knows whether it's taggable or not and renders the tags as part of its toCloudFormation()) seems solid to me. I'm imagining something like the following: // Taggable resource is identified by having a 'tags' property of type 'TagManager'.
// 'tags' is used in renderProperties.
//
// This is all generated by cfn2ts.
class TaggableGeneratedResource extends Resource {
public readonly tags: TagManager;
public renderProperties() {
return {
// ...,
tags: tags.render(singleTagRenderingFunction)
};
}
}
// An aspect calsl the methods on 'tags'.
class Tag implements IAspect {
constructor(private readonly key: string, private readonly value: string) {
}
public visit(construct: Construct) {
if (TagManager.isTaggable(construct)) {
construct.tags.set(this.key, this.value);
}
}
}
interface ITaggable {
tags: TagManager;
}
// TagManager has a static function to do some duck typing
class TagManager {
// Anything that has a 'tags' property is probably taggable.
// All hail Javascript
public static isTaggable(x: any): x is Taggable {
return x.tags !== undefined;
}
public set(key: string, value: string) {
// ...
}
public remove(key: string) {
// ...
}
public render(singleTagRenderCallback: (t: Tag) => any) {
// ...
}
}
I prefer this since aspects are a concept that exceeds stack contents. If we have the function: app.applyAspects(); That is not too big of a hassle to call inside tests that test aspects (imo). |
I have not explored the idea of having selectors any more in my head yet, other than mentioning it above. It would be worthwhile to think about what that might look like. It might be as simple as adding a decorator class: class RestrictAspect implements IAspect {
constructor(private readonly inner: IAspect, private readonly pathPattern: string) {
}
public visit(construct: Construct) {
if (this.pathMatches(construct.path)) {
this.inner.apply(construct);
}
}
}
// To be used as:
vpc.apply(new RestrictAspect(new Tag('key', 'value'), '*/Resource'); This is a rough sketch, to make it really ergonomic we probably need more information from the framework, such as where the tag was defined (so we can do relative resource paths) or (gasp) construct class names so we can select on those in addition to selecting on ids. EDIT: Could also be that vpc.apply(new Tag('key', 'value')).restrictTo('*/Resource'); Nicer to read, though not as generic as the decorator. |
Aspect framework is added. Aspects can be used to affect the construct tree without modifying the class hierarchy. Tagging has been reimplemented to leverage the Aspect framework and simplify the original tag design. Tag Manager still exists, but is no longer intended for use by L2 construct authors. There are two new classes `cdk.Tag` and `cdk.RemoveTag`. Code generation has been modified to add tag support to any CloudFormation Resource that supports tagging, by creating a Tag Manager for that resource as a `tags` attribute. Breaking Change: if you are using TagManager the API for this object has completely changed. You should no longer use TagManager directly, but instead replace this with Tag Aspects. `cdk.Tag` has been renamed to `cdk.CfnTag` to enable `cdk.Tag` to be the Tag Aspect. Fixes aws#1136.
A generalized aspect framework is added. Aspects can be used to affect the construct tree without modifying the class hierarchy. This framework is designed to be generic for future use cases. Tagging is the first implementation. Tagging has been reimplemented to leverage the aspect framework and simplify the original tag design. Tag Manager still exists, but is no longer intended for use by L2 construct authors. There are two new classes `cdk.Tag` and `cdk.RemoveTag`. As new resources are added tag support will be automatic as long as they implement one of the existing tag specifications. All L2 constructs have removed the TagManager. Code generation has been modified to add tag support to any CloudFormation Resource with a matching specification, by creating a Tag Manager for that resource as a `tags` attribute. The schema code now includes the ability to detect 3 forms of tags which include the current CloudFormation Resources. BREAKING CHANGE: if you are using TagManager the API for this object has completely changed. You should no longer use TagManager directly, but instead replace this with Tag Aspects. `cdk.Tag` has been renamed to `cdk.CfnTag` to enable `cdk.Tag` to be the Tag Aspect. Fixes aws#1136.
A generalized aspect framework is added. Aspects can be used to affect the construct tree without modifying the class hierarchy. This framework is designed to be generic for future use cases. Tagging is the first implementation. Tagging has been reimplemented to leverage the aspect framework and simplify the original tag design. Tag Manager still exists, but is no longer intended for use by L2 construct authors. There are two new classes `cdk.Tag` and `cdk.RemoveTag`. As new resources are added tag support will be automatic as long as they implement one of the existing tag specifications. All L2 constructs have removed the TagManager. Code generation has been modified to add tag support to any CloudFormation Resource with a matching specification, by creating a Tag Manager for that resource as a `tags` attribute. The schema code now includes the ability to detect 3 forms of tags which include the current CloudFormation Resources. BREAKING CHANGE: if you are using TagManager the API for this object has completely changed. You should no longer use TagManager directly, but instead replace this with Tag Aspects. `cdk.Tag` has been renamed to `cdk.CfnTag` to enable `cdk.Tag` to be the Tag Aspect. Fixes #1136 Fixes #1497 Related #360
(Follow-up discussion in response to #1007)
We should add an Aspect system to the construct tree, doing similar things as
Entity/Component/System systems do in games. (I'm mostly ignoring the System
part of ECS in the rest of this discussion)
Bear with me on (inconsistent use of) the proper names, and I'm open to suggestions.
Background
EC systems have the same purpose in game/physics modeling as they do in our system: make sense of a mess of objects that
share common functionality, but for which it becomes problematic to derive a correct inheritance hierarchy. For example, a game
consists of many different "objects" in a very large list of objects, and on every game tick:
Modeling these different aspects every object can have correctly requires a complex class hierarchy,
and sometimes even becomes inexpressible.
In our system, we have something something similar: we have a giant tree of objects (similar to the list
in game land, but hierarchically organized for added fun) that we can walk over, and we have several behaviors
that a construct can have:
Motivation
Because we have encapsulation, if we want to go the "classic OO" route, if we wanted to expose all the
functionality of underlying constructs, we'd have to repeat all features of every encapsulated construct
at every intermediate constructs.
For example, in the following example, even though
Bucket
andPipeline
arethe only
Taggable
objects (let's say exemplified by the present of asetTags()
method),SecretBucket
andBestPipeline
would need to bothimplement this method and forward to the correct subobjects.
Even though this is strictly correct in an OO sense, there is a certain burden
associated with doing this forwarding work that will make it harder for users
to write "correct" constructs (as there will be a lot of busywork involved with
doing it), and so the most probable end result will be that users writing
constructs won't consider all potential uses which will make it unnecessarily
hard (or even impossible) to reuse their constructs.
This will probably lead to user frustration and fragmentation of the construct
ecosystem that we're hoping to foster.
Benefits
The essense of a component system is to disentangle certain aspects of an
object's behavior from its identity (whereas typical OO modeling would tie
those together).
It allows us to express commands like:
In our case we have the added constraint that we want to select
a subtree to apply our operations to.
Current Situation
Our tree model currently has provisions for iterating over all children of
a subtree. However, we have no generalized model for applying an operation
to all nodes it makes sense to apply this operation to. Right now, we
do it in an ad-hoc fashion:
and if a construct happens to have a certain method, we'll call it.
constructs that happen to have a certain public property and if it exists,
assume it's a set of tags that also need to be applied to us.
Never mind that the mechanism we have for tags doesn't even allow specifying
tags on an arbitrary part of the tree: it HAS to be on a construct that is
already tag-aware, otherwise there is no
construct.tags.setTag()
attribute toeven be called.
Proposed Implementation
We already have most of the tools necessary to achieve this solution. The only
thing we need to do is an composite way of encoding aspects to a
Construct
,and some mechanism to walk the tree to do something to every construct that
a certain aspect applies to. Let's call those "visitors" for now, but we could
call them something more memorable as well (aspectors, twiddlers, what have
you). We have the question of how to match up the visitor with the aspect,
given that things like
instanceof
are not reliable. A string indicatinga type should do, it seems simple enough.
The following mechanism would do it:
Hopefully it's clear enough how
apply()
might work to satisfy the goalswe want.
(Normally we'd use generics to make aspect visiting typesafe, but since we don't
have generics we're going to have to do typecasting in the visitor. An
acceptable compromise.)
Example 1: Tags
So if we want to apply tags to a subtree, we can now do this (assuming some
sort of
TaggableAspect
class):How would using
TaggableAspect
look from the perspective of a taggableResource
?Example 2: Rendering CloudFormation (and other synthesis)
For example:
And in
Stack
we'd go:Laziness
The
apply()
statement as modeled right now will be applied as soon as it'scalled, which (though very simple, which is a plus) requires its users to be
aware of ordering, and makes it possible to make mistakes against ordering.
I'd prefer it if we keep the CDK constructs as declarative as possible, with
as little potential to make ordering mistakes as possible.
For example:
A simple extension is make a reactive visitor system, where
apply()
recursesover the children immediately, but constructs also remember all visitors that
are applied to them and will apply the visitors to any new children that get
added later. This pattern crops up a number of times already in the CDK code,
since in many places we want to be invariant against the order in which things
happen (ELBv2 example: first adding a Target to a TargetGroup and then adding
the TargetGroup to a Listener, vs first adding a TargetGroup to a listener and
then adding a Target to it -- in both cases do we want to associate open up the
appropriate SecurityGroup ports, and so an (unformalized) reactive system is in
place).
If desired, we could even make this behavior configurable:
The text was updated successfully, but these errors were encountered: