Easier identification of experimental modules

[eladb]

PR	Champion
#

Description

The AWS CDK is a large class library which is released through a single version. In order to allow innovation and quick iteration, we decided to leverage the model employed by the node.js project and tag APIs either as STABLE or EXPERIMENTAL (as described in our user guide).

Technically, each API element (class, enum, method, property) can be have a stability tag. If it is not tagged, it will inherit it's stability from it's parent, all the way up to the module level.

Practically, the current "high resolution" module causes confusion and oftentimes results in users coding against experimental APIs without their knowledge. This is because most of the construct library's modules include "CFN resources" which are always STABLE, while other constructs can still be EXPERIMENTAL. Users are also not accustomed to reviewing the API documentations for every module they use.

The purpose of this RFC is to improve the experience around consuming experimental modules, so that it will be easier for customers (in the common case) to acknowledge that they are using experimental APIs.

@ccfife proposes a solution where experimental constructs will be available only under the "@aws-cdk/aws-xxx-experimental" module and only move to the main module as they stabilize. The main module initially will only include "CFN resources" (which are stable).

Progress

• Tracking Issue Created
• RFC PR Created
• Core Team Member Assigned
• Initial Approval / Final Comment Period
• Ready For Implementation
• Resolved

[eladb]

For the record, I think we're going to bring pain onto ourselves if we want to split off experimental APIs into a separate package. Here are some thought exercises:

• What if we want to add an experimental method to an otherwise stable class? Do we copy the complete class?
• What if we want to add an experimental property to an otherwise stable props struct? Do we copy the complete class?
• What if we want to provide a stable API that depends on an experimental interface? (Remember, we can easily do this and there is no reason we shouldn't: the breakage caused by any changes to the experimental API will be detected by our complete in-tree build and corrected by the change maker, leaving to trace of it to downstream consumers) We won't be able to do that as we'll get cyclic dependencies.
• What if we stabilize an experimental class? Do we move it from the experimental package to the stable package, thereby breaking everyone in the process?

The better solution is to vend 2 different flavors of CDK from a single codebase:

• stable APIs only
• stable+experimental APIs

This will solve all issues mentioned above:

• Add experimental features to stable classes: just will not appear in the stable flavor, WILL appear in the s+e flavor.
• Depend on experimental APIs from stable: will just work, experimental API is just considered private in the s flavor.
• Stabilizing a class: s+e flavor will continue to work as usual, class now appears in s flavor.

Originally posted by @rix0rrr in https://github.com/_render_node/MDIzOlB1bGxSZXF1ZXN0UmV2aWV3VGhyZWFkMjMwOTM0MzY2OnYy/pull_request_review_threads/discussion

[eladb]

@rix0rrr how will this work for typescript modules?

[rix0rrr]

We remove the hidden APIs from the .d.ts files.

The one I don't have a solution for will be JavaScript users, but how will they know certain functions exist in the first place? And don't JS editors still use .d.ts files (if present) for autocomplete?

[eladb]

We remove the hidden APIs from the .d.ts files.

Sounds tricky but might work :-) I am not concerned about JavaScript as they use .d.ts and the reference documentation as well.

We can consider to add "@internal" to all "@experimental" elements.

---

I think auto-generating an experimental variant of the library is an interesting idea. With the mono-bundle (#6) it might even be an awesome developer experience. We can prototype this together with the prototype for mono-cdk.

[eladb]

A few more thoughts:

1. If we release two CDK variants, how would 3rd party libraries work? If I consumes two 3rd party modules, one that peer-depends on aws-cdk and one that peer-depends on aws-cdk-experimental, which CDK do I depend on? How do they interoperate?
2. Today, we have stable APIs that return non-stable types (e.g. app.synth() which is stable returns cxapi.CloudAssembly which is experimental). This works. Users can use app.synth() and consider it stable as long as they don't explicitly pick into the cloud assembly. The "correct" behavior would probably be to "erase" the experimental type from the signature of the stable method (i.e. it needs to return void), but that seems like it's going to be way more tricky than the original plan to just erase @experimental elements, no?

[rix0rrr]

app.synth() which is stable returns cxapi.CloudAssembly which is experimental

Yeah, that doesn't make sense, and I'm having a hard time thinking of models in which we can express this. In C, we would model this using opaque pointers. That would correspond to the type (name) cxapi.CloudAssembly being stable, but all of its members being experimental (i.e., invisible).

That would express the desired intent, but TypeScript's structural typing may play havoc on us (for instance, APIs that expect to take a cxapi.CloudAssembly, with all of its fields erased would also accept a {} and that is definitely not what is intended.

We may need to force TypeScript into nominal typing.

If we release two CDK variants, how would 3rd party libraries work?

In the most base approach, it's fine to ignore the peerDependencies requirements. You can substitute your own, knowing that they will work.

Slightly more sophisticatedly, semver has some semantics around version extensions that might come in useful ("@aws-cdk/cdk": "^2.0" vs "^2.0-experimental"). Have to do some research into that though.

[eladb]

In the most base approach, it's fine to ignore the peerDependencies requirements. You can substitute your own, knowing that they will work.

How will it work? One of the imports will fail...

[eladb]

Oh wait, now I understand that you want to release the experimental module under the same name but with a different version. Ok that’s interesting.

[thesurlydev]

I'm curious. Is the experimental qualifier necessary because you want to continue pinning the same version to all modules? Without knowing all of the complexities why not promote stable modules to a 1.x.x version and keep the unstable modules < 1.x.x and classify as release candidates as they approach a 1.x.x? Then from this point forward follow semantic versioning.