Supporting non-python CDK projects

[jfalkenstein]

This change adds support for referencing a cdk.json as the Template Handler's path, enabling support for CDK infrastructure that isn't even defined in Python!

[jfalkenstein]

@X-Guardian I've resolved the validation errors you've pointed out and have even buried the useless stdout and stderr results from when we check if the commands exist. I've also explicitly caught the CalledProcessError that gets raised when running subprocesses and am re-raising them wrapped in a SceptreException. This should hide the python stacktrace part (that isn't useful except for the actual message itself). It will not hide the JavaScript stacktrace, though. And that's good, because that's ultimately the source of the issue.

I believe this should cover all the concerns you've raised.

[zaro0508]

excellent work @jfalkenstein! I've looked thru the code and it generally looks good to me. I just have a few suggestions but nothing that would block approval. It looks like there are a few alternative ways to setup sceptre to deploy cdk apps. I have only tested sceptre with python (without cdk.json) and it looks like it's generally working. There are of course some scenarios that I think need improving however I think it can be handled as follow on changes. This PR is pretty large as is so I recommend we merge and create issues to handle with follow on PRs.

[zaro0508]

I agree with @X-Guardian here. your sentence has some extra words, it reads like this ... in your app in order to use the use a cdk.json file with the CDK handler

[zaro0508]

mixing in this example with the existing python example makes things sort of confusing. I think it would clear things up if we could put the two in completely separate folders. Another option is to provide a tree list of the sceptre-example folder and provide an overview of the file relationships

[jfalkenstein]

@zaro0508, Having this example here demonstrates how using Sceptre can tie together CDK projects and other resources into an environment. I don't see how this makes things confusing. Are you able to elaborate on why you find this confusing?

[zaro0508]

it's confusing because files for a non-python project are mixed in with files for a python example. Also python are put in the sceptre standard templates folder while non-python files are not. Those are all fine except i think new users looking at this handler would be confused if they just looked at sceptre-example folder for the first time.

[zaro0508]

i find this confusing. It's saying you cannot use nested values in cdk.json but it also says to use cdk.json if you need to specify such values. I assume such values refers to nested values?

The sentence in the readme is pretty clear..

Complex data structures (i.e. lists and dicts) are not supported in your CDK context from Sceptre.
You can explicitly set those up on your cdk.json (if they're required), but the CDK template handler
will only accept simple key/value pairs for your context that it passes

[jfalkenstein]

Sure! I'd can repeat that phrasing here.

[zaro0508]

i assume we do this because cdk.json is on in templates folder? why not setup this example within the scpetre folder convention?

[jfalkenstein]

Yes, it's because the cdk.json is outside the templates directory. Since a CDK project can be a lot more substantial than a single template file, it's likely users of this handler would want to place their CDK project outside of templates. In several senses, a CDK project is not a "template" and it doesn't make so much sense to bury a CDK project within a directory named templates. In fact, with the way that CDK initializes repositories and such, it generally makes more sense to have the CDK project be in a different directory than the Sceptre stuff.

I could put a comment to this effect here to clear up the confusion, though.

[zaro0508]

yes, users would probably be confused without a little explanation.

[X-Guardian]

I am receiving the following message when I launch my cdk.json stack even though I am not specifying or using sceptre_user_data:

[2023-03-06 10:00:22] - The cdk_json deployment_type does not support sceptre_user_data. Any values passed to your stack must be done via the cdk context. All values in your sceptre_user_data will be ignored.

[dboitnot]

I didn't spot anything troublesome, but I'm honestly not familiar yet with a lot of what's happening in these modules.

[jfalkenstein]

@X-Guardian I'm not sure how that would be happening. Both an empty_dict and None resolve to falsy in Python. This line of code is intended to only make that warning show up when sceptre_user_data is passed. Are you positive you don't have sceptre_user_data on your Stack Config? Are you able to post your StackConfig so I can try to replicate what might be causing this error?

[X-Guardian]

@X-Guardian I'm not sure how that would be happening. Both an empty_dict and None resolve to falsy in Python. This line of code is intended to only make that warning show up when sceptre_user_data is passed. Are you positive you don't have sceptre_user_data on your Stack Config? Are you able to post your StackConfig so I can try to replicate what might be causing this error?

Ok, I worked out what was causing this. It is a Sceptre feature/bug whereby if you have for example a cdk.yaml file without a sceptre_user_data property and then in the parent directory you have another cdk.yaml file with a sceptre_user_data property, running sceptre on the first file will cause the sceptre_user_data to be inherited from the parent file of the same name. I'm sure this is not intentional...

[jfalkenstein]

@X-Guardian is it possible that you have defined sceptre_user_data in a Stack Group config (i.e. in one of the config.yaml files)?

[X-Guardian]

@X-Guardian is it possible that you have defined sceptre_user_data in a Stack Group config (i.e. in one of the config.yaml files)?

Nope. Try it yourself. Copy the special-file-asset-cdk-json.yaml file from the PR into a subdirectory, edit the original file and add a sceptre_user_data block, then generate the stack in the subdirectory and you will get the sceptre_user_data warning. Rename the file in the parent directory and the warning will not be shown.

[jfalkenstein]

@X-Guardian Are you saying you have two files of the same name, one in a parent StackGroup and one in a child StackGroup?

[jfalkenstein]

Alright, I think this PR has gotten a lot of comments and reviews. Thank you all very much @zaro0508 , @ericabrauer , @X-Guardian , @dboitnot. Since I've addressed/resolved all requested changes and the only lingering issue seems like, if it's an issue at all, it's an issue with Sceptre itself and not this handler, I'm going to get this out!