sceptre 4.5.0 fails on template path #1506

zaro0508 · 2024-08-28T17:31:03Z

Sceptre version 4.5.0 fails to load template when using the path key. This is a regression, it works without error in sceptre v4.4.2

Environment

sceptre v4.5.0
python v3.11
MAC OSX

Steps to reproduce

Create a basic Sceptre config with file..

template:
  path: local-template.yaml
stack_name: test-template-path

run sceptre launch

➜  AWS_PROFILE=sageit-admin AWS_DEFAULT_REGION=us-east-1 sceptre launch test.yaml

/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/stack.py:365: DeprecatedWarning: template_path is deprecated as of 4.0.0. Use the template Stack Config key instead.
  and self.template_path == stack.template_path
Traceback (most recent call last):
  File "/Users/zaro0508/.pyenv/versions/py311-sceptredev/bin/sceptre", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/cli/helpers.py", line 46, in decorated
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/cli/launch.py", line 58, in launch_command
    launcher.print_operations(prune)
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/cli/launch.py", line 86, in print_operations
    deploy_plan = self._create_deploy_plan()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/cli/launch.py", line 110, in _create_deploy_plan
    plan = self._make_plan(self._context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/plan/plan.py", line 51, in __init__
    self.graph = StackGraph(all_stacks)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/config/graph.py", line 34, in __init__
    self._generate_graph(stacks)
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/config/graph.py", line 80, in _generate_graph
    self._generate_edges(stack, stack.dependencies)
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/config/graph.py", line 97, in _generate_edges
    cycle = nx.find_cycle(self.graph, orientation="original")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/networkx/algorithms/cycles.py", line 444, in find_cycle
    if previous_head is not None and tail != previous_head:
                                     ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/stack.py", line 365, in __eq__
    and self.template_path == stack.template_path
        ^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/deprecation.py", line 260, in _inner
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zaro0508/.pyenv/versions/3.11.2/envs/py311-sceptredev/lib/python3.11/site-packages/sceptre/stack.py", line 453, in template_path
    return self.template_handler_config["path"]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
KeyError: 'path'

The text was updated successfully, but these errors were encountered:

Pin to Sceptre 4.4.2 due to issue Sceptre/sceptre#1506

alex-harvey-z3q · 2024-08-29T01:16:14Z

@zaro0508 I can't reproduce this, although I do feel a bit worried that we don't have enough testers any more! Could you try reverting this commit and see if it still happens?

993ef0937d9a49a1683feb9591c4526ee0fe29af

zaro0508 · 2024-08-29T17:58:50Z

I dug a little deeper. I can definitely reproduce this problem however not consistently. I created a simple test project and ran the exact same launch command 26 times and was able to reproduce it 20 out of 26 runs. When it does not work I immediately get this error KeyError: 'path'. When it does work i get Do you want to launch 'prod/noopC.yaml' [y/N]:

I believe it has something to do with multiple nested dependencies using both path and http type of templates. Test project files are here: https://plnkr.co/edit/zi2Q5TdjBdQXxbPD?open=lib%2Fscript.js

(py311-sceptre451) ➜  sageit pwd
/Users/zaro0508/temp/sageit

(py311-sceptre451) ➜  sageit python -V
Python 3.11.2

(py311-sceptre451) ➜  sageit sceptre --version
Sceptre, version 4.5.1

(py311-sceptre451) ➜  sageit tree
.
├── config
│   ├── config.yaml
│   └── prod
│       ├── noopA.yaml
│       ├── noopB.yaml
│       └── noopC.yaml
└── templates
    └── noop.yaml

4 directories, 5 files

Log of test runs: https://gist.github.com/zaro0508/e2827741ccb67927f3111776587a05ba

zaro0508 · 2024-08-29T19:12:32Z

@zaro0508 I can't reproduce this, although I do feel a bit worried that we don't have enough testers any more! Could you try reverting this commit and see if it still happens?
993ef0937d9a49a1683feb9591c4526ee0fe29af

I reverted commit 993ef09 from my local Sceptre checkout (master branch) then installed Sceptre from that branch (pip install -e .) I was NO longer able to reproduce this issue when running with this version.

alex-harvey-z3q · 2024-08-30T08:00:16Z

I have reproduced using your test files. Adding --debug

A good run:

root@d9e5c643bdbd:/app/test# sceptre --debug launch  prod/noopC.yaml 
[2024-08-30 07:55:04] - Adding yaml constructors for the entry point groups ['sceptre.hooks', 'sceptre.resolvers']
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.hooks.asg_scaling_processes.ASGScalingProcesses'> with node tag !asg_scaling_processes
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.hooks.asg_scaling_processes.ASGScalingProcesses'> with node tag !asg_scheduled_actions
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.hooks.cmd.Cmd'> with node tag !cmd
[2024-08-30 07:55:04] - Added constructor for <class 'hook.stack_termination_protection.StackTerminationProtection'> with node tag !stack_termination_protection
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.resolvers.environment_variable.EnvironmentVariable'> with node tag !environment_variable
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.resolvers.file_contents.FileContents'> with node tag !file_contents
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.resolvers.join.Join'> with node tag !join
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.resolvers.no_value.NoValue'> with node tag !no_value
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.resolvers.select.Select'> with node tag !select
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.resolvers.split.Split'> with node tag !split
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.resolvers.stack_attr.StackAttr'> with node tag !stack_attr
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.resolvers.stack_output.StackOutput'> with node tag !stack_output
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.resolvers.stack_output.StackOutputExternal'> with node tag !stack_output_external
[2024-08-30 07:55:04] - Added constructor for <class 'sceptre.resolvers.sub.Sub'> with node tag !sub
[2024-08-30 07:55:04] - Added constructor for <class 'resolver.file.File'> with node tag !file
[2024-08-30 07:55:04] - Added constructor for <class 'main.Aws'> with node tag !aws
[2024-08-30 07:55:04] - Added constructor for <class 'resolver.rcmd.SceptreResolverCmd'> with node tag !rcmd
[2024-08-30 07:55:04] - Added constructor for <class 'resolver.ssm.SSM'> with node tag !ssm
[2024-08-30 07:55:04] - Added constructor for <class 'resolver.stack_output.StackOutput'> with node tag !stack_output
[2024-08-30 07:55:04] - Added constructor for <class 'resolver.minify_file_contents.MinifyFileContents'> with node tag !minify_file_contents
[2024-08-30 07:55:04] - Reading in 'prod/config.yaml' files...
[2024-08-30 07:55:04] - Config: {'project_path': '/app/test', 'stack_group_path': 'prod', 'project_code': 'sage-bionetworks', 'region': 'ap-southeast-2', 'template_bucket_name': 'kayodatalake-nonprod-sceptre-cfn'}
[2024-08-30 07:55:04] - Reading in 'prod/noopC.yaml' files...
[2024-08-30 07:55:04] - Config: {'project_path': '/app/test', 'stack_group_path': 'prod', 'project_code': 'sage-bionetworks', 'region': 'ap-southeast-2', 'template_bucket_name': 'kayodatalake-nonprod-sceptre-cfn', 'template': {'path': 'noop.yaml'}, 'stack_name': 'noopC', 'dependencies': ['prod/noopB.yaml']}
[2024-08-30 07:55:04] - Reading in 'prod/noopB.yaml' files...
[2024-08-30 07:55:04] - Config: {'project_path': '/app/test', 'stack_group_path': 'prod', 'project_code': 'sage-bionetworks', 'region': 'ap-southeast-2', 'template_bucket_name': 'kayodatalake-nonprod-sceptre-cfn', 'template': {'type': 'http', 'url': 'https://raw.githubusercontent.com/Sage-Bionetworks/aws-infra/v0.7.3/templates/noopB.yaml'}, 'stack_name': 'noopB', 'dependencies': ['prod/noopA.yaml']}
[2024-08-30 07:55:04] - Reading in 'prod/noopA.yaml' files...
[2024-08-30 07:55:04] - Config: {'project_path': '/app/test', 'stack_group_path': 'prod', 'project_code': 'sage-bionetworks', 'region': 'ap-southeast-2', 'template_bucket_name': 'kayodatalake-nonprod-sceptre-cfn', 'template': {'type': 'http', 'url': 'https://raw.githubusercontent.com/Sage-Bionetworks/aws-infra/v0.7.3/templates/noopA.yaml'}, 'stack_name': 'noopA'}
[2024-08-30 07:55:04] - Generate dependencies for stack prod/noopC
[2024-08-30 07:55:04] -   Added dependency: prod/noopB
[2024-08-30 07:55:04] - Generate dependencies for stack prod/noopB
[2024-08-30 07:55:04] -   Added dependency: prod/noopA
[2024-08-30 07:55:04] - Generate dependencies for stack prod/noopA
Do you want to launch 'prod/noopC.yaml' [y/N]:

A bad run:

root@d9e5c643bdbd:/app/test# sceptre --debug launch  prod/noopC.yaml                                                                                                                17:56:28 [29/3462]
[2024-08-30 07:56:27] - Adding yaml constructors for the entry point groups ['sceptre.hooks', 'sceptre.resolvers']                                                                                    
[2024-08-30 07:56:27] - Added constructor for <class 'sceptre.hooks.asg_scaling_processes.ASGScalingProcesses'> with node tag !asg_scaling_processes                                                  
[2024-08-30 07:56:27] - Added constructor for <class 'sceptre.hooks.asg_scaling_processes.ASGScalingProcesses'> with node tag !asg_scheduled_actions                                                  
[2024-08-30 07:56:27] - Added constructor for <class 'sceptre.hooks.cmd.Cmd'> with node tag !cmd                                                                                                      
[2024-08-30 07:56:27] - Added constructor for <class 'hook.stack_termination_protection.StackTerminationProtection'> with node tag !stack_termination_protection                                      
[2024-08-30 07:56:27] - Added constructor for <class 'sceptre.resolvers.environment_variable.EnvironmentVariable'> with node tag !environment_variable                                                
[2024-08-30 07:56:27] - Added constructor for <class 'sceptre.resolvers.file_contents.FileContents'> with node tag !file_contents                                                                     
[2024-08-30 07:56:27] - Added constructor for <class 'sceptre.resolvers.join.Join'> with node tag !join                                                                                               
[2024-08-30 07:56:27] - Added constructor for <class 'sceptre.resolvers.no_value.NoValue'> with node tag !no_value                                                                                    
[2024-08-30 07:56:27] - Added constructor for <class 'sceptre.resolvers.select.Select'> with node tag !select                                                                                         
[2024-08-30 07:56:28] - Added constructor for <class 'sceptre.resolvers.split.Split'> with node tag !split                                                                                            
[2024-08-30 07:56:28] - Added constructor for <class 'sceptre.resolvers.stack_attr.StackAttr'> with node tag !stack_attr                                                                              
[2024-08-30 07:56:28] - Added constructor for <class 'sceptre.resolvers.stack_output.StackOutput'> with node tag !stack_output                                                                        
[2024-08-30 07:56:28] - Added constructor for <class 'sceptre.resolvers.stack_output.StackOutputExternal'> with node tag !stack_output_external                                                       
[2024-08-30 07:56:28] - Added constructor for <class 'sceptre.resolvers.sub.Sub'> with node tag !sub                                                                                                  
[2024-08-30 07:56:28] - Added constructor for <class 'resolver.file.File'> with node tag !file                                                                                                        
[2024-08-30 07:56:28] - Added constructor for <class 'main.Aws'> with node tag !aws                                                                                                                   
[2024-08-30 07:56:28] - Added constructor for <class 'resolver.rcmd.SceptreResolverCmd'> with node tag !rcmd                                                                                          
[2024-08-30 07:56:28] - Added constructor for <class 'resolver.ssm.SSM'> with node tag !ssm                                                                                                           
[2024-08-30 07:56:28] - Added constructor for <class 'resolver.stack_output.StackOutput'> with node tag !stack_output                                                                                 
[2024-08-30 07:56:28] - Added constructor for <class 'resolver.minify_file_contents.MinifyFileContents'> with node tag !minify_file_contents                                                          
[2024-08-30 07:56:28] - Reading in 'prod/config.yaml' files...                                                                                                                                        
[2024-08-30 07:56:28] - Config: {'project_path': '/app/test', 'stack_group_path': 'prod', 'project_code': 'sage-bionetworks', 'region': 'ap-southeast-2', 'template_bucket_name': 'kayodatalake-nonpro
d-sceptre-cfn'}                                                                                                                                                                                       
[2024-08-30 07:56:28] - Reading in 'prod/noopC.yaml' files...                                                                                                                                         
[2024-08-30 07:56:28] - Config: {'project_path': '/app/test', 'stack_group_path': 'prod', 'project_code': 'sage-bionetworks', 'region': 'ap-southeast-2', 'template_bucket_name': 'kayodatalake-nonpro
d-sceptre-cfn', 'template': {'path': 'noop.yaml'}, 'stack_name': 'noopC', 'dependencies': ['prod/noopB.yaml']}                                                                                        
[2024-08-30 07:56:28] - Reading in 'prod/noopB.yaml' files...                                                                                                                                         
[2024-08-30 07:56:28] - Config: {'project_path': '/app/test', 'stack_group_path': 'prod', 'project_code': 'sage-bionetworks', 'region': 'ap-southeast-2', 'template_bucket_name': 'kayodatalake-nonprod-sceptre-cfn', 'template': {'type': 'http', 'url': 'https://raw.githubusercontent.com/Sage-Bionetworks/aws-infra/v0.7.3/templates/noopB.yaml'}, 'stack_name': 'noopB', 'dependencies': ['prod/noopA.yaml']}
[2024-08-30 07:56:28] - Reading in 'prod/noopA.yaml' files...
[2024-08-30 07:56:28] - Config: {'project_path': '/app/test', 'stack_group_path': 'prod', 'project_code': 'sage-bionetworks', 'region': 'ap-southeast-2', 'template_bucket_name': 'kayodatalake-nonprod-sceptre-cfn', 'template': {'type': 'http', 'url': 'https://raw.githubusercontent.com/Sage-Bionetworks/aws-infra/v0.7.3/templates/noopA.yaml'}, 'stack_name': 'noopA'}
[2024-08-30 07:56:28] - Generate dependencies for stack prod/noopB
[2024-08-30 07:56:28] -   Added dependency: prod/noopA
[2024-08-30 07:56:28] - Generate dependencies for stack prod/noopA
[2024-08-30 07:56:28] - Generate dependencies for stack prod/noopC
Traceback (most recent call last):

After reverting the above-mentioned commit, I note that the ordering of the dependencies generation was not deterministic before or after that change. I'm continuing to dig.

Way back in 2018, the original team were adding the debug command, and in that context set about to improve the output in some debug circumstances during inspection of Sceptre's internal objects. In response to #570, a custom `__eq__` method was added in df9fc20. Based on what can be seen of the original implementation of the custom `__eq__`, it has not been maintained in a long time, other than changes to it that appeared to cause issues for the current maintainers. -- A recent change to improve the output during cyclical dependency errors has introduced a call to `nx.find_cycle`. This function explicitly searches for a cycle in the graph. This appears to resolve in code testing for equality of nodes and thus calls to our custom `__eq__` function. The `__eq__` method however has been technically broken ever since `template_path` was made optional. This then results in `__eq__` failing. This fails as the code to indicate that `template_path` is deprecrated is then executed, which will fail if the setting has not really been defined. -- This PR proposes to simply remove `__eq__` since it is believed that it is no longer in use by anything. It has been more than 2 years since `template_path` was deprecrated and made optional, and the fact of this bug having never been observed until now is good evidence that it is not used or needed.

This reverts commit 20adde2 (#1200) Sceptre v4.5.2 fixed issue Sceptre/sceptre#1506

zaro0508 added a commit to zaro0508/organizations-infra that referenced this issue Aug 28, 2024

Fix build

97591c3

Pin to Sceptre 4.4.2 due to issue Sceptre/sceptre#1506

zaro0508 mentioned this issue Aug 28, 2024

Fix build Sage-Bionetworks-IT/organizations-infra#1200

Merged

zaro0508 added a commit to Sage-Bionetworks-IT/organizations-infra that referenced this issue Aug 28, 2024

Fix build (#1200)

20adde2

Pin to Sceptre 4.4.2 due to issue Sceptre/sceptre#1506

alex-harvey-z3q closed this as completed in 6c9ccd9 Aug 30, 2024

zaro0508 mentioned this issue Sep 3, 2024

Revert "Fix build" Sage-Bionetworks-IT/organizations-infra#1205

Merged

zaro0508 added a commit to Sage-Bionetworks-IT/organizations-infra that referenced this issue Sep 3, 2024

Revert "Fix build (#1200)" (#1205)

c9a5531

This reverts commit 20adde2 (#1200) Sceptre v4.5.2 fixed issue Sceptre/sceptre#1506

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sceptre 4.5.0 fails on template path #1506

sceptre 4.5.0 fails on template path #1506

zaro0508 commented Aug 28, 2024 •

edited

Loading

alex-harvey-z3q commented Aug 29, 2024

zaro0508 commented Aug 29, 2024 •

edited

Loading

zaro0508 commented Aug 29, 2024 •

edited

Loading

alex-harvey-z3q commented Aug 30, 2024

sceptre 4.5.0 fails on template path #1506

sceptre 4.5.0 fails on template path #1506

Comments

zaro0508 commented Aug 28, 2024 • edited Loading

Environment

Steps to reproduce

alex-harvey-z3q commented Aug 29, 2024

zaro0508 commented Aug 29, 2024 • edited Loading

zaro0508 commented Aug 29, 2024 • edited Loading

alex-harvey-z3q commented Aug 30, 2024

zaro0508 commented Aug 28, 2024 •

edited

Loading

zaro0508 commented Aug 29, 2024 •

edited

Loading

zaro0508 commented Aug 29, 2024 •

edited

Loading