Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CALM-DSL changes for v4.0.0 #318

Merged
merged 68 commits into from
Dec 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
375846a
Post account delete, ignore exceptions in deleting the entity from ca…
bekkamaditya Mar 4, 2024
f9912b8
DSL changes for Custom Provider feature (#449)
bekkamaditya May 27, 2024
0d377b2
Fix TC JSONs. #CALM-45281 (#459)
bekkamaditya May 27, 2024
fc46f51
added code for creating downloaded provider file (#452)
imranpasha1 May 29, 2024
450e149
CP Bug fixes - Part 1 (#464)
bekkamaditya Jun 10, 2024
818275d
HTTP code range support (#450)
vartikanutanix Jun 10, 2024
d2bdf9a
Cp decompile fixes (#479)
imranpasha1 Jun 12, 2024
a9e8bc5
Add HTTPResponseHandle import in builtins (#480)
vartikanutanix Jun 12, 2024
e0a4e6e
Fix for insert_uuid function used to insert uuids in action_list refe…
richakumarii Jun 24, 2024
b96b947
Handle credentials in name_uuid_map - insert_uuid related bug fix (#487)
richakumarii Jun 25, 2024
7b71ecd
Bug fix for #CALM-46070 (#488)
bekkamaditya Jun 25, 2024
882fc2c
Addition of unique uuid to edges in dag task attrs (#486)
richakumarii Jun 27, 2024
a2514af
Lint fixes in repo (#490)
dwivediprab Jul 1, 2024
db91392
moved cache update to create_account (#492)
utkarshb07 Jul 2, 2024
00c27ab
Support status mapping for Decision, While loop, Execute, Set Variabl…
vartikanutanix Jul 3, 2024
e347cdd
Lint fixes (#494)
dwivediprab Jul 4, 2024
d842506
Fix UTs for backward compatibility (#498)
vartikanutanix Jul 5, 2024
092c4d2
Fix UT (#499)
vartikanutanix Jul 9, 2024
e5e483e
Change runbook pod UT version to 390 (#505)
dwivediprab Jul 11, 2024
3ddc6d0
Added all responses back to minimise the automation fix (#506)
utkarshb07 Jul 16, 2024
19eae7c
Adding View Container permissions in ACPs via DSL (#502)
Geet-Sahitya-Dulam Jul 17, 2024
22f282f
(#CALM-39775) Add clone runbook command (#503)
jelisaveta-j Jul 17, 2024
83eb4db
(#CALM-45118) runbook output variables as list of objects (#458)
eqbalomar Jul 18, 2024
078e07f
(#ENG-674126) Adding View_Container permissions for ACPs via DSL from…
Geet-Sahitya-Dulam Jul 19, 2024
5e82eb8
CALM-46812: Fix calm describe app command (#517)
dwivediprab Jul 24, 2024
8abf2ad
Minor fix (#522)
bekkamaditya Jul 25, 2024
b7845c1
Fix regression in provider's watch/abort commands. #CALM-46875 (#525)
bekkamaditya Jul 30, 2024
790909a
Run make black (#528)
bekkamaditya Jul 30, 2024
bb54bae
Shell and powershell support - pending changes (#520)
richakumarii Aug 2, 2024
bf1fa32
(#CALM-47142) Query all runbooks (#535)
jelisaveta-j Aug 12, 2024
cc94956
HTTP EP Support for BP HTTP task (#526)
utkarshb07 Aug 12, 2024
7595c4e
Added quotas marker in tests (#539)
dwivediprab Aug 20, 2024
b1e74ce
Add HTTP Endpoint DSL Tests (#540)
shubhijainn Aug 21, 2024
e11aa23
fixed http var options when endpoint is not present (#541)
utkarshb07 Aug 22, 2024
5a21136
updated_delay (#524)
prasaddevi11 Aug 29, 2024
454baec
automation fix for escript_kubernetes.py (#543)
sankarspk Aug 29, 2024
3951124
Add project to http endpoint (#542)
shubhijainn Aug 29, 2024
544dcab
Custom Provider Bug Fixes (#545)
bekkamaditya Sep 2, 2024
9f3c08d
Added approval policy marker (#550)
dwivediprab Sep 11, 2024
7a7255d
improve describe app command to handle all providers (#548)
Jetx23 Sep 6, 2024
9f1d911
CALM-48037: Publish existing bp in accordance with latest existing mp…
dwivediprab Sep 16, 2024
479711c
CP Bug Fixes - Part 2 (#549)
bekkamaditya Sep 17, 2024
d67c0a1
CALM-39565: Block bp launches if there is no snapshot-policy data in …
dwivediprab Sep 25, 2024
5a53723
CALM-47578: Modify generated bp to use kNormalNic while doing calm in…
dwivediprab Sep 25, 2024
1bec7a2
Added a null check if deafult_value present in user_attrs object (#562)
richakumarii Sep 26, 2024
b62354f
CALM-47860: Publish bp to marketplace without platform dependent fiel…
dwivediprab Oct 3, 2024
e22587b
CALM-47418: Fix name in endpoint decompile (#559)
dwivediprab Oct 3, 2024
43b77b2
Add DSL file for preseeded provider in examples. #CALM-48185 (#566)
bekkamaditya Oct 4, 2024
3c2aef0
Change response code for failure in VM endpoints (#568)
vartikanutanix Oct 7, 2024
f702602
Change endpoint UT version to 4.0.0 (#572)
dwivediprab Oct 9, 2024
619c948
CALM-47711: AB Fix for Calm DSL get_job function (#574)
dwivediprab Oct 11, 2024
d108a17
CALM-47522 : Remove centos image from calm init bp command (#563)
dwivediprab Oct 14, 2024
f55585c
CALM-8700: Fix exec_app_action method to not query deleted apps. (#575)
dwivediprab Oct 14, 2024
e230b32
task/Automate_CALM-48812 (#579)
sankarspk Oct 16, 2024
8487f6f
Add step of adding vm details in init command (#588)
dwivediprab Nov 5, 2024
ef6d45f
CALM-49368: Add windev command in DSL Makefile (#591)
dwivediprab Nov 8, 2024
c4fffc4
CP TOI materials (#593)
utkarshb07 Nov 12, 2024
53daa6f
Minor fixes in a provider command docstring & init template (#595)
bekkamaditya Nov 13, 2024
3fd3425
undo unnecessary change
dwivediprab Nov 19, 2024
41f714c
Fix UTs affected due to change of output_variables to output_variable…
vartikanutanix Nov 14, 2024
408ed49
Change HTTP task compile, to avoid sending code_range_list if it's em…
vartikanutanix Nov 14, 2024
5a6f4df
CALM-48136: Fix AB by adding exec_target_reference in known json (#599)
dwivediprab Nov 14, 2024
67265fc
Bump version to 400 (#600)
dwivediprab Nov 19, 2024
c61554a
update release metadata to 400
dwivediprab Nov 21, 2024
e911011
Add 400 release notes (#603)
dwivediprab Dec 2, 2024
04461da
Merge branch 'dsl-400-cut' of github.com:nutanix/calm-dsl into dsl-40…
dwivediprab Dec 2, 2024
f9657d1
CALM-49890: Fix logs during failed MPI launch (#608)
dwivediprab Dec 2, 2024
d88551d
black format
dwivediprab Dec 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion CalmVersion
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.8.1
4.0.0
9 changes: 9 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,15 @@ dev:
venv/bin/pip3 install --no-cache -r requirements.txt -r dev-requirements.txt
venv/bin/python3 setup.py develop

windev:
# Setup our python3 based virtualenv in windows machine
# This step assumes python3 is installed on your dev machine
[ -f venv/Scripts/python3 ] || python -m venv venv
venv/Scripts/python -m pip install --upgrade pip
venv/Scripts/pip install setuptools --upgrade --ignore-installed
venv/Scripts/pip install --no-cache -r requirements.txt -r dev-requirements.txt
venv/Scripts/python setup.py develop

test-bed: dev
venv/bin/python3 tests/testprep.py

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

![Build](https://github.com/nutanix/calm-dsl/workflows/Setup%20&%20build%20calm-dsl/badge.svg)

`Latest release version: 3.8.1, Latest-release-tag: v3.8.1`
`Latest release version: 4.0.0, Latest-release-tag: v4.0.0`

`Latest Release Notes:` [read here](release-notes/3.8.1)
`Latest Release Notes:` [read here](release-notes/4.0.0)


# Nutanix Cloud Manager (NCM) Self Service (formerly Calm) DSL
Expand Down
257 changes: 257 additions & 0 deletions calm/dsl/api/provider.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,47 @@
import json
import os

from calm.dsl.log import get_logging_handle

from .resource import ResourceAPI
from .connection import REQUEST
from .util import strip_provider_secrets, patch_secrets, strip_uuids

LOG = get_logging_handle(__name__)


class ProviderAPI(ResourceAPI):
def __init__(self, connection):
super().__init__(connection, resource_type="providers", calm_api=True)

self.CREATE = self.PREFIX
self.BULK_CREATE = self.CREATE + "/bulk"
self.BULK_UPDATE = self.ITEM + "/bulk"
self.BULK_READ = self.ITEM + "/bulk"
self.COMPILE = self.ITEM + "/compile"
self.TEST_PROVIDER_VERIFY = self.PREFIX + "/{}/actions/{}/test_run"
self.ABORT_RUN = self.ITEM + "/runlogs/{}/abort"
self.POLL_RUN = self.ITEM + "/runlogs/{}"
self.CHILD_RUNLOG_LIST = self.ITEM + "/runlogs/{}/children/list"
self.RUNLOG_OUTPUT = self.ITEM + "/runlogs/{}/children/{}/output"
self.RUNLOG_LIST = self.CREATE + "/runlogs/list"
self.CLONE = self.ITEM + "/clone"
self.IMPORT_JSON = self.PREFIX + "/import_json"
self.IMPORT_FILE = self.PREFIX + "/import_file"
self.EXPORT_JSON = self.ITEM + "/export_json"
self.EXPORT_FILE = self.ITEM + "/export_file"

def check_if_provider_already_exists(self, provider_name):
params = {"filter": "name=={};state!=DELETED".format(provider_name)}
res, err = self.list(params=params)
if err:
return None, err

response = res.json()
entities = response.get("entities", None)
if entities and len(entities) > 0:
return entities[0], None
return None, None

def create(self, provider_payload):
return self.connection._call(
Expand All @@ -16,3 +51,225 @@ def create(self, provider_payload):
method=REQUEST.METHOD.POST,
timeout=(5, 300),
)

# This is equivalent to "compile" of whole Provider tree as present in Calm UI
def preview_validate(self, uuid):
return self.connection._call(
self.COMPILE.format(uuid), verify=False, method=REQUEST.METHOD.GET
)

def bulk_create(self, provider_payload):
res, err = self.connection._call(
self.BULK_CREATE,
verify=False,
request_json=provider_payload,
method=REQUEST.METHOD.POST,
)
if err:
return res, err

return self.preview_validate(res.json()["metadata"]["uuid"])

def bulk_read(self, id):
return self.connection._call(
self.BULK_READ.format(id), verify=False, method=REQUEST.METHOD.GET
)

def bulk_update(self, uuid, provider_payload):
res, err = self.connection._call(
self.BULK_UPDATE.format(uuid),
verify=False,
request_json=provider_payload,
method=REQUEST.METHOD.PUT,
)
if err:
return res, err

return self.preview_validate(uuid)

def run(self, uuid, action_uuid, payload):
return self.connection._call(
self.TEST_PROVIDER_VERIFY.format(uuid, action_uuid),
verify=False,
request_json=payload,
method=REQUEST.METHOD.POST,
)

def list_child_runlogs(self, provider_uuid, runlog_uuid):
return self.connection._call(
self.CHILD_RUNLOG_LIST.format(provider_uuid, runlog_uuid),
verify=False,
request_json={},
method=REQUEST.METHOD.POST,
)

def list_runlogs(self, payload=None):
return self.connection._call(
self.RUNLOG_LIST,
verify=False,
request_json=payload,
method=REQUEST.METHOD.POST,
)

def runlog_output(self, provider_uuid, runlog_uuid, child_runlog_uuid):
return self.connection._call(
self.RUNLOG_OUTPUT.format(provider_uuid, runlog_uuid, child_runlog_uuid),
verify=False,
method=REQUEST.METHOD.GET,
)

def poll_action_run(self, uuid, runlog_uuid, payload=None):
if payload:
return self.connection._call(
self.POLL_RUN.format(uuid, runlog_uuid),
request_json=payload,
verify=False,
method=REQUEST.METHOD.POST,
)
else:
return self.connection._call(
self.POLL_RUN.format(uuid, runlog_uuid),
verify=False,
method=REQUEST.METHOD.GET,
)

def abort(self, uuid, runlog_uuid):
return self.connection._call(
self.ABORT_RUN.format(uuid, runlog_uuid),
verify=False,
request_json={},
method=REQUEST.METHOD.POST,
)

def clone(self, uuid, clone_payload):
return self.connection._call(
self.CLONE.format(uuid),
verify=False,
request_json=clone_payload,
method=REQUEST.METHOD.POST,
)

def import_json(self, provider_json):
return self.connection._call(
self.IMPORT_JSON,
verify=False,
request_json=provider_json,
method=REQUEST.METHOD.POST,
)

def export_json(self, uuid):
return self.connection._call(
self.EXPORT_JSON.format(uuid), verify=False, method=REQUEST.METHOD.GET
)

def export_file(self, uuid, passphrase=None):
if passphrase:
return self.connection._call(
self.EXPORT_FILE.format(uuid),
verify=False,
method=REQUEST.METHOD.POST,
request_json={"passphrase": passphrase},
files=[],
)
return self.connection._call(
self.EXPORT_FILE.format(uuid), verify=False, method=REQUEST.METHOD.GET
)

def export_provider(self, uuid, passphrase=None):
current_path = os.path.dirname(os.path.realpath(__file__))
if passphrase:
res, err = self.connection._call(
self.EXPORT_FILE.format(uuid),
verify=False,
method=REQUEST.METHOD.POST,
request_json={"passphrase": passphrase},
files=[],
)
else:
res, err = self.connection._call(
self.EXPORT_FILE.format(uuid), verify=False, method=REQUEST.METHOD.GET
)

if err:
raise Exception("[{}] - {}".format(err["code"], err["error"]))

with open(current_path + "/" + uuid + ".json", "wb") as downloaded_file:
for chunk in res.iter_content(chunk_size=2048):
downloaded_file.write(chunk)

return current_path + "/" + uuid + ".json"

def upload_using_import_file(self, payload, files):
return self.connection._call(
self.IMPORT_FILE,
verify=False,
files=files,
request_json=payload,
method=REQUEST.METHOD.POST,
)

def upload_with_decompiled_secrets(
self,
provider_payload,
passphrase,
decompiled_secrets=[],
):
"""
Used to create a provider if it contains encrypted secrets from decompilation

Args:
provider_payload (dict): payload of the provider
passphrase (string): passphrase for creating provider with secrets (it should be same as the one provided while decompilation)
decompiled_secrets (list): contains all the secrets that were present in the decompiled provider
"""
secret_map = {}
secret_variables = []
not_stripped_secrets = []
provider_name = provider_payload["spec"]["name"]
provider_resources = provider_payload["spec"]["resources"]
LOG.debug("provider_resources pre-stripping secrets")
LOG.debug(provider_resources)
strip_provider_secrets(
provider_name,
provider_resources,
secret_map,
secret_variables,
decompiled_secrets=decompiled_secrets,
not_stripped_secrets=not_stripped_secrets,
)
LOG.debug("provider_resources post-stripping secrets")
LOG.debug(provider_resources)

strip_uuids(provider_resources)
LOG.debug("provider_resources post-stripping UUIDs")
LOG.debug(provider_resources)
provider_payload["spec"]["resources"] = provider_resources
files = {"file": ("file", json.dumps(provider_payload))}
res, err = self.upload_using_import_file(
{"name": provider_name, "passphrase": passphrase}, files
)
if err:
return res, err

# Add secrets and update provider
provider = res.json()
del provider["status"]
LOG.info("Patching newly created/updated secrets")
for k in secret_map:
LOG.debug("[CREATED/MODIFIED] credential -> '{}'".format(k))
for s in secret_variables:
LOG.debug("[CREATED/MODIFIED] variable -> '{}' path: {}".format(s[2], s[0]))

patch_secrets(
provider["spec"]["resources"],
secret_map,
secret_variables,
not_stripped_secrets,
)
LOG.debug("Update provider payload:")
LOG.debug(provider)

# Update provider
update_payload = provider
uuid = provider["metadata"]["uuid"]
return self.bulk_update(uuid, update_payload)
20 changes: 20 additions & 0 deletions calm/dsl/api/resource_type.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@ class ResourceTypeAPI(ResourceAPI):
def __init__(self, connection):
super().__init__(connection, resource_type="resource_types", calm_api=True)
self.CREATE = self.PREFIX
self.LIST = self.PREFIX + "/list"
self.TEST_RUNBOOK = self.PREFIX + "/{}/test_runbook/{}/run"
self.PLATFORM_LIST = self.PREFIX + "/platform_list"
self.UPDATE = self.PREFIX + "/{}"
self.TEST_EXECUTE = self.PREFIX + "/{}/actions/{}/test_run"

def create(self, resource_type_payload):
return self.connection._call(
Expand All @@ -19,6 +21,16 @@ def create(self, resource_type_payload):
timeout=(5, 300),
)

def list(self, payload={}):
if not payload.get("length"):
payload["length"] = 20
return self.connection._call(
self.LIST,
verify=False,
request_json=payload,
method=REQUEST.METHOD.POST,
)

def update(self, uuid, resource_type_payload):
return self.connection._call(
self.UPDATE.format(uuid),
Expand All @@ -27,6 +39,14 @@ def update(self, uuid, resource_type_payload):
method=REQUEST.METHOD.PUT,
)

def run(self, resource_type_uuid, action_uuid, payload):
return self.connection._call(
self.TEST_EXECUTE.format(resource_type_uuid, action_uuid),
request_json=payload,
verify=False,
method=REQUEST.METHOD.POST,
)

def run_test_runbook(self, resource_type_id, action_id, payload):
return self.connection._call(
self.TEST_RUNBOOK.format(resource_type_id, action_id),
Expand Down
23 changes: 23 additions & 0 deletions calm/dsl/api/runbook.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
import os
import sys
from distutils.version import LooseVersion as LV


from .resource import ResourceAPI
from .connection import REQUEST
from .util import strip_secrets, patch_secrets
from calm.dsl.config import get_context
from calm.dsl.log import get_logging_handle
from .project import ProjectAPI

LOG = get_logging_handle(__name__)


class RunbookAPI(ResourceAPI):
def __init__(self, connection):
Expand All @@ -34,6 +38,7 @@ def __init__(self, connection):
self.MARKETPLACE_EXECUTE = self.PREFIX + "/marketplace_execute"
self.MARKETPLACE_CLONE = self.PREFIX + "/marketplace_clone"
self.VARIABLE_VALUES = self.ITEM + "/variables/{}/values"
self.CLONE = self.PREFIX + "/{}/clone"

def upload(self, payload):
return self.connection._call(
Expand Down Expand Up @@ -435,3 +440,21 @@ def variable_values(self, uuid, var_uuid, payload={}):
return self.connection._call(
url, verify=False, method=REQUEST.METHOD.POST, request_json=payload
)

def clone(self, uuid, payload):
from calm.dsl.store.version import Version

calm_version = Version.get_version("Calm")

if LV(calm_version) < LV("4.0.0"):
LOG.error(
"Runbook clone is supported from Calm version 4.0.0. Please upgrade your Calm version to use this feature."
)
sys.exit("Runbook clone is supported from calm version 4.0.0 onwards")

return self.connection._call(
self.CLONE.format(uuid),
verify=False,
request_json=payload,
method=REQUEST.METHOD.POST,
)
Loading
Loading