Feature/1325 add multi type support and metadata option to buildtemplate and builddefinition

[JoernBerkefeld]

PR details

What changes did you make? (Give an overview)

• closes [FEATURE] add --metadata option to buildTemplate and buildDefinition #1325
• closes [BUG] buildTemplate and buildDefinition do not handle unknown keys gracefully #1296

Further details (optional)

...

Checklist

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• test scripts updated
• Wiki updated (Feature/1325 add multi type support and metadata option to buildtemplate and builddefinition #1327 (comment))

[github-actions]

Coverage Report

Commit:b039032
Base: develop@65257cb

Type	Base	This PR
Total Statements Coverage	67.24%	67.27%  (+0.03%)
Total Branches Coverage	66.61%	66.82%  (+0.21%)
Total Functions Coverage	77.54%	77.7%  (+0.16%)
Total Lines Coverage	67.24%	67.27%  (+0.03%)

Details (changed files):

File	Statements	Branches	Functions	Lines

[JoernBerkefeld]

https://github.com/Accenture/sfmc-devtools/wiki/06.c-~-Templating-Commands/_edit

To create templates, one can use retrieveAsTemplate (rt - this is now deprecated) or buildTemplate (bt). Choosing one or the other command depends on your use case. rt needed to be implemented per type, and we never got around to that - which means there are types for which it wasn't implemented. bt on the other hand, relies on you executing retrieve (r) first and then works for all types you downloaded. It's more generic because it doesn't need to handle downloading a single file.

If you want to use templates, then you have essentially three (+1) ways:

1. Use the commands r, bt, bd/bdb, d
2. Use r, then manually copy & adapt files from retrieve/ to template/, then run bd/bdb, d
3. (deprecated) Use the commands rt, bd/bdb, d

Option 4 is also available and relies on you using a git repo and the commands createDeltaPkg & deploy together. That's mostly for CI/CD-enabled projects, but there are teams that use it manually as well.

buildTemplate

Command: mcdev buildTemplate <business unit> [type] [key] [market] [--metadata] --market

Alias: mcdev bt

The bt command uses previously retrieved metadata on your local computer and uses your market configuration in .mcdevrc.json to replace strings with variables. The result is then stored in your template/ folder. Please note that files stored here will keep their original name, despite this possibly containing market-specific suffixes or similar. Also note, that contrary to the deploy & retrieve folders, you will not see credential- or Business Unit-sub-folders here.

This command is a prerequisite for the buildDefintion command. Alternatively, you can copy-paste retrieved metadata from your retrieve/ folder to your template/ folder and update it manually - or even create it from scratch.

Note: Before using this command, you need to configure your markets first! Check out our guide on Market Configuration to understand how to use templating and prepare your market config.

Currently supported types: Check out Metadata Type Support.

Example:

mcdev bt MyProject/DEV dataExtension MyUserTable pilotMarketDEV1
mcdev bt MyProject/DEV --market pilotMarketDEV1 -m dataExtension:MyUserTable

This will result in MyUserTable.dataExtension-meta.json being created in your template/ directory:

buildTemplate for multiple sources:

You can also create multiple templates with multiple sources at once. Simply specify them in a comma-separated list and put that list in quotes:

mcdev bt MyProject/DEV dataExtension "table1,table2,table3" pilotMarketDEV1
mcdev bt MyProject/DEV query "sql1,sql2" pilotMarketDEV1

mcdev bt MyProject/DEV --market pilotMarketDEV1 -m dataExtension:table1 -m dataExtension:table2 -m dataExtension:table3 -m query:sql1 -m query:sql2

This will result in the following files being created in your template/ directory:

• table1.dataExtension-meta.json
• table2.dataExtension-meta.json
• table3.dataExtension-meta.json
• sql1.query-meta.json + sql1.query-meta.sql
• sql2.query-meta.json + sql2.query-meta.sql

buildDefinition

Command: mcdev buildDefinition <business unit> [type] [template name] [market] [--metadata] --market

Alias: mcdev bd

The buildDefinition command allows to prepare the deployments to one or multiple targets based on templates and Market Configuration.
After you have created your templates via retrieveAsTemplate (or manually) in your template/dataExtension/ folder run this command to create the final deployable files in your respective deploy/<business unit>/ folder.

This allows you to double-check if you changed something by comparing the before and after using your favorite Git client. You then have to manually copy the files you want to deploy into the respective deploy/ folder.

Note: Before using this command, you need to configure your markets first! Check out our guide on Market Configuration to understand how to use templating and prepare your market config.

Currently supported types: Check out Metadata Type Support.

Example:

mcdev bd MyProject/QA dataExtension MyUserTable pilotMarketQA1
mcdev bd MyProject/QA --market pilotMarketDEV1 -m dataExtension:MyUserTable

This will result in the following files being created in your deploy/MyProject/QA/dataExtension/ directory:

• MyUserTable.dataExtension-meta.json

buildDefinition for multiple sources:

You can also create definitions based on multiple templates at once. Simply specify them in a comma-separated list and put that list in quotes:

mcdev bd MyProject/QA dataExtension "table1,table2,table3" pilotMarketDEV1
mcdev bd MyProject/QA query "sql1,sql2" pilotMarketDEV1

mcdev bd MyProject/QA --market pilotMarketDEV1 -m dataExtension:table1 -m dataExtension:table2 -m dataExtension:table3 -m query:sql1 -m query:sql2

This will result in the following files being created in your deploy/MyProject/QA directory:

• table1.dataExtension-meta.json
• table2.dataExtension-meta.json
• table3.dataExtension-meta.json
• sql1.query-meta.json + sql1.query-meta.sql
• sql2.query-meta.json + sql2.query-meta.sql

buildDefinitionBulk

Command: mcdev buildDefinitionBulk <market list name> <type> <template name>

Alias: mcdev bdb

With buildDefinitionBulk you get a very powerful command that wraps around buildDefinition, which it executes internally. It allows you to create the definitions for multiple Business Units and multiple markets at once.

Instead of passing in the name of credentials, Business Units and markets, you simply refer to a pre-defined market list in your .mcdevrc.json. This enables you to re-use the same configs over and over again, across multiple deployments and share it with your team.

Note: Before using this command, you need to configure your markets first! Check out our guide on Market List Configuration and Market Configuration to understand how to use bulk templating and prepare your market config.

Example:

mcdev bdb pilotMarketsQA dataExtension MyUserTable

retrieveAsTemplate

[DEPRECATED]

Command: mcdev retrieveAsTemplate <business unit> <type> <name> <market>

Alias: mcdev rt

The rt command retrieves metadata from the server and uses your market configuration in .mcdevrc.json to replace strings with variables. The result is then stored in your template/ folder. Please note that files stored here will keep their original name, despite this possibly containing market-specific suffixes or similar. Also note, that contrary to the deploy & retrieve folders, you will not see credential- or Business Unit-sub-folders here.

This command is a prerequisite for the buildDefintion command. Alternatively, you can copy-paste retrieved metadata from your retrieve/ folder to your template/ folder and update it manually - or even create it from scratch.

Note: Before using this command, you need to configure your markets first! Check out our guide on Market Configuration to understand how to use templating and prepare your market config.

Currently supported types: Check out Metadata Type Support.

Example:

mcdev rt MyProject/DEV dataExtension MyUserTable pilotMarketDEV1

This will result in MyUserTable.dataExtension-meta.json being created in your template/ directory:

retrieveAsTemplate for multiple sources:

You can also create multiple templates with multiple sources at once. Simply specify them in a comma-separated list and put that list in quotes:

mcdev rt MyProject/DEV dataExtension "table1,table2,table3" pilotMarketDEV1

This will result in the following files being created in your template/ directory:

• table1.dataExtension-meta.json
• table2.dataExtension-meta.json
• table3.dataExtension-meta.json