custom registry variants

[mshanemc]

You know how CustomObject is a metadata API type, but the CLI breaks it down into individual files for CustomField, ValidationRule, etc? But other types look the same as the mdapi uses, except the extension?

That's because of the Metadata Registry which defines how all the types should behave.

We have a number of requests to do that process differently.
#1159
#2356
#2376
forcedotcom/source-deploy-retrieve#1145 (the PR is using a different strategy, but Registry Variants would get most of the way where it's headed--if a Profile is retrieved without certain elements, those files would not be deleted)

Changing these is painful (breaking changes to how deploy/retrieve behave, etc). And trying to get consensus on these among customers is going to be hard ("hey, it was fine before...why'd you change i!?")

my idea: put your own registry fragments in your sfdx-project.json

Use case: 2376 wants to break permissionSets into individual files.

{
  // inside sfdx-project.json
  metadataRegistry: {
    "types": {
      "permissionset": {
        "id": "permissionset",
        "name": "PermissionSet",
        "suffix": "permissionset",
        "directoryName": "permissionsets",
        "inFolder": false,
        "strictDirectoryName": false
        "children": {
          "types": {
            "permissionsetclassaccess": {
              "id": "permissionsetclassaccess",
              "name": "PermissionSetClassAccess",
              "directoryName": "classAccesses",
              "suffix": "classAccess"
            },
            // you get the idea...decompose as many of the child types as you want
        },
        // strategies refer to a set of common patterns in SDR for decomposing/recomposing metadata.  
        "strategies": {
          "adapter": "decomposed",
          "transformer": "decomposed",
          "decomposition": "folderPerType"
        }
    } 
  }
}

Obviously, we'd have several tasks to do:

1. good public docs about what the strategies/adapters/transformers are and how to use them
2. some "recipes" for common patterns (ex: break something into files like CustomObject does). And make it possible for y'all to share recipes, too!
3. Have a command that will validate your custom registry (same rules and logic that SDR uses on the entire registry)

Pro

1. Make it do what you think it should have done
2. Anyone who works on your project gets the same behavior since this is checked in as part of the project config
3. Use unsupported types earlier (joined some weird pilot that's not official in the mdapi yet? Some new release feature added to the CLI but you can't wait 2 weeks for it to get through RC into latest? You can make some custom registry types to handle those!)
4. [Salesforce internal: easier testing and contribution of new types ahead of the release...no need to link plugins and libraries to try out your changes]. For externals, that's "it's more likely to work."
5. All deploy operations are already reading the file, so it's not another fs operation (perf). There's just the JSON merging
6. Reduce some git diff noise on large xml files (easier merge conflicts)

Con

1. Complexity! More ways to screw up your metadata but blame it on the CLI 🤪. It will be harder to help with those issues, and we'll need your customRegistry stuff with issue reports.
2. You might get used to something in your main project, and have context-switching when working on another project that isn't configured like yours
3. Other tooling. VSCode and DevOpsCenter would get this automatically (maybe with a bit of lag due to release cadences), but if you use other tools they might not (depending on how they're calling CLI commands/libraries if at all).
4. 🐔 🥚 with changing them. You'd have to change your project files and sfdx-project.json together, otherwise, you could have an undeployable combo.

[amtrack]

Sounds interesting!
Regarding CON-4. : could there be a new „convert“ command which transforms metadata between standard registry and custom registry or other variants offline?

[mshanemc]

hmm...at first glance that sounds hard because of how SDR is built. I'd have to do some experimentation to see how you'd load 2 of them at once.

Maybe in-memory you could convert from old source to mdapi and then flip registries to convert it back? It'd probably be something hacky like that.

[daveespo]

The few example 'feature requests' that you linked to are probably 90% of the reasonable use cases for decomposed metadata. And to be clear: Permission Sets and Custom Labels are the only ones you should spend much time thinking about. Profiles are the third rail for a million other reasons, and both Workflow Rules and Profiles are on a slow path to deprecation.

I think the Cons are pretty significant. It's like adding a million feature flags all at once and if history is any guide with regard to the CLI and edge cases around metadata support, this will make for a never ending gift of problem reports and difficult to reproduce unexpected behavior. (That wasn't meant to be a jab -- just being realistic)

Isn't the sourceApiVersion attribute in sfdx-project.json meant to describe the 'shape' of the source files in the project? Couldn't a versioned change be introduced to change the format of some of the aforementioned metadata types (and others in the future) so that we all get to a common decomposed source representation over time? I know that the sourceApiVersion is a big chunky number that only changes 3 times per year so that does slow down the rate that you could deliver new source formats. Perhaps you could get creative using a different versioning system for this sort of feature delivery?

In summary: The items listed in the Pros section sound mostly of interest to people who are knee deep in CLI plugin development, not to end users of the CLI who aren't immersed in the nuances of metadata types and have no interest in becoming immersed :-)

[mshanemc]

I'd also expect people to eventually complain about

• matchingrules
• things that have versions inside them and become huge (flow, bots,aiscoringmodeldefinition, form/formsection, decisionmatrixdefinition, etc).

[mshanemc]

Isn't the sourceApiVersion attribute in sfdx-project.json meant to describe the 'shape' of the source files in the project

That means the metadata API version that we're using. That number gets puts in the package.xml that goes to the mdapi once the project is converted. We wouldn't want to overload that with intent about how to decompose/recompose source.

source format has never really been a "spec" and it's always meant, "whatever the CLI is doing". Tightening that up and versioning these changes as something like converterVersion would definitely be possible, where everything to date has been 1.0 and we do it differently in 2.0.

[AllanOricil]

This feature is going to be awesome.
I just think that the schema to enable it could be simplified more, if possible.

[AllanOricil]

I see these transformations as some kind of prettier/lint tool for how xml files could be decomposed/composed. Since xml diffs are something difficult in git, maybe this could become a new standalone tool that would work on any type of xml schema.

Input: set of rules to instruct the tool to break xml into files and/or subdirectories in a json format.

[aidan-harding]

Having Labels, Permission Sets, Profiles, etc. decomposed would be really great. As @daveespo said, if you cover those, then most folks will never have to go any further than that.

Maybe if those particular metadata types had some first-class support with flags to enable decomposing them rather than directly including bits of the metadata registry config, this would make it more friendly for 99% of users and suggest which cases are important for other tooling to support.

Tooling support is absolutely the one to watch here for me. One of the triumphs of DX and the CLI is the common source format. Today, I can easily switch my tooling choices and that's fine because everything is in DX format in git so the method of getting metadata in and out is fairly interchangeable. For example, as a consultancy, we can come in and build a project for a customer using tooling that's as expensive as we're willing to pay for. And, because we leave them with a repo in SFDX format, they can do their day-to-day maintenance/improvements with DevOps Centre or an alternative expensive tool. Our choices don't end up coupled.

Constraining that tooling choice is something that I'd be very wary of.

[mshanemc]

DevOpsCenter also uses the SDR library, so the change should work there. I don't have enough knowledge of other tooling to know how they support source format...if they're using the CLI it should all be fine.

[FabienTaillon]

I really love that ! Lot's of issues with XML merging, creating conflicts that should not be conflicts, especially around picklist values, and record types.

One thing I'm wondering is, is there any impact on packaging ?
Meaning, even though on the Metadata API side CustomFields are part of CustomObject big xml files, they can be separated in 2 projects and packaged separately. Same for CustomLabels. But I assume this won't be possible for everything. Would packaging commands recreate the metadata file before ? I assume that's how it works today ?

[mshanemc]

Yeah, on CustomObject etc they merge the files into one file to send up, regardless of the local directory structure. These should work similarly.

But like CustomObject you might need an "empty" top-level/parent in both directories to make it work right.

[scolladon]

I think it is a very good idea.
It will ease the life of many people:

• Developer because it will be easier to understand what is changed in the files. No more huge file
• Code reviewer for the same reason.
• Ops because it will be easier to understand what is shipped and issues will be easier to locate.

I've always wondered why doing this only for CustomObject and not for everything else 🤔

Focus on the impacted metadata types

In sfdx-git-delta we treat all those whole metadata definition located inside a holder file with a specific transformer so we can deal with them unitarily and recompose the holder file with only what as changed between 2 revisions. Here is the list of kind of metadata we have identified during our exploration:

• assignmentRules
• autoResponseRules
• escalationRules
• globalValueSetTranslations (only scope the file with what as changed)
• labels
• matchingRules
• objectTranslations (only scope the file with what as changed)
• profiles (only scope the file with what as changed)
• sharingRules
• standardValueSetTranslations (only scope the file with what as changed)
• translations (only scope the file with what as changed)
• workflows

Some of the metadata are just "scopeable" (they can be deployed partially) but cannot be listed in the package.xml.

permissionsets does not appear in this list because they are not "scopeable", their content reflect exactly what it contains in the org when deployed since v40

Focus on Pros and Cons

As usual, the complexity lives in the details 😈 , I cannot foresee what issues it could have in the future with new metadata kind and specific customer usage, but here are my thoughts:

• PRO2 can lead to complex merge with long running branches if this conf changes in the target branch. It happens on huge customer program where each stream have its own branch for a long time and merging is used only to deploy (git flow use case)
• PRO3 I love it, though it could be driven byt the API version attribute
• PRO4 could be a Cons for end users. Because it can introduce breaking change which according Murphy's law it always land in the worst most interesting time.
• All CONS: I totally agree, it could lead to RTFM/WAD frustrating level 3 investigation just before huge customer go live.
• Another CONS I would like to add, it may be complex for sf plugins to adapt to this behavior and provide the same feature as before considering all the combinations possible. I'm totally biased on this one 😄

My 2 cents

I would apply the "CustomObject" metadata content split pattern to every metadata kind listed above, without exposing a way to customize it for customers. And I would do the same for every new metadata that can fall in this pattern.
It could be gated by a flag in the sfdx-project.json (preferred) or a parameter at the command line or by the API version attribute in the sfdx-project.json.
I think this way to split the metadata content is much more easier to use than having everything in one file, and so I think it should be the pattern applied by default as the source v2 format.

[FabienTaillon]

Oh yes you're right, I didn't thought about the custom plugins that may not find the files they expect !

[RupertBarrow]

Yes Fabien, they would have to be sure to find them by always working with SDR.

I think the merging nightmare on long-running branches, mentioned above, would make this a no-go solution, unless strictly managed by the SDR library.

[nvuillam]

It seems interesting, while it does not require additional configuration (metadata split should be opiniotated -> same for everyone activating the option), and more important: respect ascending compatibility with existing projects :)

[catalandres]

If we are going to open up an interface to allow for the customization of how metadata files are decomposed and synthesized, we would be ultimately making room for any arbitrary transformation to any metadata component, including going from XML to other, human-friendlier formats.

Even if transformation is in the far-future backlog, in order to handle the synthesis-transformation-decomposition, we need a stronger semantic layer for metadata (robust TypeScript types for metadata, wink wink) that is used by all plugins, instead of relying on a canonical representation in XML.

[RupertBarrow]

Yes !

And if you do so, please specify ordering, sorting rules to make diffs and merging xml-meta easier.

[alan-morey]

💯 Ordering/Sorting
Such a pain dealing with XML diffs where order of elements is the only difference, otherwise the files are equivalent.

Also, some metadata components when retrieved seem to change order of elements each time they are retrieved, e.g. <fieldMappings> in CleanDataService.

[mcarvin8]

@mshanemc - One benefit of using the General XML Disassembler package I developed would be ensuring consistent sorting. That's one thing I added to the package to ensure an XML is always sorted in the same way when decomposed and recomposed, based on the file names (which come from the unique ID elements for nested elements).

I forget which meta type I've seen this issue for, but I have seen sorting be different when retrieving the same meta from an org. Something like where rules 10 and higher would sometimes be placed before rules 1-9 when retrieving one time and in other retrievals 10 and higher would be after 1-9.

But based on how the XML is disassembled and named using a unique ID element, my package would ensure it uses file names to sort them when recomposing them. One of the tests I run in this package is comparing contents to "baselines" to ensure sorting is always done in the same way.

I haven't tested this extreme use case yet, but I'm sure if you processed the right sub-directories and used the --postPurge flag in the disassembler package, you could probably disassemble an XML further down til a desired Depth Level is reached?? It's something I've been thinking about if a user wanted to disassemble an XML further past the default depth level of 1. That's the part of the reason i added some delete/purge flags that allow a user to delete the original XML file after disassembling it or delete the disassembled directories after reassembly. That would allow a user to do this advanced case assuming they handle passing the correct XML file paths without needing to add this to the package itself.

[mshanemc]

a little teaser

[nvuillam]

Quick Question: Won't it multiply the number of files uploaded in a single deploy, and might force us to split deployments to not reach the 10000 files on big orgs ?

[FabienTaillon]

I think that like for objects, the CLI will convert to the expected Metadata file format before deploying, and thus won't change the number of files deployed.

[ls-jcflores]

When is gonna be available:

 "registryPresets": [
        "decomposePermissionSet"
    ]

Thanks

[mshanemc]

it's in the nightly now (note: it's named decomposePermissionSetBeta so you can try it and give feedback before we make it official). Watch for the release notes on Wednesday.

[mcarvin8]

Not trying to step on anyone's feet here, but I did write up a custom plugin that handles decomposing most of the metadata listed throughout this chain. I started this as a pet project in Python for my own team and then I wanted to take a stab at creating a CLI plugin for the 1st time.

I didn't see this thread till after the fact. Willing to talk to someone on the Salesforce CLI team if there is any overlap/conflicts or any desire to contribute to my plugin - https://github.com/mcarvin8/sfdx-decomposer-plugin

We could keep this concept as a plugin and integrate CLI hooks to automate the conversion back and forth (@AllanOricil 's idea not mine). Keep the CLI structure the same and just have people install a plugin if they wish to decompose specific metadata further.

[mshanemc]

@mcarvin8 the plan is to put this in SDR library so that VSCode, DevOpsCenter, etc (things that aren't always using CLI/plugins) get the same behavior.

And the tracking/conflict equivalents (oh, you edited FooPermSet/etc/etc ? I know that that permset is modified and needs to be deployed when you deploy changes)

[mcarvin8]

@mshanemc - If you would like, feel free to use the XML-disassembler package @AllanOricil is referring to above (assuming you have something similar already but if you want to remove that from your code base and just import this package to simplify your code base - I basically just did that myself by moving a good junk of the sfdx-decomposer code base to the xml disassembler).
https://www.npmjs.com/package/xml-disassembler

Here is how I'm using the xml disassembler in my sfdx decomposer plugin (specifically for custom labels and metadata in sub-folders like bots):

Disassemble Use Case
Reassemble Use Case

With the --postpurge flag, you could run my package on the original metadata files and delete them after you're done disassembling them into smaller files. That way, you could keep how the CLI retrieves the metadata files like Custom Labels, Workflows ,etc. for those who do not wish to use a Custom Registry and just run the XML Disassembler as an additional function when someone wants to use a Custom Registry.

[mshanemc]

@mcarvin8 I checked it out, pretty cool, but it didn't quite align with how SDR needs to work (and we already had most of this defined for CustomObject and its children).

[mcarvin8]

Gotcha, yea big thing I noticed that SDR does with decomposed files is adding the child's XML root element name to the file.

<CustomFieldxmlns="http://soap.sforce.com/2006/04/metadata">

Which makes sense if SDR is reading the decomposed files directly over reassembling them into parent files behind the scenes. I briefly had that feature in my XML disassembler package to allow users to override the default XML root elements in the files if they needed to for this type of use case. But if the goal is to allow XMLs to be broken down and reassembled, maintaining the same root element made more sense.

Guess the concern for SDR would be the added complexity of maintaining the original registry structure and the custom registry structure. Is there any plans in place for the future to migrate to the custom registry structure similar to Custom Objects? Think it would make sense for long term to use the custom object registry structure for various meta types like labels, workflows, etc. as the default structure. Just would need to slowly phase out the current structure.

[mshanemc]

We've talked about having sf project generate use some presets by default.

Otherwise, flipping it becomes this weird breaking change where some projects don't work with newer CLI version, and it's a really bad experience.

[jonathanwiesel]

Any sneak peek we can get on the roadmap of types to support?

I would like to bring forward this other issue that captured some types that would also be interesting, the following ones would be the remaining of the mentioned ones leaving out the ones already covered in the beta

• Flows
• Layouts
• Flexipages
• Approval Processes

[mshanemc]

Yeah, I could actually all of those being reasonable candidates for this.

[mshanemc]

any arguments against these 4 mdTypes?

[RupertBarrow]

DigitalExperienceBundle ? ... although this would be a different type of job.

[scolladon]

I think we should distinguish what is listable atomically (like CustomLabel) in package.xml file (and deployable atomically), from what is contained inside a broader metadata type and can be deployed atomically (like Profile sub types), from what is contained inside a broader metadata type (like PermissionSet) (and not deployable atomically without having an impact on other sub types definition), at least at the registry level.

We could distinguish them as ListableDecomposed, StandaloneDecomposed and ContainedDecomposed "kind".

• ListableDecomposed kind have there subtype listable in the package.xml and deployable atomically.
• StandaloneDecomposed kind have there subtype not listable in the package.xml but deployable unitarily. Meaning building a file with just this subtypes, listing the main type in the package.xml and deploy it will just update this sub type and not the others the main type contains.
• ContainedDecomposed kind have special behavior.
  • It is not possible to deploy this type partially. Meaning deploying only the sub file that has changed will erase every other sub type definition not present (try with PermissionSet)
  • Sub Types cannot be listed in the package.xml (it is not a type) either for retrieve or deploy action
  • Using delta deployment when a change is detected is tricky because:
    • other subTypes (not changed) should be deployed as well to avoid erasing them
    • deletion of a sub type file should be treated as a modification if other sub types exists (the related main types should be listed in the package.xml)
    • deletion of a sub type file should be treated as a deletion if there is no other sub types files (the related main types should be listed destructiveChanges.xml)

Per example, we could categorize the existing ones the following way:

• Workflow would be ListableDecomposed as WorkflowRule and other sub type contained in a Workflow file can be listed in the package.xml and deployed atomically (by just having this metadata def in the workflow file)
• CustomLabel would be ListableDecomposed as CustomLabel can be listed in the package.xml and deployed atomically (by just having this metadata def in the label file)
• SharingRuleswould be ListableDecomposed as SharingRule subtypes can be listed in the package.xml and deployed atomically (by just having this metadata def in the sharingRules file)
• PermissionSet would be ContainedDecomposed as any first level sub attribut of a permission set is not deployable atomically and means the whole permission set should be deployed every time a sub type changes.

I think we should consider those mdtypes as well to be Decomposed, with this categorization:

• AssignmentRule as a ListableDecomposed
• AutoResponseRule as a ListableDecomposed
• EscalationRule as a ListableDecomposed
• MatchingRule as a ListableDecomposed
• GlobalValueSetTranslation as a StandaloneDecomposed
• StandardValueSetTranslation as a StandaloneDecomposed
• Profile as a StandaloneDecomposed
• Translation as a StandaloneDecomposed
• CustomObjectTranslation as a StandaloneDecomposed

[RupertBarrow]

Don't forget CustomField which is part of CustomObject, and all the others.

When you use the Metadata API to list the contents of a Package which is in fact a Change Set, and this Change Set contains, for example, a CustomField, well the Metadata API result refers only to the Custom Object to which that field belongs. You need to sf cli retrieve the Change Set/Package to find the effective Custom Fields it refers to.

[mshanemc]

we have a few props in the SDR registry that control these behaviors.

isAddressable = false => this child type can't deploy on its own and has to be deployed inside the parent. The logic around those use .every() to determine that ALL the child types are isAddressable=false. It roughly maps to your ContainedDecomposed

unaddressableWithoutParent = true => CustomObjectFieldTranslation where it is addressable (can have package.xml entry) but it has to be deployed with the parent. We use this to also say, "if you retrieve one, you have to put in the same project as its parent since it can't deploy alone". Maybe that's roughly StandaloneDecomposed

your ListableDecomposed is the default (if isAddressable isn't set to false). I've found these to be troublesome, like sometimes it's simpler to conservatively say, isAddressable = false and always build up the parent file and deploy/retrieve that. Even though you could deploy/retrieve an individual child.

[azlam-abdulsalam]

@mshanemc are you saying, even if its child can be deployed individually, deploy the whole for ListableDecomposed

[mcarvin8]

While testing decomposePermissionSetBeta with today's latest CLI , I get this Metadata API request failure when initially retrieving all of our permission sets from a rather large org. Permission sets do get retrieved, but I'm unable to tell if all of our permission sets in the org were retrieved and then decomposed.

sf project retrieve start --metadata PermissionSet
Retrieving v60.0 metadata from ..... using the v60.0 SOAP API
Preparing retrieve request... MetadataTransferError
Error (1): Metadata API request failed: Maximum call stack size exceeded

[mshanemc]

ooh, thanks for reporting that. how many permissions sets are we talking about?

[mcarvin8]

our org has 1191 permission sets, which is way too many but that is a different problem (lots of tech debt)

EDIT: I double-checked the # of permission sets we actually had retrieved into our git repo and that total is 519. Assuming the other permission sets in our org come from packages, etc.

But I still have these issues with retrieving all 519 we have with the decompose beta enabled.

Takes a few retrieval attempts to get everything retrieved.

[mcarvin8]

With the shear # of permission sets we have in this org, decomposing all of our permission sets doesn't seem feasible since it increases the size of our repository greatly and I can't even push the shear number of changes over SSH without it being rejected due to the size of the transfer.

The only workaround I've done to address the git push rejections is to make multiple commits that commit a subset of permission sets.

So this is definitely an extreme use-case but I'm curious if other users are facing this in other larger orgs.

For other orgs with similar tech debts, that would be the argument for leaving Permission Sets in the default registry structure (1 mega XML file for each perm set) and providing the decomposed registry as an alternate for those who don't have large orgs.

[mshanemc]

I can't even push the shear number of changes over SSH without it being rejected due to the size of the transfer.

I didn't even know there was a limit there!

[mcarvin8]

@mshanemc - Can we look into updating the Custom Labels custom registry to not add decomposed labels into a new CustomLabels sub-directory? I would prefer they just get decomposed directly in the labels directory just to decrease the path size.

This is something I ran into with my decomposer plugin, which I resolved by doing some file-moving operations with the labels files during decomposing and recomposing.

[mshanemc]

I tried...a LOT. It's possible, but it really twists the framework in a direction we're trying to get away from (right now CustomLabels are have a lot of their own code and carve-out behaviors) and I'd rather not have that). I think I'd rather have them behave like all the other types for code simplicity/testability. The number of edge cases we've had with CustomLabel(s) has been awful.

Anyone else bothered by an extra folder?

[VivekMChawla]

@mcarvin8 - We're finalizing the design for decomposing Custom Labels.

Please review and vote on the updated design here: Source Decomposition Options for CustomLabels.

Thanks!

[VivekMChawla]

Closing Remarks

Thank you all for participating in this discussion. Your ideas and perspectives greatly contributed to finalizing this feature.

Final Design

We decided not to support custom registry variants. Instead, some metadata types will have a fixed source decomposition option that users can choose to enable on a project-by-project basis. This ensures consistency across SFDX projects that enable these options, simplifying troubleshooting, support, and maintenance of the Salesforce CLI.

As a result, it's more important than ever that we get the design right for new source behaviors offered by the CLI.

Process for Releasing New Source Behavior Options

1. The CLI Team will propose source decomposition designs for specific metadata types, soliciting feedback from the community for a period of time.
   • You can see this in action with the Custom Labels metadata type.
2. At the end of the review period, the finalized design for that type will be implemented.
3. When released, decomposition for that type will be available as a BETA source behavior option.
4. During the BETA period, feedback will be gathered from users who opt-in to the new source behavior option.
   • During the BETA period, changes to how a type is decomposed may happen, depending on user feedback.
5. Once a source behavior option goes GA, its design is set and will not be modified.

Thank You!

Thanks again for your collaboration. We're incredibly grateful to this community for your help and support!