Align metadata between orgs and/or projects

[mshanemc]

Align metadata between orgs and/or projects

Use cases

1. Given an org and a project, what is different between the org and my project?
2. Given two orgs, what is different between them?
3. What would I have to do to make one side match the other?

Why (examples)

1. some metadata is allowed to change in production (ex: list views, reports, email templates) and those changes need to be identified and added to version control and/or “lower” environments
   1. And you don’t want to deploy your “old” copy of those items over the production changes, so you need those updates
2. Customers - even those who try to follow source driven dev best practices - still have admins that will go into the production org and make changes.
   1. In this case, the developers wouldn't know it and those changes could be clobbered by the next dev deployment.
3. Refreshing a sandbox (esp full) can be a long process, and you lose work done there. But you need to ensure that the sandbox matches prod metadata

Relationship with source tracking

• source tracking only works on a subset of orgs (not prod org, not some sandbox types)
• source tracking does not cover Use Case 3
• source tracking starts at “zero” after a scratch org is created, even though the org has lots of metadata in it

Design Proposal

• specify which side of the diff is the “target” that you intend to modify ( org | project )
• produce manifest files based on Use Case 1,2 that would allow metadata operations to achieve Use Case 3
  • the manifest file(s) will represent what you would deploy/retrieve to get the org and project aligned
  • the package.xml file would include anything that doesn’t exist in the “target” or doesn’t match the “target”
  • this could include a destructiveChanges.xml file if the target is an org and it has metadata not in the project
• produce a file with commands to run
  • needed to support non-manifest operations see Make a project match an org

Rationale

• We chose not to put these as options on deploy/retrieve operations, since users need to see what will happen
• Manifests are both organized/human-readable for review
• Manifests are executable by CLI, VSCode, and most other tooling
• The operations may fail for all sorts of reasons, and manifests simplify tracking, backup, and modifications
• By using the deploy/retrieve commands, the new command doesn’t have to be aware of or manage Source Tracking, but the final state of the org and project are “correct” for Source Tracking purposes if the org supports it.
• By using the deploy/retrieve commands, .forceignore is automatically handled consistently. ex: a full retrieve shows includes Profiles in the generated manifest, but the user has Profiles in .forceignore No profiles are retrieved/deployed/deleted/modified.

Limitations

• This all sits on top of the Metadata API. Its bugs, gaps, and limits will impact these commands, too.
• To get the state of an org will require a full retrieve. This is obviously going to take some time
• The resulting files may be unusable “as is” due to mdapi component limits (ex: a diff results in a manifest file that exceeds 10k components, and is therefore unusable)
• Will use exact-match files from the org. Space/indentation/xml order will be considered “different”. If your local project is doing style-lint on save, etc you may get false positives in the diff.
  • These are annoying, but should ultimately not impact the end result (extra components are deployed repeatedly)
  • Building logic to ignore whitespace (trivial) and to ignore xml node ordering (nontrivial)...we’re interested in how much value you see for those.
  • The commands could use flags to give you choices about how strict the comparisons should be
• The CLI has never had a solid handle on file move/rename, either local or remote. This won’t identify matching files with name changes, either.

Usage/Examples

Make a newly created project match an org

# output-dir defaults to a timestamp
$ sf project diff --modify project
wrote files to diff-1680528024

What’s in the output folder?

1. package.xml file (represents all metadata available in the org)
2. commands file will contain
   1. sf project retrieve start -x diff-1680528024/package.xml -o username@target.org (username is always specified for portability, even if the given command uses a default org or an alias)

Alternatives/enhancements:

1. Add an optional org flag to the project generate command for a better experience when starting a project with an existing org. This could also attempt to create the scratch-org-def.json based on the org’s settings

Make an org match a project

imagine the org has some metadata that the project does not

$ sf project diff --modify org
wrote files to diff-1680528024

What’s in the output folder?

1. package.xml file (will exclude anything already in the org with the same value as the project)
2. destructiveChanges.xml (you can decide if pre or post makes more sense. We’ll default to post as it’s more likely to be successful)
3. commands file will contain
   1. sf project deploy start -x diff-1680528024/package.xml --post-destructive-changes diff-1680528024/destructiveChanges.xml -o username@target.org

Make a project match an org

imagine the project has some metadata that the org does not

$ sf project diff --modify project --target-org someAlias
wrote files to diff-1680528024

What’s in the output folder?

1. package.xml file (will exclude anything already in the org with the same value as the project)
2. commands file will contain
   1. sf project retrieve start -x diff-1680528024/package.xml -o username@target.org
   2. rm force-app/main/default/classes/foo* force-app/main/default/lwc/cmpYouNeverDeployedButForgot

Make an org match another org

This is a two-step operation conducted “through” the local project.

1. Make the project match the “source org”
   1. sf project diff --modify project --target-org orgToCopyFrom
2. Then make the “destination org” match the project
   1. sf project diff --modify org --target-org destinationOrg

Alternatives considered

1. having a separate command to do this in one operation, with 2 flags for orgs
2. having the project diff command output an entire retrieved project from the “source” org into the output dir in mdapi format
   1. avoided because mdapi format doesn’t work with source tracking

The proposed design is simpler (re-use of the command) and has the positive side effect

• the project is in sync with the org after Step 2
• users can see the intermediate steps (what’s retrieving, deleting)
• users can manage discrete metadata operations (use —async? resume in the case of network failure or timeout?) that would be complicated if wrapped in 1 command.

If the user wants to diff 2 orgs without modifying a real project locally, they would create an empty project.

[ImJohnMDaniel]

@mshanemc -- Just level setting on terms:

• When you say "project", you are referring solely to the metadata local to the Salesforce CLI execution and/or "all files managed in packageDirectories defined by sfdx-project.json. Is that correct?
• The term "project" in no way implies "application" within an org; custom or otherwise.
• So your goal here is to take "all metadata elements of a source org" and sync that up with a target org. Is that correct?

[mshanemc]

yes, those are all the definitions I'm using.

[ImJohnMDaniel]

Thanks for clarifying.

[daveespo]

I have some PTSD about a "full org retrieve" and how useful this feature will be. Most admins and developers think of Custom Objects, LWCs, Apex Classes and a few other things as being "their work" but the moment you do the full org retrieve, you get a million metadata types and a million records within those metadata types that are part of the happy soup that no one understands.

Further, some of those metadata types don't roundtrip well, or at all, (per your point about the known MDAPI bugs) so it will be a heavy lift to filter out the ones that AREN'T relevant or material to the project.

As a somewhat relevant example: We try to pull the Settings metadata (using the * wildcard) from orgs at the beginning of each new Release Preview window so that we can apply the Release Updates in our scratch orgs so that developers can test with those Release Updates turned on. The exact same metadata that is retrieved from a source:retrieve will often not go right back into the same org with a source:deploy

I sympathize with the "Why?" on this feature proposal but I don't know how useful the final feature will be for any org of moderate complexity.

[ImJohnMDaniel]

+1 to @daveespo's comments here. Especially the concern over how useful this feature would ever me considering all of the hurtles inherent to Salesforce development.

[mshanemc]

yeah, you might end up with a rather large .forceignore file.

and for clarity, this is only about the metadata, not the records. Not even the records that should have been metadata (hi, orgWideEmailAddress) !

[SCWells72]

Along the lines of a "full org retrieve", there are some metadata types/objects that are inherently problemmatic. I supported that as a subscription type in IC for quite a while, and folks would run into one of three broad categories of issues:

1. They would get GACKs on retrieve of such a request.
2. The retrieve would succeed but a subsequent deployment attempt of the retrieved metadata to the exact same org would GACK.
3. The retrieve would succeed but a subsequent deployment attempt of the retrieved metadata to a different org would GACK or fail due to org-specific information in the retrieved metadata.

And unfortunately there never seemed to be a specific set of metadata types/objects that caused these behaviors, or at least we never spent enough time trying to isolate them. Generally it was always much easier to create a discrete metadata subscription for the subset of metadata that constitutes the actual customizations/extensions to the org for the project/application.

So I guess I would challenge that it's probably more productive to build up to a known list of metadata that should be included than to shave down (via .forceignore) to what is likely a superset of what should be included. My $0.02 based on having tried to help folks repeatedly (and unsuccessfully) over the years with the notion of a "full org dump".

[pgonzaleznetwork]

This is already possible today using sfdx diff.

1- connect org A to sfdx project
2- retrieve metadata you want to compare
3- change the default username to point to org B
4- use sfdx diff against org and that's it

How will this be different?

[mshanemc]

Mostly the use case "What would I have to do to make one side match the other." In this proposal, you'd end up with manifest files and commands that you could use to deploy/retrieve stuff to make them match.

[mcarvin8]

I've followed Pablo's article to create the org's manifest file (https://www.pablogonzalez.io/how-to-retrieve-your-entire-salesforce-metadata-with-2-commands/) and then retrieve from the org using the manifest file.

Creating the manifest is pretty fast, but the retrieval has never completed successfully for me since my work's org is pretty large and old.

The major limitation for any large org to sync projects and orgs seems to be on the metadata API retrieving everything from these size orgs.

[gfarb]

Will use exact-match files from the org.

Would be great if we could get some 👀 on forcedotcom/salesforcedx-vscode#4162 to mitigate some of the potential false positives that folks would receive because of style-linting, particularly with Prettier. Would love to leverage this but in our case most if not everything that we have Prettier format would likely result in a false positive.

[mshanemc]

I've reminded them

[aheber]

I'd love to see this become built-in. This process is exactly how we do our production deploys and have so for several years.

Some things I'd like to see added though, I don't just want to target an org. If the CLI is going to have the ability to diff metadata I'd like to provide a local copy of metadata (in either format because I do see the benefit of not converting what comes out of the Metadata API). This is because I use cached copies of the org's current metadata to speed up CI evaluation. Imagine executing a deployment to a Sandbox but running that deployment in validation-only, the export could be relied upon to be reusable and would only need to be updated as often as you complete a proper deployment, speeding up CI actions by several minutes or more.

XML ordering is interesting. I think we should be able to, at a CLI/MDT Registry level, understand where ordering doesn't matter (default to it does matter to lower risk). CustomLabels are an example of something that can deploy a lot of unnecessary changes if even a single entity changes and the ordering doesn't matter. Custom Objects are something where partial deployments are possible but depending at what layer you perform you diff calculation, you could be causing a LOT of overhead and redeploy.

I think Deletes in the target org is something to be taken very carefully, possibly even off by default. There are some Metadata types, or even pattern matching names, that should or should not be deleted, varying by individual. Not all things will be tracked and deleting things you found in the org that are NOT found in source could get you in trouble if you're not confident in the correct choice. In my prod org we have ListViews that we track and some that we don't. Instead of fine-tuning our .forceignore we use some simple delete logic, only delete from specific Metadata Types; ApexClass, ApexTrigger, Aura, LWC being examples. Where I see interesting complications is things like Custom Metadata, there are a few types that I want to fully reconcile with source and other types that I need to allow to be independent in prod, yet still others that I want a mix, the records I track with source control can be separate from the records that were manually created (maybe a few that I track because they are useful for dev/testing/QA activity in dev environments)

I know there are several comments all wrapped up here. I have a terrible code base that does a lot of this inefficiently and using some very old ideas but it works well enough that I'm happy to share if that helps.

[azlam-abdulsalam]

It would be also nice to have an option to limit to packageDirectory(s)

[mshanemc]

@azlam-abdulsalam can you say more about how that would work and what you'd need it for?

[mitchspano]

@mshanemc This is an awesome feature and I am really excited about it! Making the org look like the project is super helpful and should be the de-facto deployment mechanism for unpackaged metadata as it will enable auto-destructive changes and support actual rollbacks to a prior commit ( 🥳 ). This is a very necessary step in the direction of source driven development instead of selective delta modifications of a project.

We have built a tool like this internally to make the org match the project and there are a lot of nuances/gotchas to watch out for:

Any diff is treated as a delta as you mentioned, but this is particularly challenging when your team uses the Apex prettier code plugin. This plugin appends a new line at the end of each class file which won't be there upon retrieval from Salesforce - therefore every Apex class in your org will show up as a false positive of a difference. Can we insert a post retrieve, pre diff-generation hook for performing formatting or other CLI scripting to remediate such false positives?

There are some metadata types with "composite" XML representation - where one single file contains multiple pieces of metadata. Custom Labels and Workflow files are the two most notorious offenders of this. In these files, sometimes the ordering can get a little bit out of sync, especially if the XML is modified by hand as it often is on super large projects with many workflow rules/actions. A strict character by character file diff here is extremely likely to produce a false positive of a diff here, so what we do is parse the XML into an object in memory and perform the comparison on the object's properties. This has proved to be a helpful tactic for XML based comparison.

There are also some special character things to watch out for. Some metadata files have their values encoded - ex %apos instead of '. Decoding all of these before performing the diff check is very helpful in reduction of false positives. Another one which was a real headscratcher for us involving carriage returns and new line characters. Some multiline values have \r\n to end lines in files originally authored on Windows machines. However the same source when authored on a unix machine will only have \n, and some metadata files returned from the target org will include the \r - again leading to false positives on the difference.

Another thing we have done which is really nice is the definition of a .deltaIgnore file which can be used to enumerate types or components you don't care about including in your diff check. I would strongly recommend supporting a mechanism like this.

[MarcTCruz]

@mshanemc
I implemented the proposed feature discussed using git cli + sf cli: git add . + git status --porcelain=1 -z on an org I am working.

Parse the contents and you have all diff files needed to send from your local to an org
There is free versioning using git and works with any org not just scratch and developer like Salesforce source tracking.
All changes are visual within a repository, changes can be managed with existing solutions like GitHub desktop.

Proposed solution seems to be the case for using a third party as GitHub CLI, this reduces maintenance as well.

[mitchspano]

@VivekMChawla @mshanemc any additional traction on this idea?

It's very unfortunate that customers have to do gymnastics just to identify what to deploy - especially for orgs above the 10k metadata component limit. Parsing git diffs and manual assembly of package.xml and postDestructiveChanges.xml files is truly an unnecessary burden; this proposal appears to be a valid antidote.

[VivekMChawla]

@mitchspano - Thanks for your feedback on this idea. I'm currently prioritizing our backlog. I'll provide an update once I have a clearer idea of where this idea falls.

[VivekMChawla]

I'd like to thank everyone for contributing to this discussion.

Due to other priorities, we won't be moving forward with this idea at this time.

Rationale:

• The intended outcomes are already possible, though not easy, using existing tooling (CLI+git+XML compare tools).
• XML node ordering differences between org and source, combined with linter changes increase complexity and decrease utility.
• MDAPI "irregularities" and problematic types make this difficult to implement and will potentially cause seasonal issues as metadata types change.
• MDAPI limits on the number of deployed/retrieved components would require us to implement special handling logic to support large orgs.
  • If we were to do something like this, I'd prefer to make project deploy and project retrieve work with > 10,000 metadata files instead.

We'll periodically review this idea to determine if future MDAPI/CLI changes make implementation and support more feasible.

[RupertBarrow]

"XML node ordering differences" is provided by "sf-xml-formatter" or "SFDX essentials".