Add multiple paths property under packageDirectories in sfdx-project.json for MPD and packages over 10,000 files

[jmccotter]

Is your feature request related to a problem? Please describe.
I would like to use packageDirectories with pushPackageDirectoriesSequentially to deploy our source in our core package which is over the 10,000 file limit in 2-3 batches. Our current package directory, which predates SFDX, MPD, etc, is broken up into dozens of modules. Because there are many directories and we only need to break up the directories deployed into two or three batches, the packageDirectories command does not work well for our case. In order to use it in its current state, we have to create a new layer of directories purely for representing the batches which is creating a directory structure based on only the push command rather than our business domain.

What are you trying to do
I would instead like to be able to specify multiple paths under one package directory like you can do with force:source:deploy -p

Describe the solution you'd like
Instead of an sfdx-project.json packageDirectories like this where we create a new folder layer:

{
  ...
  "packageDirectories": [
    {
      "path": "src/Namespace/folder-just-for-SFDX-1",
      "default": true
    },
    {
      "path": "src/Namespace/folder-just-for-SFDX-2",
    },
    {
      "path": "src/Namespace/folder-just-for-SFDX-3",
    }
  ],
  "pushPackageDirectoriesSequentially": true
}

We need to do something like this to replace our current scripts. You could use the existing path property with some logic to differentiate between a single path and list or create a new paths or batchedPaths property or something to indicate the difference:

{
  ...
  "packageDirectories": [
    {
      "paths": ["src/Namespace/domain1", "src/Namespace/domain2", "src/Namespace/domain3"]
      "default": true
    },
    {
      "paths": ["src/Namespace/domain4", "src/Namespace/domain5", "src/Namespace/domain6"]
    },
    {
      "paths": ["src/Namespace/domain7", "src/Namespace/domain8", "src/Namespace/domain9"]
    }
  ],
  "pushPackageDirectoriesSequentially": true
}

Describe alternatives you've considered

• We currently have to do a batch push script using forceignore with the legacy push command to get source tracking to work for development. We use a batch deploy script for CI and cases where we don't need source tracking. The push script isn't ideal since the legacy commands will eventually be discontinued, but the script does not work with the new source tracking and our metadata. I've been working on trying to isolate which files are behaving differently with the new push vs the legacy push causing the issue before submitting an issue for that.
• We could try to use the new deploy command with new source tracking but that would still require creating a script to run the deploy command with the appropriate paths and our own JSON of the paths in each batch. Ideally, that is why we'd like this change to work with the functionality already created without rearranging our directory structure so devs can just run push. Then SFDX via the sfdx-project.json file handles that config and logic.
• We are also in the process of breaking up the monolithic repo to likely move some code out to another package at some point. However it was monolithic because it has existed for longer than SFDX. That work is likely a year or more out to finish and possibly partially dependent on figuring out the issues with 2GP so it could be even further out. For now, we still need a way to address deploying packages with over 10,000 files.

Additional context
N/A

[github-actions[bot]]

Thank you for filing this feature request. We appreciate your feedback and will review the feature at our next grooming or sprint planning session. We prioritize feature requests with more upvotes and comments.

[uip-robot-zz]

This issue has been linked to a new work item: W-11139620

[mshanemc]

have you tried using the rest metadata api? (sfdx config:set restDeploy=true) ? It had its own issues when it came out, but it does have the advantage of supporting larger deployments.

I'd try that without pushPackageDirectoriesSequentially just to see if it is possible to do it all in one go. If I understand correctly, that might make the whole question moot.

If all that doesn't work, I think introducing a new folder level that represents "how deployments are grouped" would be the best option--pkgDirs affect both sfdx push/pull and 2GP uses those folders so we probably can't easily change the behavior to do groupings inside sfdx-project.json for push without making a mess for 2GP users.

[jmccotter]

@mshanemc We're over the 10,000 files limit rather than the 39 MB limit so I don't think the REST deploy would make a difference. However, there seems to be a bug getting it to actually do REST deploy. It seems to completely disregard the config and environment variable.

% SFDX_REST_DEPLOY=true sfdx force:source:push -f
*** Pushing with SOAP API v52.0 ***
ERROR running force:source:push:  INVALID_OPERATION: Too many files in zip

% sfdx config:set restDeploy=true
=== Set Config

 Name       Value Success 
 ────────── ───── ─────── 
 restDeploy true  true    
% sfdx force:source:push -f 
*** Pushing with SOAP API v52.0 ***
ERROR running force:source:push:  INVALID_OPERATION: Too many files in zip

That's with this SFDX version:

 CLI Version : 
	sfdx-cli/7.156.1

 Architecture: 
	darwin-x64

 Node Version : 
	node-v14.19.3

 Plugin Version: 
	@oclif/plugin-autocomplete 0.3.0 (core)
	@oclif/plugin-commands 1.3.0 (core)
	@oclif/plugin-help 3.3.1 (core)
	@oclif/plugin-not-found 1.2.6 (core)
	@oclif/plugin-plugins 1.10.11 (core)
	@oclif/plugin-update 1.5.0 (core)
	@oclif/plugin-warn-if-update-available 1.7.3 (core)
	@oclif/plugin-which 1.0.4 (core)
	@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)
	alias 2.1.0 (core)
	apex 0.13.0 (core)
	auth 2.2.0 (core)
	community 2.0.0 (core)
	config 1.4.12 (core)
	custom-metadata 2.0.0 (core)
	data 2.0.4 (core)
	generator 2.0.1 (core)
	info 2.0.1 (core)
	isvte-sfdx-plugin 1.1.15 (1.1.15)
	limits 2.0.1 (core)
	org 1.13.2 (core)
	salesforce-alm 54.6.0 (core)
	schema 2.1.1 (core)
	sfdx-cli 7.156.1 (core)
	signups 1.2.0 (core)
	source 2.0.2 (core)
	telemetry 2.0.0 (core)
	templates 55.0.0 (core)
	trust 2.0.1 (core)
	user 2.1.0 (core)

 OS and Version: 
	Darwin 20.6.0

I went down to SFDX v 7.155 and it looks like REST deploy is broken or it's related to too many files. I get this same error in lower versions.

% sfdx force:source:push -f
*** Pushing v52.0 metadata with REST API v54.0 connection ***
ERROR running force:source:push:  Metadata API request failed: The deploy operation is missing a job ID. Initialize an operation with an ID, or start a new job.

It doesn't have to be that specific property. It could be a rearrangement of the brackets around the packageDirectories or a property on each packageDirectory. I'm less fond of the latter because it would mean you could put them in a different order than they deploy in using the group number or something like that. With that in mind something like this to maintain backwards compatibility? Then you would just need to flatten the list of lists in all cases except force:source:push when pushPackageDirectoriesSequentially is true.

{
  ...
  "packageDirectories": [
    [
      {
        "path": "src/Namespace/domain1",
        "default": true
      },
      {
        "path": "src/Namespace/domain2"
      },
      {
        "path": "src/Namespace/domain3"
      }
    ],
    [
      {
        "path": "src/Namespace/domain4"
      },
      {
        "path": "src/Namespace/domain5"
      },
      {
        "path": "src/Namespace/domain6"
      }
    ],
  ],
  "pushPackageDirectoriesSequentially": true
}

Alternatively, the latter option which seems a little more fragile to me, but doesn't require rewriting any of the path code except for push when pushPackageDirectoriesSequentially is true:

{
  ...
  "packageDirectories": [
    {
      "path": "src/Namespace/domain1",
      "default": true,
      "pushGroup": 1
    },
    {
      "path": "src/Namespace/domain2",
      "pushGroup": 1
    },
    {
      "path": "src/Namespace/domain3",
      "pushGroup": 1
    },
    {
      "path": "src/Namespace/domain1",
      "default": true,
      "pushGroup": 2
    },
    {
      "path": "src/Namespace/domain2",
      "pushGroup": 2
    },
    {
      "path": "src/Namespace/domain3",
      "pushGroup": 2
    }
  ],
  "pushPackageDirectoriesSequentially": true
}

[github-actions[bot]]

This issue has not received a response in 60 days. It will auto-close in 7 days unless a response is posted.

[mshanemc]

@jmccotter here's a discussion about pushPackageDirectoriesSequentially that may interest you.
#2402

[jmccotter]

The main thing that discussion seems to miss is the source tracking updates which are pretty time consuming before and after each sf project deploy start. We switched to using sf project deploy start -d specifying our package directories for the first two batches with the last batch just running without the directories specified, but pushPackageDirectoriesSequentially (PPDS) had the advantage of checking source tracking once, doing all 3 deploys, and updating source tracking after. Now for three batches we're doing the source tracking check before and source tracking update after each time instead so 3 source tracking checks, 3 deploys, 3 source tracking updates instead of 1 source tracking check, 3 deploys, 1 source tracking update. The difference with the force:source:push with PPDS was deploying around 17000 files in 30-40 minutes with source tracking instead of 75-100 minutes with the three separate project deploy starts. Source tracking using sf project deploy start is also much slower on large numbers of files now compared to the initial beta source push tracking. Even for pushing a single file, it's around 3-5+ minutes for source tracking compared to just deploying without force:source:deploy or old versions of force:source:push.

The only equivalently useful alternative to us specifying the directories to push sequentially would be something like this post which would be some kind of dependency graphing and automatically breaking up the deployment into separate deployments below the 10,000 file limit based on dependencies while also keeping only one source tracking check before and one source tracking update after those deployments to get similar performance. Essentially the PPDS and multiple paths idea was just a way for us to manually tell it the dependencies without changing our folder structure.

[mshanemc]

Even for pushing a single file, it's around 3-5+ minutes for source tracking compared to just deploying without force:source:deploy or old versions of force:source:push.

That sounds like a bug. Is this pushing to a basic scratch org? And just for my sanity, you're not using VSCode for this push, but the CLI itself?

[jmccotter]

Yes the CLI itself, but we have 18,228 files in the SF project so I think the issue is the source tracking is checking and updating all of those even when only one file has changed.

[mshanemc]

source tracking is checking and updating all of those

yeah, but it shouldn't do that. Once you've deployed the beast initially, the ones after that (like your single file example) should be really quick. I'm trying to replicate this now with EDA, the largest repo I have access to.

[mshanemc]

but pushPackageDirectoriesSequentially (PPDS) had the advantage of checking source tracking once, doing all 3 deploys, and updating source tracking after.

The reason we don't want to go back to that method is that if it timed out there was no way to resume/recover from it. The new approach updates tracking after the deployments finish.

[mshanemc]

I'm marking this as not planned. I'd rather have

1. a github issue with a sample repo that reproduces a long delay in tracking operations because of a really large project. If we can optimize that I think it'll help with this problem and more generally
2. examples of any files that are holding open the SourceMember polling after a large deploy (these should now show up in warnings from the deployments because of feat: warn about missing sourceMembers source-tracking#511) so if you're seeing some patterns in there, it would be helpful