sf package version create use profiles not only from path

[olektymchenko]

Summary

sf package version create use component not only from specified path

Steps To Reproduce

I have this folders structure and only "core-package" should be packaged
core-package
unlocked-package
internal-components (which I don't want to package at all)

The following commands were executed
sf package create --name "AAA_Test" --package-type Managed --path "aaa-core" --description "AAA Test Package Description" --target-dev-hub AAA_Prod

sf package version create --package "AAA_Test" --definition-file config/project-scratch-def.json --wait 20 --installation-key-bypass --code-coverage --target-dev-hub AAA_Prod

Expected result

Package is created

Actual result

Creation fails because sf is trying to package components not from path specified during package creation

System Information

powershell 5
{
"cliVersion": "@salesforce/cli/2.1.7",
"architecture": "win32-x64",
"nodeVersion": "node-v18.15.0",
"osVersion": "Windows_NT 10.0.19044",
"shell": "cmd.exe",
"rootPath": "C:\Users\sssssssss\AppData\Local\sf\client\2.1.7-355d833",
"pluginVersions": [
"@oclif/plugin-autocomplete 2.3.2 (core)",
"@oclif/plugin-commands 2.2.18 (core)",
"@oclif/plugin-help 5.2.13 (core)",
"@oclif/plugin-not-found 2.3.31 (core)",
"@oclif/plugin-plugins 3.1.6 (core)",
"@oclif/plugin-search 0.0.20 (core)",
"@oclif/plugin-update 3.1.26 (core)",
"@oclif/plugin-version 1.3.7 (core)",
"@oclif/plugin-warn-if-update-available 2.0.43 (core)",
"@oclif/plugin-which 2.2.25 (core)",
"@salesforce/cli 2.1.7 (core)",
"apex 2.3.6 (core)",
"auth 2.8.9 (core)",
"data 2.5.1 (core)",
"deploy-retrieve 1.16.0 (core)",
"info 2.6.31 (core)",
"limits 2.3.26 (core)",
"login 1.2.20 (core)",
"org 2.9.25 (core)",
"packaging 1.21.1 (user)",
"schema 2.3.20 (core)",
"settings 1.4.20 (core)",
"sobject 0.1.37 (core)",
"source 2.10.25 (core)",
"telemetry 2.2.3 (core)",
"templates 55.5.5 (core)",
"trust 2.4.32 (core)",
"user 2.3.24 (core)",
"@salesforce/sfdx-scanner 3.14.0 (user)",
"sfdx-git-delta 5.24.1 (user)"
]
}

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[amtrack]

@olektymchenko Is this the same issue as described in #620?

[olektymchenko]

@amtrack
Hello,
Yes, I think that it's the same
Quote tricky to find a reasonable workaround.
Approach with deleting profile to creating a package version is little bit painful.

[ecampamavens]

We are experiencing the same issue with profiles that do not even belong to the package getting included in the package.

[burntsugar]

Also observing the issue with profiles being pulled into packages. @salesforce/cli/2.2.7 darwin-arm64 node-v18.16.1

[burntsugar]

Is there an update for this please?

[ecampamavens]

Please...

[calvinle1]

Same issue experienced here - is there an update or workaround?

[ecampamavens]

It's been a week with no updates, we are blocked at this time. Can you at least provide a temporary workaround?

[shetzel]

I know this will be an unpopular response, but the behavior for packaging commands is owned by a team who does not monitor this issues repo. If you are blocked by this I highly recommend contacting customer support so they can open a support case for that team, which is how they respond to issues like this. As a workaround, there may be something from #620 that can help.

[ecampamavens]

As a workaround we had to create a script that changes the format of every "profile" file from ".profile-meta.xml" to be a ".txt" file before the command
sf package version create -p PACKAGE_NAME gets executed. Once we finish creating and installing the packages, we proceed to "restore" the profiles back to their corresponding metadata format and proceed.

[shetzel]

@ecampamavens - is the package version created successfully if you add those profiles to your .forceignore file (as a workaround)?

[ImJohnMDaniel]

@ecampamavens and @shetzel -- I believe this is expected behavior.

The behavior of the package version creation is to take all metadata specified in the sfdx-project.json's packageDirectory that is being used to build the package PLUS any profiles found anywhere in the project source code regardless of whether or not those files exist in a specified "packageDirectory.path" in the sfdx-project.json.

This behavior was a surprise to many folks when we first saw it a couple of years ago. Here is a post from Dileep Burki, Salesforce Product Manager for Packaging, on the "Unlocked Packages" group; specifically look for Dileep's comment on the thread from Feb 28, 2019, 4:14 PM EST -- the 3rd bullet point. In that section, Dileep says the following:

3. When a package version is created (force: package:version:create), the system looks for the entire DX project directory (even outside of package directory specified via the --path flag) to see if there are any profiles. Once the profile files are found, only those aspects that relate to the metadata in the package directory are used to create profile snippets for the package.

As @amtrack mentioned above, there have also been similar issues raised about this behavior (#540 and #620).

The only effective workarounds that I have found are one of the following:

• rename the profile files to an extension of .txt so it won't be included in package version creation operations.
• don't manage profiles in package projects to begin with and simply stay with PermSets/PermSetGroups/etc.

The second option is where most of my teams stay. In our case, we are working with clients that already have well established orgs and user profiles. Since we are adding new "apps" to existing orgs via our packages, our "apps/packages" should not "own" the profile metadata and it should be managed elsewhere by the client. That may not fit everyone's scenario but it does give insight into the thought process that we have using.

For what it is worth, here is an idea that is worth upvoting that is related to this question.

I hope this helps.

[ecampamavens]

@ImJohnMDaniel I wonder how this "started" to be an issue as of a week ago for us when we have been creating package version with the same project structure for over 2 years. I am testing a package version creation using the ".forceignore" suggestion from @shetzel. I will update you once complete.

[ImJohnMDaniel]

Yeah, I got nothing for you on that question. I just remember the issue from a couple of years ago.

[shetzel]

@ecampamavens - are you saying that without any changes to the metadata in the project you can no longer run the same command used for the past 2 years? If that's the case, then it's possible a change to the packaging plugin (or more likely the packaging library it depends on) has changed behavior. If you run sf version --verbose it will display the plugin versions. Look for the packaging plugin version and you can install older or newer versions and re-run the packaging commands. If you find that a specific version works while other versions don't, please post the versions here.

[ecampamavens]

@shetzel update on packaging issue, I tried building the same package with the forceignore and got the same failure behavior.

[ecampamavens]

@olektymchenko, @shetzel and @ImJohnMDaniel I switched to version sf plugins install packaging@1.17.3 and that seems to be playing nice with the profiles being present. Give it a try!

[git2gus]

This issue has been linked to a new work item: W-13932401

[shetzel]

I'm trying to reproduce the problem using this repo from issue 620. I'm not seeing the profile included in the package. I tried with unlocked and managed packages. It would be very helpful if there was a github repo I could use to reproduce the problem and work on a fix.

[olektymchenko]

@amtrack @ecampamavens @calvinle1
Could you please help with creating a repo to reproduce an issue?
I was trying to reproduce, but for some reason I didn't receive an error during version creation, maybe your approach will help.

[ecampamavens]

@olektymchenko aren't you the author? Unfortunately I cannot provide a repo since we are IP protected, I do confirm that for the packaging@1.17.3 it is working so whatever they were doing on this version please apply it for the next release.

[ethan-sargent]

This issue is related to the issues I've been experiencing with sfdx-scanner.
oclif/plugin-plugins#642
forcedotcom/sfdx-scanner#1153

I was able to give a team member a copy of my package.json and yarn.lock files from my plugin install directory and they were able to create packages with the latest version of this library.
Currently working on a repro dockerfile for this issue, similar to the repro dockerfiles for the scanner issue.

In the meantime @shetzel we are able to consistently reproduce this by running packaging commands in the salesforce/cli:latest docker containers

[Mars-PrakashSaini]

@olektymchenko Can you please add "includeProfileUserLicenses": true in your sfdx-project.json file and try to re-create the package version

[amtrack]

@olektymchenko @shetzel There is not necessarily a user-facing error message.
The issue is that the package version includes unrelated profiles.
This is visible when checking the logs or inspecting the metadata component list of the created package version afterwards.

Here is a repo with detailed instructions for reproduction: https://github.com/mdapi-issues/mre-2gp-unrelated-profiles.

I can still confirm this issue with the following sf version and packaging plugin versions:

$ sf --version
@salesforce/cli/2.6.7 darwin-arm64 node-v18.12.1
$ sf plugins
packaging 1.22.1
$ sf plugins update
$ sf plugins
packaging 1.23.0

[ethan-sargent]

@shetzel It looks like this is also reproducible if you locally install an npm package in your project that contains a profile, which is commonly used in packages for tests.
For example, the vlocity build tool itself does this.
This error (specifically the third profile which I did not recognise from my project):

is caused by the profile in my node_modules directory here:

Git worktrees can also cause this to fail if you have a worktree that still contains profiles.
It looks like sf is just searching for all profile-meta.xml files below the current working directory, not just isolated to other package directories.
There's no reason sf should be looking anywhere except package directories for metadata anyway...

[shetzel]

@amtrack @ethan-sargent - thanks for the repros and extra info. I have a fix about ready to go in for this.

[shetzel]

A fix to the package version create behavior is in the CLI release candidate published about an hour ago. sf v2.11.7. See the release notes here or run sf whatsnew -v latest-rc.
tldr; --> Profiles are only searched within package directories defined within sfdx-project.json.

More news: a new property on "packageDirectories" in sfdx-project.json is coming soon (possibly next week) that will further scope Profile searching to only the directory being packaged.

[nvuillam]

Problem still visible here because I have an Admin.profile in another folder than the packaged one :(

Had to remove the profile that was used in scratch orgs ! :(

[ecampamavens]

@nvuillam I believe we have to remove any folders from the sfdx-project.json file. If a profile exists within a folder that is named in the sfdx-project.json file, then the script will grab it. I removed these folders from the file and it works fine now.

[nvuillam]

I used the workaround, but is has no sense to browse files that are outside of the folder corresponding to the package we want to generate 😩

[petter-eikeland]

@ecampamavens as @shetzel mentioned, you can use the workaround "scopeProfiles": "true" attribute on all directories (that you package) in sfdx-project.json. So you don't have to remove these folders anymore.

Of course, it's a weird built-in behavior that maybe made sense some time ago.