Error syncing users: An error occurred while calling https://uaa/Users

[burgerjeffrey]

Using version 1.0.74 of cf-mgmt

Not sure why this started or how it happened but I cannot figure out how to get past this, any help is appreciated.
update-org-users and update-space-users tasks are both failing for this same reason.

update-space-users
Version: [1.0.74], Commit: [93e74cd5ba7a8f0236dc65ed5c0ae780d5f91f74] 2023/12/13 20:42:04 W1213 20:42:04.516024 22 ldap.go:108] No users found under group: sg-app-dg-cf_devint_cmn-role-spcaud 2023/12/13 20:42:04 W1213 20:42:04.721002 22 ldap.go:108] No users found under group: sg-app-dg-cf_devint_cst-role-spcaud 2023/12/13 20:42:04 W1213 20:42:04.86749 22 ldap.go:108] No users found under group: sg-app-dg-cf_devint_inf-role-spcaud 2023/12/13 20:42:05 W1213 20:42:05.036627 22 ldap.go:108] No users found under group: sg-app-dg-cf_devint_inv-role-spcaud 2023/12/13 20:42:06 W1213 20:42:06.279509 22 ldap.go:108] No users found under group: sg-app-dg-cf_devint_prd-role-spcaud 2023/12/13 20:42:06 W1213 20:42:06.36749 22 ldap.go:108] No users found under group: sg-app-dg-cf_int_cmn-role-spcaud 2023/12/13 20:42:06 W1213 20:42:06.458015 22 ldap.go:108] No users found under group: sg-app-dg-cf_int_cst-role-spcaud 2023/12/13 20:42:06 W1213 20:42:06.547021 22 ldap.go:108] No users found under group: sg-app-dg-cf_int_inf-role-spcaud 2023/12/13 20:42:06 W1213 20:42:06.634963 22 ldap.go:108] No users found under group: sg-app-dg-cf_int_inv-role-spcaud 2023/12/13 20:42:06 W1213 20:42:06.907867 22 ldap.go:108] No users found under group: sg-app-dg-cf_int_ord-role-spcaud 2023/12/13 20:42:06 W1213 20:42:06.995384 22 ldap.go:108] No users found under group: sg-app-dg-cf_int_prd-role-spcaud error: got errors processing update space users [Error syncing users for org dev, space X, role developer: adding ldap users: An error occurred while calling https://uaa.sys.<redacted>/Users]

update-org-users
Version: [1.0.74], Commit: [93e74cd5ba7a8f0236dc65ed5c0ae780d5f91f74] 2023/12/14 14:13:11 I1214 14:13:11.100913 18 yaml_config.go:535] Using environment provided ldap user <redacted> instead of 2023/12/14 14:13:11 I1214 14:13:11.100983 18 yaml_config.go:546] Using environment provided ldap host <redacted> instead of 2023/12/14 14:13:18 W1214 14:13:18.704188 18 ldap.go:108] No users found under group: sg-app-dg-cf_devint-role-billmgr 2023/12/14 14:13:18 W1214 14:13:18.731803 18 ldap.go:108] No users found under group: sg-app-dg-cf_devint-role-orgaud 2023/12/14 14:13:19 W1214 14:13:19.187882 18 ldap.go:108] No users found under group: sg-app-dg-cf_ft-role-billmgr 2023/12/14 14:13:21 W1214 14:13:21.762141 18 ldap.go:108] No users found under group: sg-app-dg-cf_int-role-billmgr 2023/12/14 14:13:21 W1214 14:13:21.789289 18 ldap.go:108] No users found under group: sg-app-dg-cf_int-role-orgaud error: got errors processing update org users [Error syncing users for org ft role org-manager: adding ldap users: An error occurred while calling https://uaa.sys.<redacted>/Users]

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[calebwashburn]

@burgerjeffrey Looks like there was a regression added several versions ago where validation of system domain, userid, password were not being raised as an error. Created a pull request to add this validation back but can be fixed without a new release by ensuring you have set the system domain which appears to be blank.

[calebwashburn]

@burgerjeffrey Also published a develop tag with latest fixes if you want to validate this fixes your issue.

pivotalservices/cf-mgmt:develop

[burgerjeffrey]

@burgerjeffrey Looks like there was a regression added several versions ago where validation of system domain, userid, password were not being raised as an error. Created a pull request to add this validation back but can be fixed without a new release by ensuring you have set the system domain which appears to be blank.

I am not understanding where the system domain would be blank, can you explain more about this?

[calebwashburn]

@burgerjeffrey if running via concourse this is exported as an environment variable. Am curious if you use the "develop" tag does this fix your issue and either show an error or work as expected.

[burgerjeffrey]

@calebwashburn yes, and i did find I have that exported as as environment variable. I will get it tested out, I see develop on the docker hub to make my image from now.

[burgerjeffrey]

@calebwashburn issue is the same with the develop tag

Version: [DEV], Commit: [b511f74a45ffcc365660bfd6f6f4df332b85f2e4] 2023/12/14 20:13:57 I1214 20:13:57.448526 18 yaml_config.go:535] Using environment provided ldap user <redacted> instead of 2023/12/14 20:13:57 I1214 20:13:57.448588 18 yaml_config.go:546] Using environment provided ldap host <redacted> instead of 2023/12/14 20:14:05 W1214 20:14:05.260307 18 ldap.go:108] No users found under group: sg-app-dg-cf_devint-role-billmgr 2023/12/14 20:14:05 W1214 20:14:05.287217 18 ldap.go:108] No users found under group: sg-app-dg-cf_devint-role-orgaud 2023/12/14 20:14:05 W1214 20:14:05.727354 18 ldap.go:108] No users found under group: sg-app-dg-cf_ft-role-billmgr 2023/12/14 20:14:08 W1214 20:14:08.011898 18 ldap.go:108] No users found under group: sg-app-dg-cf_int-role-billmgr 2023/12/14 20:14:08 W1214 20:14:08.038538 18 ldap.go:108] No users found under group: sg-app-dg-cf_int-role-orgaud error: got errors processing update org users [Error syncing users for org ft role org-manager: adding ldap users: An error occurred while calling https://uaa.sys.<redacted>/Users]

[calebwashburn]

@burgerjeffrey I added some information logging to help debug this to print out the system domain and uaa target. This has been re-pushed to develop with the following sha / digest

develop: digest: sha256:d3e5772ccc003adfef3fbfba90031ace5536c355400a66e5e44efb0fe40bd985

Let me know if you can re-run with this to help triage this issue.

[burgerjeffrey]

@calebwashburn looks like those are correct.

Version: [DEV], Commit: [7df3f2aa8f25d2b0f9646a143637e6d0305ca1fb] 2023/12/14 20:31:32 I1214 20:31:32.310341 19 yaml_config.go:535] Using environment provided ldap user <redacted> instead of 2023/12/14 20:31:32 I1214 20:31:32.31042 19 yaml_config.go:546] Using environment provided ldap host <redacted> instead of 2023/12/14 20:31:32 I1214 20:31:32.392847 19 initialize.go:80] Using system domain [sys.<redacted>] 2023/12/14 20:31:32 I1214 20:31:32.392925 19 uaa.go:43] Using uaa target [https://uaa.sys.<redacted>] 2023/12/14 20:31:40 W1214 20:31:40.080032 19 ldap.go:108] No users found under group: sg-app-dg-cf_devint-role-billmgr 2023/12/14 20:31:40 W1214 20:31:40.10734 19 ldap.go:108] No users found under group: sg-app-dg-cf_devint-role-orgaud 2023/12/14 20:31:40 W1214 20:31:40.4879 19 ldap.go:108] No users found under group: sg-app-dg-cf_ft-role-billmgr 2023/12/14 20:31:42 W1214 20:31:42.775471 19 ldap.go:108] No users found under group: sg-app-dg-cf_int-role-billmgr 2023/12/14 20:31:42 W1214 20:31:42.802884 19 ldap.go:108] No users found under group: sg-app-dg-cf_int-role-orgaud error: got errors processing update org users [Error syncing users for org ft role org-manager: adding ldap users: An error occurred while calling https://uaa.sys.<redacted>/Users]

Note, only 1 of my 8 CF environments is experiencing this error out of the blue, so wasn't sure where to start or if there is an issue with UAA possibly.

[calebwashburn]

@burgerjeffrey Sorry for the confusion. Didn't realize you had redacted the uaa domain in the error message so this is what I was trying to track down. Let me add more context to uaa errors and have you retest.

develop: digest: sha256:889f29f967167131d75143c55d37144c9ef13a81010360daec86e65ab9bab74f

[burgerjeffrey]

@calebwashburn this is the results form your addtional context added:

Version: [DEV], Commit: [3d5ddf23a16a0da9c3f18a25902dc3abcc630dd5] 2023/12/14 21:39:03 I1214 21:39:03.478486 18 yaml_config.go:535] Using environment provided ldap user <redacted> instead of 2023/12/14 21:39:03 I1214 21:39:03.478546 18 yaml_config.go:546] Using environment provided ldap host <redacted> instead of 2023/12/14 21:39:03 I1214 21:39:03.532439 18 initialize.go:80] Using system domain [sys.<redacted>] 2023/12/14 21:39:03 I1214 21:39:03.53251 18 uaa.go:43] Using uaa target [https://uaa.sys.<redacted>] 2023/12/14 21:39:11 W1214 21:39:11.371933 18 ldap.go:108] No users found under group: sg-app-dg-cf_devint-role-billmgr 2023/12/14 21:39:11 W1214 21:39:11.398969 18 ldap.go:108] No users found under group: sg-app-dg-cf_devint-role-orgaud 2023/12/14 21:39:11 W1214 21:39:11.771859 18 ldap.go:108] No users found under group: sg-app-dg-cf_ft-role-billmgr 2023/12/14 21:39:13 E1214 21:39:13.535404 18 uaa.go:81] Error adding user to uaa [An error occurred while calling https://uaa.sys.<redacted>/Users] 2023/12/14 21:39:13 W1214 21:39:13.562292 18 ldap.go:108] No users found under group: sg-app-dg-cf_int-role-billmgr 2023/12/14 21:39:13 W1214 21:39:13.589718 18 ldap.go:108] No users found under group: sg-app-dg-cf_int-role-orgaud error: got errors processing update org users [Error syncing users for org ft role org-manager: adding ldap users: An error occurred while calling https://uaa.sys.<redacted>/Users]

[calebwashburn]

@burgerjeffrey Sorry for all the back and forth but the underlying UAA client library cf-mgmt leverages is swallowing the error unless we turn on verbosity, which I have set to true in this build to see what is actually the error when calling UAA to guide us to figuring out resolution.

develop: digest: sha256:f4b2143bb507f4dc5f9af5ee3c4850c83140fd5499b3dda24654fa591689129a size: 1368

[burgerjeffrey]

@calebwashburn my issue is now resolved. I was able to use the uaa-go cli to resolve the issue on UAA.

The error that provided the clue was update-space-users when it first errored out, saying:

error: got errors processing update space users [Error syncing users for org dev, space <username>, role developer: adding ldap users: An error occurred while calling https://uaa.sys.<redacted>/Users

I then used uaa-go cli to remove the user from UAA:

to obtain the password to login to UAA:
om -e env.yml credentials -p cf --credential-reference .uaa.admin_client_credentials -t json | om interpolate --path /password

to target UAA:
uaa-go target https://uaa.sys.<redacted>

to login to UAA as an admin:
uaa-go get-client-credentials-token admin -s <secret from above step>

get the user:
uaa-go get-user <username>

delete the user:
uaa-go delete-user <username>

I then re-ran the update-org-users and it ran successfully this time:
Version: [DEV], Commit: [3d5ddf23a16a0da9c3f18a25902dc3abcc630dd5] 2023/12/14 21:51:25 I1214 21:51:25.311396 19 yaml_config.go:535] Using environment provided ldap user <redacted> instead of 2023/12/14 21:51:25 I1214 21:51:25.311461 19 yaml_config.go:546] Using environment provided ldap host <redacted> instead of 2023/12/14 21:51:25 I1214 21:51:25.365196 19 initialize.go:80] Using system domain [sys.<redacted>] 2023/12/14 21:51:25 I1214 21:51:25.365245 19 uaa.go:43] Using uaa target [https://uaa.sys.<redacted>] 2023/12/14 21:51:33 W1214 21:51:33.058311 19 ldap.go:108] No users found under group: sg-app-dg-cf_devint-role-billmgr 2023/12/14 21:51:33 W1214 21:51:33.085293 19 ldap.go:108] No users found under group: sg-app-dg-cf_devint-role-orgaud 2023/12/14 21:51:33 W1214 21:51:33.456658 19 ldap.go:108] No users found under group: sg-app-dg-cf_ft-role-billmgr 2023/12/14 21:51:35 I1214 21:51:35.325375 19 manager_org.go:56] Add User <username> to role manager for org ft 2023/12/14 21:51:40 W1214 21:51:40.647823 19 ldap.go:108] No users found under group: sg-app-dg-cf_int-role-billmgr 2023/12/14 21:51:40 W1214 21:51:40.674776 19 ldap.go:108] No users found under group: sg-app-dg-cf_int-role-orga

[binayakmohanty]

@burgerjeffrey @calebwashburn We are facing same issue on our foundation when cf-mgmt was updated to 1.0.73.

Error:

Error syncing users for org , space , role developer: adding ldap users: An error occurred while calling https://uaa./Users]

PS: I have edited the error msg for org-name, space-name and uaa endpoint. We don't have uaa-go tool.

Please suggest a suitable solution to this. Let us know if this issue is resolved in the later versions of cf-mgmt.

[calebwashburn]

@binayakmohanty will need to add additional context to the uaa errors to help triage this as there is a conflict with what is in uaa so need more specifics in their error message and that library doesn’t share the raw error by default.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[binayakmohanty]

Hey @calebwashburn , thanks for the reply. But I'm not sure how to add more context to the errors which belongs to uaa-client-library as we are getting the same error from cf-mgmt package.

[calebwashburn]

@binayakmohanty - FYI... Here's a branch with additional logging - https://github.com/vmware-tanzu-labs/cf-mgmt/tree/issue_467_uaa_logging that is the basis for a PR to add that additional context.

If able to test there is a new tag for the docker image under docker.io/pivotalservices/cf-mgmt:develop that you can use as pre-release to see if this sheds any light on the issue and if additional fixes could me made to remediate this.

[binayakmohanty]

Hey @calebwashburn , Can you provide the link for the cf-mgmt binary with the additional features!
Purpose for the ask is we create our own docker image pulling binaries from our personal artifactory and we aren't allowed to download anything from open internet.
Additional context: Yesterday I had a word with Jeffrey Sdoeung about this issue. It would be better if we can connect sometime through a screen sharing session.

[calebwashburn]

Fixed in v1.0.85 - https://github.com/vmware-tanzu-labs/cf-mgmt/releases/tag/v1.0.85

[binayakmohanty]

Thanks @calebwashburn for fixing this.