Correct Default for MaximumPageSize #20453

ltcarbonell · 2023-05-01T20:20:49Z

This pull request addresses a problem that was introduced in a previous pull request (#19032). The earlier PR added a new configuration option called max_page_size to the LDAP secret engine. This option allows users to adjust the size of the paged search results. By default, it is set to 2147483647 (math.MaxInt32), but it can also be set to -1 to use non-paged searching.

The issue occurs when upgrading Vault to 1.13.2, 1.12.6, or 1.11.10, that has an existing LDAP Auth configuration. This would result in the max_page_size being set to 0 instead of the intended default of 2147483647.

Resolves #20416

sdk/helper/ldaputil/client.go

maxb · 2023-05-10T22:22:58Z

After the comments I made earlier today, I spent some more time investigating. I uncovered some interesting information.

First, it's kind of weird this option is named max_page_size. IMO it would have been more logical to just call it page_size. It is not actually treated as a maximum, it is simply the page size we ask the LDAP server to provide us with.

Next, sending a page size of zero is always wrong, as this is a special value that instructs the server to terminate an in-progress paging result set, and if sent with an initial query, will prevent the server returning any results (hence the regression that was reported).

Since a page size of zero is invalid, I propose that this PR should not change the Go type used for MaximumPageSize from int to *int, as the simple zero value is available to use as an unspecified value.

Additionally, requesting a page size of 2147483647 never makes any sense, because that's what we send as our size limit - and RFC 2696 that defines the paging control explicitly says a page size equal or greater to the size limit is to be ignored by servers.

Taking all this into account, I agree with the proposal to change the default behaviour to "no paging", because the only useful use of paging, is one set to cooperate with the server defined limits. In a default Active Directory installation, the maximum page size the server will permit is 1000, so Active Directory users will probably want to set max_page_size=1000.

sdk/helper/ldaputil/client.go

sdk/helper/ldaputil/config.go

website/content/api-docs/auth/ldap.mdx

CodeGlitcher · 2023-05-11T08:20:24Z

We run into the same issue.
any value besides 0 works.

Any change this could be added to the release notes as a known issue.
Would have saved me some searching.

Cajga · 2023-05-17T11:27:38Z

As I wrote it in the related issue opened by others, this bug closed us out from our vault cluster.
We have all policies bind to LDAP groups and we revoked the root token.

It would be nice if you could mention this regression in the CHANGELOG. So, people who use LDAP and read release notes before upgrade (like I did) will not run into this.

website/content/docs/upgrading/upgrade-to-1.12.x.mdx

website/content/docs/upgrading/upgrade-to-1.13.x.mdx

website/content/docs/upgrading/upgrade-to-1.11.x.mdx

Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>

calvn · 2023-05-17T18:03:38Z

sdk/helper/ldaputil/client.go

@@ -553,7 +553,7 @@ func (c *Client) GetLdapGroups(cfg *ConfigEntry, conn Connection, userDN string,
 	if cfg.UseTokenGroups {
 		entries, err = c.performLdapTokenGroupsSearch(cfg, conn, userDN)
 	} else {
-		if paging, ok := conn.(PagingConnection); ok && cfg.MaximumPageSize >= 0 {
+		if paging, ok := conn.(PagingConnection); ok && cfg.MaximumPageSize > 0 {


I'm not sure if there's already a test case that covers this, but wondering if we should make sure that setting max_page_size to zero doesn't actually result in an group search that returns no group (which was the main issue with the regression).

@calvn When 0 is set it swaps out to the non-paged search which ldaputil has always used.

Right, the removal of = makes it so that the zero value doesn't result in a paged search. I was mostly asking whether we should have a test case to cover this to prevent future regression.

ltcarbonell · 2023-05-17T18:27:52Z

As I wrote it in the related issue opened by others, this bug closed us out from our vault cluster. We have all policies bind to LDAP groups and we revoked the root token.

It would be nice if you could mention this regression in the CHANGELOG. So, people who use LDAP and read release notes before upgrade (like I did) will not run into this.

Nice suggestion @Cajga. This will be added to this doc here: https://developer.hashicorp.com/vault/docs/upgrading/upgrade-to-1.13.x#known-issues

changelog/20453.txt

Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>

* default max page size for config * Add changelog * update test int to *int * add testing defaults * update default to -1, i.e. dont paginate * update test * Add error message for invalid search * Make 0 the default * cleanup * Add to known issues doc * Update website/content/docs/upgrading/upgrade-to-1.13.x.mdx * Update website/content/docs/upgrading/upgrade-to-1.11.x.mdx Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com> * Update website/content/docs/upgrading/upgrade-to-1.13.x.mdx Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com> * Update website/content/docs/upgrading/upgrade-to-1.12.x.mdx Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com> * Add workaround to docs * Update changelog/20453.txt Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com> --------- Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com> Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>

* default max page size for config * Add changelog * update test int to *int * add testing defaults * update default to -1, i.e. dont paginate * update test * Add error message for invalid search * Make 0 the default * cleanup * Add to known issues doc * Update website/content/docs/upgrading/upgrade-to-1.13.x.mdx * Update website/content/docs/upgrading/upgrade-to-1.11.x.mdx * Update website/content/docs/upgrading/upgrade-to-1.13.x.mdx * Update website/content/docs/upgrading/upgrade-to-1.12.x.mdx * Add workaround to docs * Update changelog/20453.txt --------- Co-authored-by: Luis (LT) Carbonell <lt.carbonell@hashicorp.com> Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com> Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>

default max page size for config

3de4b64

ltcarbonell requested a review from yhyakuna as a code owner May 1, 2023 20:20

ltcarbonell requested a review from a team May 1, 2023 20:20

ltcarbonell mentioned this pull request May 1, 2023

Regression 1.13.2: "no LDAP groups found in groupDN" #20416

Closed

ltcarbonell requested a review from davidadeleon May 1, 2023 20:50

ltcarbonell added this to the 1.13.3 milestone May 1, 2023

ltcarbonell added backport/1.11.x labels May 1, 2023

Add changelog

26532e7

hghaf099 requested a review from austingebauer May 2, 2023 13:22

ltcarbonell added 2 commits May 2, 2023 08:49

update test int to *int

c95acc8

add testing defaults

5edd22c

ltcarbonell force-pushed the ldap-max-page-default branch from dc17a2b to 5edd22c Compare May 2, 2023 15:20

austingebauer reviewed May 4, 2023

View reviewed changes

sdk/helper/ldaputil/client.go Outdated Show resolved Hide resolved

ltcarbonell added 2 commits May 4, 2023 14:36

update default to -1, i.e. dont paginate

54e6c0d

update test

4b292a2

vercel bot deployed to Preview May 5, 2023 11:57 View deployment

ltcarbonell requested a review from austingebauer May 9, 2023 11:32

Add error message for invalid search

8749a5b

maxb reviewed May 10, 2023

View reviewed changes

ltcarbonell requested a review from jasonodonnell May 15, 2023 13:24

Make 0 the default

77750b8

vercel bot deployed to Preview May 16, 2023 14:25 View deployment

cleanup

066f35e

jasonodonnell approved these changes May 17, 2023

View reviewed changes

Add to known issues doc

a4ce574

Update website/content/docs/upgrading/upgrade-to-1.13.x.mdx

bb13939

vercel bot deployed to Preview May 17, 2023 16:06 View deployment

jasonodonnell reviewed May 17, 2023

View reviewed changes

website/content/docs/upgrading/upgrade-to-1.12.x.mdx Outdated Show resolved Hide resolved

jasonodonnell reviewed May 17, 2023

View reviewed changes

website/content/docs/upgrading/upgrade-to-1.13.x.mdx Outdated Show resolved Hide resolved

jasonodonnell reviewed May 17, 2023

View reviewed changes

website/content/docs/upgrading/upgrade-to-1.11.x.mdx Outdated Show resolved Hide resolved

ltcarbonell and others added 4 commits May 17, 2023 11:32

Update website/content/docs/upgrading/upgrade-to-1.11.x.mdx

b4d085f

Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>

Update website/content/docs/upgrading/upgrade-to-1.13.x.mdx

cd566a2

Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>

Update website/content/docs/upgrading/upgrade-to-1.12.x.mdx

5032278

Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>

Add workaround to docs

0920b12

vercel bot deployed to Preview May 17, 2023 16:41 View deployment

calvn reviewed May 17, 2023

View reviewed changes

austingebauer approved these changes May 17, 2023

View reviewed changes

changelog/20453.txt Outdated Show resolved Hide resolved

Update changelog/20453.txt

90e020a

Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>

ltcarbonell enabled auto-merge (squash) May 17, 2023 20:21

ltcarbonell merged commit 21b3262 into main May 17, 2023

ltcarbonell deleted the ldap-max-page-default branch May 17, 2023 20:56

This was referenced May 17, 2023

Backport of Correct Default for MaximumPageSize into release/1.11.x #20637

Merged

Backport of Correct Default for MaximumPageSize into release/1.12.x #20638

Merged

Backport of Correct Default for MaximumPageSize into release/1.13.x #20639

Merged

fopina-ci mentioned this pull request Jun 8, 2023

Upstream tag v1.13.3 (revision 3bedf816c) fopina/vault#51

Closed

cipherboy mentioned this pull request Jun 14, 2023

Vault 1.13.2 custom policy is not working as previous vault version i.e. Vault 1.13.0 #20979

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Correct Default for MaximumPageSize #20453

Correct Default for MaximumPageSize #20453

ltcarbonell commented May 1, 2023

maxb commented May 10, 2023

CodeGlitcher commented May 11, 2023

Cajga commented May 17, 2023 •

edited

Loading

calvn May 17, 2023

jasonodonnell May 17, 2023

calvn May 17, 2023

ltcarbonell commented May 17, 2023 •

edited

Loading

Correct Default for MaximumPageSize #20453

Correct Default for MaximumPageSize #20453

Conversation

ltcarbonell commented May 1, 2023

maxb commented May 10, 2023

CodeGlitcher commented May 11, 2023

Cajga commented May 17, 2023 • edited Loading

calvn May 17, 2023

Choose a reason for hiding this comment

jasonodonnell May 17, 2023

Choose a reason for hiding this comment

calvn May 17, 2023

Choose a reason for hiding this comment

ltcarbonell commented May 17, 2023 • edited Loading

Cajga commented May 17, 2023 •

edited

Loading

ltcarbonell commented May 17, 2023 •

edited

Loading