Unsanitize user and org names in DB #4762

pat-s · 2025-01-23T11:22:51Z

As discussed in chat, preferred to use non-sanitized values everywhere.

Store user and org names using the casing provided by the forge
Allow to search for users / orgs using case insensitive names
Removed all sanitation in the code
AFAICS only the org names were stored sanitized in the DB. User names and repo names are not affected

codecov · 2025-01-23T11:31:39Z

Codecov Report

Attention: Patch coverage is 40.00000% with 18 lines in your changes missing coverage. Please review.

Project coverage is 28.28%. Comparing base (fac744d) to head (e638c62).
Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
...ore/migration/024_unsanitize_org_and_user_names.go	39.28%	14 Missing and 3 partials ⚠️
server/store/datastore/migration/common.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4762      +/-   ##
==========================================
- Coverage   28.29%   28.28%   -0.01%     
==========================================
  Files         398      399       +1     
  Lines       28295    28318      +23     
==========================================
+ Hits         8005     8011       +6     
- Misses      19580    19594      +14     
- Partials      710      713       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

server/store/datastore/migration/024_sanitize_user_names.go

server/store/datastore/migration/024_unsanitize_org_and_user_names.go

server/store/datastore/org.go

…ames.go Co-authored-by: Robert Kaussow <mail@thegeeklab.de>

xoxys · 2025-01-23T16:21:59Z

Tests are failing.

server/store/datastore/migration/common.go

server/store/datastore/org.go

server/store/datastore/repo.go

Co-authored-by: Anbraten <6918444+anbraten@users.noreply.github.com>

xoxys · 2025-01-24T21:38:33Z

After thinking of it again, I'm not sure if thats the best approach. Cant really say why but I see a lot of potential for issues and corner cases. Looks like this issue only occurs with forgejo/gitea and the api returns mixed capitalization while the forge only supports case-insensitive values. What do you think?

Edit: Tested it. You can create an org "Foo" in gitea and in that case its even displayed as "Foo" in the url.... But you cant create "foo" with the error "The organization name is already taken." This is a somewhat inconsistent behavior upstream, however I think we should switch back to the initial approach from @pat-s

pat-s · 2025-01-27T23:47:18Z

If all users are sanitized, the user list in the UI would also reflect a non-optimal state.

The simplest fix which also wouldn't require a migration would probably be to just sanitize the comparison call of user.Login and org.Name?

qwerty287 · 2025-01-28T10:29:20Z

Why does the license header of the migration start with // Excerpt from:?

pat-s · 2025-01-28T11:38:54Z

Appreciate a reply on what should be done now / which solution is preferred. There are various potentially working solutions but I don't wanna refactor back and forth.

anbraten · 2025-01-28T12:35:12Z

Would store the forges casing in the database and allow api / url routes (the db find functions like orgFindByName) to return values case insensitive (similar to how the forges handle it).

xoxys · 2025-01-28T12:40:07Z

OK with it, but in that case Im still wondering how we want to handle it if multiple case-insensitive items are returned from the DB?

anbraten · 2025-01-28T14:05:40Z

OK with it, but in that case Im still wondering how we want to handle it if multiple case-insensitive items are returned from the DB?

Why would they? Forges do not allow projects with same name but different casing (at least for what we have checked). We migrate case insensitive and change to the casing from the forge.

xoxys · 2025-01-28T14:08:54Z

If they would or not doesn't matter IMO. In worst case we end up with "Foo" and "foo" in the DB (for whatever reason) as we store it case-sensitive but expect it to be case-insensitive. If we can add constraint to the column to prevent this, I'm fine with it, but otherwise this could lead to issues later.

anbraten · 2025-01-28T16:06:28Z

I don't think this should be an issue. If we check if an org exists using its case insensitive name or forge id to update or create it, we shouldn't have any trouble here. The unique constraint should actually be the forge-id + org-id anyways as multiple forges could use the same org names. Will check the code for both situations.

pat-s · 2025-01-31T13:10:55Z

Can somebody please summarize the resolution here now, so I can make the required adjustments?

xoxys · 2025-01-31T13:17:07Z

Would store the forges casing in the database and allow api / url routes (the db find functions like orgFindByName) to return values case insensitive (similar to how the forges handle it).

Thats the way to go.

I would like to ensure we can not store the same forge/org combination twice with differene casing, @anbraten wanted to check the code, see the last comment.

pat-s · 2025-01-31T13:40:01Z

So this PR is waiting for a review from @anbraten now, is this correct? AFAICS the PR is currently

storing the raw casing for org/usernames
sanitizing API calls

Please let me know if this needs more time overall, then I'll build a standalone image and apply this as a hotfix, so the issue on CB is solved (as there are already a few pending users waiting for a fix before they can be onboarded).

xoxys · 2025-01-31T23:44:35Z

For someone to verify that we have proper unique constraints in place.

anbraten · 2025-02-01T06:07:02Z

Checked the code and can't see a place that it would add it twice. There is some issue with multiple forges mixing orgs I've detected, but that's something else.

xoxys · 2025-02-01T07:37:24Z

Ok I dont have time to look into this on my own. Feel free to go on if you think the current state is fine.

xoxys · 2025-02-01T08:59:53Z

In best case we add a test case. Try to write "org1/foo" and "org1/Foo" to the db. If it fails, we are good to go if not we should still add db level restrictions.

pat-s · 2025-02-01T11:15:11Z

Added two tests which explicitly check that attempting to add a case-sensitive duplicate for orgs and users results in an error.

xoxys · 2025-02-01T12:08:37Z

Thanks!

server/store/datastore/migration/common.go

server/store/datastore/migration/024_unsanitize_org_and_user_names.go

Comments

woodpecker-bot · 2025-02-02T14:12:12Z

Deployment of preview was successful: https://woodpecker-ci-woodpecker-pr-4762.surge.sh

anbraten · 2025-02-02T14:11:41Z

server/api/login.go

@@ -183,29 +183,34 @@ func HandleAuth(c *gin.Context) {
 	// create or set the user's organization if it isn't linked yet
 	if user.OrgID == 0 {
 		// check if an org with the same name exists already and assign it to the user if it does
-		if org, err := _store.OrgFindByName(user.Login); err == nil && org != nil {
+		// TODO: find the org by name and forgeID directly
+		org, err := _store.OrgFindByName(user.Login)


I've noticed this bug as the error was not handled by the if below as it was inline in the previous if.

xoxys · 2025-02-02T14:22:43Z

Code LGTM, have you tested the PR locally?

pat-s added 2 commits January 23, 2025 12:20

fix: add user migration

ecc7541

chore: update years

dce8939

pat-s added bug Something isn't working server labels Jan 23, 2025

pat-s added 2 commits January 23, 2025 13:54

chore: update migration to unsanitize

1aa34c1

chore: remove all toLower calls

19ac5be

qwerty287 reviewed Jan 23, 2025

View reviewed changes

server/store/datastore/migration/024_sanitize_user_names.go Outdated Show resolved Hide resolved

adjust tests to new migration

291b1be

pat-s changed the title ~~Sanitize user names in DB to match org names~~ Unsanitize user and org names in DB Jan 23, 2025

chore: spell

6a71341

pat-s mentioned this pull request Jan 23, 2025

fix: remove lower casing from orgs #4763

Closed

2 tasks

xoxys reviewed Jan 23, 2025

View reviewed changes

pat-s and others added 5 commits January 23, 2025 16:52

Update server/store/datastore/migration/024_unsanitize_org_and_user_n…

c0264f8

…ames.go Co-authored-by: Robert Kaussow <mail@thegeeklab.de>

Update server/store/datastore/migration/024_unsanitize_org_and_user_n…

fd289fa

…ames.go Co-authored-by: Robert Kaussow <mail@thegeeklab.de>

Update server/store/datastore/migration/024_unsanitize_org_and_user_n…

d05055b

…ames.go Co-authored-by: Robert Kaussow <mail@thegeeklab.de>

Update server/store/datastore/migration/024_unsanitize_org_and_user_n…

0df49fa

…ames.go Co-authored-by: Robert Kaussow <mail@thegeeklab.de>

chore: remove org test

1c3f31e

pat-s added 3 commits January 23, 2025 17:26

chore: update test

66da958

chore: format

f2df9a2

chore: adjust more tests

3f297ad

anbraten reviewed Jan 23, 2025

View reviewed changes

server/store/datastore/migration/common.go Outdated Show resolved Hide resolved

server/store/datastore/org.go Outdated Show resolved Hide resolved

server/store/datastore/repo.go Outdated Show resolved Hide resolved

pat-s and others added 4 commits January 23, 2025 18:15

Update server/store/datastore/org.go

16f2469

Co-authored-by: Anbraten <6918444+anbraten@users.noreply.github.com>

chore: review

a2d5946

import

8e0fc7a

Merge branch 'main' into fix/sanitize-usernames-db

1771ce8

clean header

e638c62

add tests checking case-sensitive DB duplicates

69c7d75

Merge branch 'main' into fix/sanitize-usernames-db

d354425

xoxys previously approved these changes Feb 1, 2025

View reviewed changes

anbraten reviewed Feb 1, 2025

View reviewed changes

server/store/datastore/migration/common.go Outdated Show resolved Hide resolved

server/store/datastore/migration/024_unsanitize_org_and_user_names.go Show resolved Hide resolved

server/store/datastore/migration/024_unsanitize_org_and_user_names.go Show resolved Hide resolved

pat-s and others added 2 commits February 2, 2025 09:58

restore toLower

73d4f19

improve user org handling

aa726dc

anbraten reviewed Feb 2, 2025

View reviewed changes

anbraten requested a review from xoxys February 2, 2025 14:13

anbraten mentioned this pull request Feb 2, 2025

Admin "repair" mode #4807

Open

fix test

31f31c4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unsanitize user and org names in DB #4762

Unsanitize user and org names in DB #4762

pat-s commented Jan 23, 2025 •

edited by anbraten

Loading

codecov bot commented Jan 23, 2025 •

edited

Loading

xoxys commented Jan 23, 2025

xoxys commented Jan 24, 2025 •

edited

Loading

pat-s commented Jan 27, 2025

qwerty287 commented Jan 28, 2025

pat-s commented Jan 28, 2025

anbraten commented Jan 28, 2025

xoxys commented Jan 28, 2025

anbraten commented Jan 28, 2025 •

edited

Loading

xoxys commented Jan 28, 2025

anbraten commented Jan 28, 2025 •

edited

Loading

pat-s commented Jan 31, 2025

xoxys commented Jan 31, 2025 •

edited

Loading

pat-s commented Jan 31, 2025

xoxys commented Jan 31, 2025

anbraten commented Feb 1, 2025

xoxys commented Feb 1, 2025

xoxys commented Feb 1, 2025

pat-s commented Feb 1, 2025

xoxys commented Feb 1, 2025

woodpecker-bot commented Feb 2, 2025 •

edited

Loading

anbraten Feb 2, 2025

xoxys Feb 2, 2025

xoxys commented Feb 2, 2025

Unsanitize user and org names in DB #4762

Are you sure you want to change the base?

Unsanitize user and org names in DB #4762

Conversation

pat-s commented Jan 23, 2025 • edited by anbraten Loading

codecov bot commented Jan 23, 2025 • edited Loading

Codecov Report

xoxys commented Jan 23, 2025

xoxys commented Jan 24, 2025 • edited Loading

pat-s commented Jan 27, 2025

qwerty287 commented Jan 28, 2025

pat-s commented Jan 28, 2025

anbraten commented Jan 28, 2025

xoxys commented Jan 28, 2025

anbraten commented Jan 28, 2025 • edited Loading

xoxys commented Jan 28, 2025

anbraten commented Jan 28, 2025 • edited Loading

pat-s commented Jan 31, 2025

xoxys commented Jan 31, 2025 • edited Loading

pat-s commented Jan 31, 2025

xoxys commented Jan 31, 2025

anbraten commented Feb 1, 2025

xoxys commented Feb 1, 2025

xoxys commented Feb 1, 2025

pat-s commented Feb 1, 2025

xoxys commented Feb 1, 2025

woodpecker-bot commented Feb 2, 2025 • edited Loading

anbraten Feb 2, 2025

Choose a reason for hiding this comment

xoxys Feb 2, 2025

Choose a reason for hiding this comment

xoxys commented Feb 2, 2025

pat-s commented Jan 23, 2025 •

edited by anbraten

Loading

codecov bot commented Jan 23, 2025 •

edited

Loading

xoxys commented Jan 24, 2025 •

edited

Loading

anbraten commented Jan 28, 2025 •

edited

Loading

anbraten commented Jan 28, 2025 •

edited

Loading

xoxys commented Jan 31, 2025 •

edited

Loading

woodpecker-bot commented Feb 2, 2025 •

edited

Loading