Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Currently online display requests all currently online users from API, thus eventually exceeding rate limits #29982

Open
bdach opened this issue Sep 24, 2024 · 7 comments

Comments

@bdach
Copy link
Collaborator

bdach commented Sep 24, 2024

Originally posted by @bdach in #29968 (comment)

Here's a log on this PR showing it still happening.

Well that was a bit of misdirection wasn't it, since the original snippet you posted showed the beatmap set request failing, but I'd be willing to bet money on the fact that it's all of these that cause the ratelimit to get exceeded:

2024-09-23 14:52:21 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=36732905&ids[]=35262412&ids[]=35875909&ids[]=29867634&ids[]=18633882&ids[]=20822457&ids[]=33201184&ids[]=18230719&ids[]=34489118&ids[]=6082303&ids[]=9049614&ids[]=10915715&ids[]=34555317&ids[]=2605324&ids[]=34399173&ids[]=33622436&ids[]=9290272&ids[]=36725275&ids[]=30553311&ids[]=36634355&ids[]=7093548&ids[]=20084327&ids[]=33407487&ids[]=34429696&ids[]=9961743&ids[]=15280942&ids[]=34035397&ids[]=21478222&ids[]=36692537&ids[]=29876889&ids[]=36182515&ids[]=12485807&ids[]=36245822&ids[]=36660354&ids[]=8985720&ids[]=22096637&ids[]=13440678&ids[]=15595510&ids[]=19972098&ids[]=36179593&ids[]=13274097&ids[]=35197617&ids[]=3128601&ids[]=10411604&ids[]=19886894&ids[]=36724819&ids[]=11358214&ids[]=27621192&ids[]=15967940&ids[]=14471463 successfully completed!
2024-09-23 14:52:21 [verbose]: GetUsersRequest finished with response size of 100,825 bytes
2024-09-23 14:52:21 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:21 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=36256780&ids[]=35607852&ids[]=21683307&ids[]=36534979&ids[]=36563271&ids[]=36715683&ids[]=10440535&ids[]=36200271&ids[]=12760514&ids[]=36598466&ids[]=35451125&ids[]=35205599&ids[]=19315458&ids[]=36536998&ids[]=34491452&ids[]=10047047&ids[]=30849975&ids[]=35075540&ids[]=33357362&ids[]=16769621&ids[]=24193502&ids[]=16884064&ids[]=15720951&ids[]=20782611&ids[]=14915346&ids[]=35024623&ids[]=35226190&ids[]=31473916&ids[]=19228035&ids[]=33309529&ids[]=24026160&ids[]=10192982&ids[]=29490192&ids[]=36559045&ids[]=35270752&ids[]=33946246&ids[]=36372274&ids[]=11139822&ids[]=7601737&ids[]=10839558&ids[]=36729023&ids[]=33590999&ids[]=17119609&ids[]=32448556&ids[]=33485239&ids[]=34849185&ids[]=6763446&ids[]=36494718&ids[]=36428479&ids[]=3371797 successfully completed!
2024-09-23 14:52:21 [verbose]: GetUsersRequest finished with response size of 98,050 bytes
2024-09-23 14:52:21 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:21 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=36206865&ids[]=34891056&ids[]=15308543&ids[]=23853662&ids[]=36636445&ids[]=36722342&ids[]=34548393&ids[]=26680420&ids[]=36732629&ids[]=19188154&ids[]=33250965&ids[]=36159593&ids[]=36634268&ids[]=3581753&ids[]=34024700&ids[]=33635230&ids[]=12612018&ids[]=28163355&ids[]=33157119&ids[]=19019042&ids[]=36540414&ids[]=24548042&ids[]=36290492&ids[]=36738436&ids[]=15158331&ids[]=14558166&ids[]=36734335&ids[]=16339271&ids[]=4615046&ids[]=14130446&ids[]=25950535&ids[]=36003268&ids[]=23424019&ids[]=32477801&ids[]=1483160&ids[]=36339083&ids[]=36668541&ids[]=36692583&ids[]=35392462&ids[]=35896616&ids[]=9230310&ids[]=33991360&ids[]=11327608&ids[]=11854911&ids[]=21706431&ids[]=20660557&ids[]=33361386&ids[]=24521426&ids[]=36732826&ids[]=34586280 successfully completed!
2024-09-23 14:52:21 [verbose]: GetUsersRequest finished with response size of 99,290 bytes
2024-09-23 14:52:21 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:22 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=36738179&ids[]=16215405&ids[]=36254961&ids[]=36176782&ids[]=33103283&ids[]=24778953&ids[]=11291084&ids[]=36585772&ids[]=28200630&ids[]=30023521&ids[]=34215242&ids[]=2632240&ids[]=4476818&ids[]=36478491&ids[]=33496129&ids[]=35502219&ids[]=16446112&ids[]=2503170&ids[]=20287733&ids[]=35015946&ids[]=32357147&ids[]=17821735&ids[]=15496039&ids[]=33180552&ids[]=33850381&ids[]=35637077&ids[]=35801873&ids[]=36495328&ids[]=33915524&ids[]=25054049&ids[]=32923736&ids[]=36732015&ids[]=32917135&ids[]=35831195&ids[]=36503785&ids[]=26008189&ids[]=32507269&ids[]=36694881&ids[]=7173453&ids[]=36301670&ids[]=36447635&ids[]=35429541&ids[]=17686915&ids[]=8370475&ids[]=21843822&ids[]=32322458&ids[]=29071427&ids[]=20746542&ids[]=21753748&ids[]=36155608 successfully completed!
2024-09-23 14:52:22 [verbose]: GetUsersRequest finished with response size of 103,764 bytes
2024-09-23 14:52:22 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:22 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=11912649&ids[]=36320162&ids[]=36016646&ids[]=36167473&ids[]=18813137&ids[]=34848135&ids[]=34838867&ids[]=13737273&ids[]=31325331&ids[]=3030502&ids[]=36738569&ids[]=36720321&ids[]=34032576&ids[]=36590886&ids[]=6572940&ids[]=36411962&ids[]=16896580&ids[]=30058248&ids[]=35931211&ids[]=32910101&ids[]=19010333&ids[]=34641461&ids[]=36689884&ids[]=9626734&ids[]=34074744&ids[]=25156562&ids[]=33038882&ids[]=17848299&ids[]=29944108&ids[]=15854832&ids[]=35194093&ids[]=10993158&ids[]=16597584&ids[]=33893911&ids[]=35372081&ids[]=22521306&ids[]=22123643&ids[]=17368545&ids[]=23765997&ids[]=26544222&ids[]=17269597&ids[]=35054605&ids[]=16161649&ids[]=25924476&ids[]=21165636&ids[]=34723790&ids[]=24515690&ids[]=34837309&ids[]=31065309&ids[]=7982003 successfully completed!
2024-09-23 14:52:22 [verbose]: GetUsersRequest finished with response size of 98,377 bytes
2024-09-23 14:52:22 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:22 [verbose]: Request to https://osu.ppy.sh/images/layout/avatar-guest@2x.png successfully completed!
2024-09-23 14:52:22 [verbose]: Request to https://a.ppy.sh/33467060?1682039646.jpeg successfully completed!
2024-09-23 14:52:22 [verbose]: Performing request osu.Game.Online.API.Requests.GetBeatmapsRequest
2024-09-23 14:52:22 [verbose]: Request to https://osu.ppy.sh/api/v2/users/?ids[]=36337052&ids[]=10730875&ids[]=36729403&ids[]=8595858&ids[]=13268182&ids[]=15037158&ids[]=25232008&ids[]=10674648&ids[]=24628787&ids[]=36393336&ids[]=24591855&ids[]=36595378&ids[]=36150562&ids[]=31402882&ids[]=36730640&ids[]=36206724&ids[]=18014083&ids[]=19331260&ids[]=17697613&ids[]=36733066&ids[]=35998860&ids[]=9519610&ids[]=31003097&ids[]=33351652&ids[]=25781802&ids[]=32563747&ids[]=34119064&ids[]=31861680&ids[]=33024696&ids[]=36738593&ids[]=19114005&ids[]=6904819&ids[]=36738536&ids[]=33819981&ids[]=34673097&ids[]=12579466&ids[]=36677444&ids[]=35998757&ids[]=35263887&ids[]=17712448&ids[]=36094734&ids[]=36734703&ids[]=12697651&ids[]=16124176&ids[]=35059914&ids[]=29405230&ids[]=35920077&ids[]=32606396&ids[]=36469769&ids[]=36271263 successfully completed!
2024-09-23 14:52:22 [verbose]: GetUsersRequest finished with response size of 93,650 bytes
2024-09-23 14:52:22 [verbose]: Performing request osu.Game.Online.API.Requests.GetUsersRequest
2024-09-23 14:52:22 [verbose]: Request to https://osu.ppy.sh/api/v2/beatmaps/?ids[]=1219690 failed with System.Net.WebException: TooManyRequests.
2024-09-23 14:52:22 [verbose]: Response was: {"error":"Too Many Attempts."}
2024-09-23 14:52:22 [verbose]: Failing request osu.Game.Online.API.Requests.GetBeatmapsRequest (osu.Game.Online.API.APIException: Too Many Attempts.
2024-09-23 14:52:22 [verbose]: ---> System.Net.WebException: TooManyRequests
2024-09-23 14:52:22 [verbose]: --- End of inner exception stack trace ---)

To resolve this we'd probably want pagination on that display there, except for the fact that there's a search box in there that allows searching by username, and we can't paginate with username search before these requests, because we only have user IDs to work with at that point....

Would probably need spectator server changes to at least send usernames across the wire. If not more.

@bdach bdach changed the title Currently online display doesn't paginate and as such requests all currently online users from API, thus eventually exceeding rate limits Currently online requests all currently online users from API, thus eventually exceeding rate limits Sep 24, 2024
@bdach
Copy link
Collaborator Author

bdach commented Sep 24, 2024

Elsewhere:

Bancho allowed lookups this fast in order for this screen to exist. We probably need the osu-web end rate limit adjusted. Paginating would be a bit of a step backwards from people's expectations, I think.

@ppy/team-web would adjusting the rate limit be feasible?

For better context, this is the screen that is triggering this:

osu_2024-09-24_14-10-21

@bdach bdach changed the title Currently online requests all currently online users from API, thus eventually exceeding rate limits Currently online display requests all currently online users from API, thus eventually exceeding rate limits Sep 24, 2024
@nanaya
Copy link

nanaya commented Sep 24, 2024

the endpoint probably returns way too much data for what's being used there though. There's a /users/lookup-users but it's not available for api yet. I'd probably rather expose that one with higher limit instead.

@nanaya
Copy link

nanaya commented Sep 25, 2024

@bdach can you check if the data returned is enough? It's mainly missing statistics_rulesets compared to the /users one

example
{
	"users": [
		{
			"avatar_url": "https://a.ppy.sh/2?1657169614.png",
			"country_code": "AU",
			"default_group": "default",
			"id": 2,
			"is_active": true,
			"is_bot": false,
			"is_deleted": false,
			"is_online": false,
			"is_supporter": true,
			"last_visit": "2024-09-25T07:18:12+00:00",
			"pm_friends_only": false,
			"profile_colour": "#3366FF",
			"username": "peppy",
			"country": {
				"code": "AU",
				"name": "Australia"
			},
			"cover": {
				"custom_url": "https://assets.ppy.sh/user-profile-covers/2/baba245ef60834b769694178f8f6d4f6166c5188c740de084656ad2b80f1eea7.jpeg",
				"url": "https://assets.ppy.sh/user-profile-covers/2/baba245ef60834b769694178f8f6d4f6166c5188c740de084656ad2b80f1eea7.jpeg",
				"id": null
			},
			"groups": [
				{
					"colour": "#0066FF",
					"has_listing": false,
					"has_playmodes": false,
					"id": 33,
					"identifier": "ppy",
					"is_probationary": false,
					"name": "ppy",
					"short_name": "PPY",
					"playmodes": null
				},
				{
					"colour": "#E45678",
					"has_listing": true,
					"has_playmodes": false,
					"id": 11,
					"identifier": "dev",
					"is_probationary": false,
					"name": "Developers",
					"short_name": "DEV",
					"playmodes": null
				}
			]
		}
	]
}

@bdach
Copy link
Collaborator Author

bdach commented Sep 25, 2024

As far as I can tell the client is using:

  • ✔️ id
  • ✔️ username
  • ✔️ avatar_url
  • ✔️ cover.url
  • ✔️ country_code
  • ✔️ last_visit
  • ✔️ is_online
  • ✔️ is_supporter
  • support_level

groups is also good to have even if the client isn't using it (we really should be).

Looking at the web friends page which doesn't have support level shown, maybe we can do without support_level though...?

@nanaya
Copy link

nanaya commented Sep 25, 2024

I don't think support level is returned in the other endpoint either. Its value is calculated every time iirc.

@bdach
Copy link
Collaborator Author

bdach commented Sep 25, 2024

Oh you're right actually. The client doesn't show any supporter markers on the currently online display, but it does on the friends display.

Yeah I think we can work with that set of fields then. We'll fix client to display supporter badge even if there is no level (is it even possible for API to return a user with {"is_supporter":1,"support_level":0}?)

@nanaya
Copy link

nanaya commented Sep 25, 2024

oh yeah the friend and group page do show it for "list mode" o_O

that's weird...

also no, the minimum value is 1 for current supporter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants