Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

“The application did not respond” - error within discord #9

Open
andrewmcd7 opened this issue Feb 10, 2024 · 69 comments
Open

“The application did not respond” - error within discord #9

andrewmcd7 opened this issue Feb 10, 2024 · 69 comments
Labels
bug Something isn't working

Comments

@andrewmcd7
Copy link

andrewmcd7 commented Feb 10, 2024

Hey, thanks for all your time and effort on this bot. It’s great when it works however we are getting almost daily “the application did not respond” within discord.

We manage a few discord servers. So we have an about 5 requestrr containers.. actually 10… 5 for regular requests and 5 for foreign requests that get added to different arr’s.

We split the containers we use between all of the discord channels to split the load. But a lot lately we are getting this error. We dont know if it’s a requestrr error or an issue with discord api. A lot of the time we either have to reboot the container (sometimes is fixes it), redeploy the container, and if that doesn’t work kick the bot and readd it to discord. It was manageable at first but now we are getting it more and more.

Im hoping we can work together to resolve this issue.
IMG_6565

@thomst08
Copy link
Owner

Hey @andrewmcd7,

That sounds like an interesting setup, this might be abit tricky.

You could be right, it might be an issue with the discord api, but the only way we can find out is check the logs of each container when they error and capture why it errored.
Can you collect a copy of the logs (in text form) and post them?
That way it should give an idea why it's happening.

If your unsure how to get the logs, this link might help: https://github.com/thomst08/requestrr/wiki/Where-are-the-logs%3F

Let me know how you go. :)

@andrewmcd7
Copy link
Author

I will next error. Thank you

@andrewmcd7
Copy link
Author

Hope this is helpful.

warn: Requestrr.WebApi.RequestrrBot.ChatBot[0]
Bot has been restarted.
warn: Requestrr.WebApi.RequestrrBot.ChatBot[0]
Removing movie notification for user with ID 979274836897595462 as it could not be found in any of the guilds.
[2024-02-10 20:27:05 +02:00] [111 /ConnectionCl] [Crit ] Connection terminated (4000, ''), reconnecting
[2024-02-10 20:44:19 +02:00] [111 /ConnectionCl] [Crit ] Connection terminated (4000, ''), reconnecting
[2024-02-10 22:25:35 +02:00] [105 /WebSocketRec] [Warn ] Unknown event: VOICE_CHANNEL_STATUS_UPDATE
payload: "status": null, "id": "954093177139253318", "guild_id": "953038263797563422"
[2024-02-11 00:00:56 +02:00] [111 /ConnectionCl] [Crit ] Connection terminated (4000, ''), reconnecting
warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
Failed to determine the https port for redirect.
Application is shutting down...
Starting Requestrr - build v2.1.4 - Hotfix 4
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
Hosting environment: Production
Content root path: /root
Now listening on: http://[::]:4545
Application started. Press Ctrl+C to shut down.
warn: Requestrr.WebApi.RequestrrBot.ChatBot[0]
Bot configuration changed: restarting bot
warn: Requestrr.WebApi.RequestrrBot.ChatBot[0]
Bot has been restarted.
warn: Requestrr.WebApi.RequestrrBot.ChatBot[0]
Bot configuration changed: restarting bot
[2024-02-11 01:11:36 +02:00] [0 / ] [Crit ] There was an error registering application commands
System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.List1.set_Capacity(Int32 value) at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<<RegisterCommands>b__0>d.MoveNext() [2024-02-11 01:11:36 +02:00] [0 / ] [Crit ] There was an error registering application commands System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray') at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable) at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection)
at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<b__0>d.MoveNext()
[2024-02-11 01:11:36 +02:00] [0 / ] [Crit ] There was an error registering application commands
System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.List1.set_Capacity(Int32 value) at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<<RegisterCommands>b__0>d.MoveNext() [2024-02-11 01:11:36 +02:00] [0 / ] [Crit ] There was an error registering application commands System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray') at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable) at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection)
at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<b__0>d.MoveNext()
[2024-02-11 01:11:36 +02:00] [0 / ] [Crit ] There was an error registering application commands
System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.List1.set_Capacity(Int32 value) at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<<RegisterCommands>b__0>d.MoveNext() [2024-02-11 01:11:36 +02:00] [0 / ] [Crit ] There was an error registering application commands System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray') at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable) at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection)
at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<b__0>d.MoveNext()
[2024-02-11 01:11:36 +02:00] [0 / ] [Crit ] There was an error registering application commands
System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.List1.set_Capacity(Int32 value) at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<<RegisterCommands>b__0>d.MoveNext() [2024-02-11 01:11:37 +02:00] [0 / ] [Crit ] There was an error registering application commands System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray') at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable) at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection)
at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<b__0>d.MoveNext()
[2024-02-11 01:11:37 +02:00] [0 / ] [Crit ] There was an error registering application commands
System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.List1.set_Capacity(Int32 value) at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<<RegisterCommands>b__0>d.MoveNext() [2024-02-11 01:11:37 +02:00] [0 / ] [Crit ] There was an error registering application commands System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray') at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable) at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection)
at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<b__0>d.MoveNext()
[2024-02-11 01:11:39 +02:00] [0 / ] [Crit ] There was an error registering application commands
System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.List1.set_Capacity(Int32 value) at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<<RegisterCommands>b__0>d.MoveNext() [2024-02-11 01:11:39 +02:00] [0 / ] [Crit ] There was an error registering application commands System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray') at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable) at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection)
at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<b__0>d.MoveNext()
warn: Requestrr.WebApi.RequestrrBot.ChatBot[0]
Bot has been restarted.
[2024-02-11 04:19:29 +02:00] [111 /ConnectionCl] [Crit ] Connection terminated (4000, ''), reconnecting

@thomst08
Copy link
Owner

Thanks for posting your error log, I have seen an error similar to what you have and I think I have replicated it.
I will need to test some more to verify the issue then correct it, but can I ask if you have "Hide slash command request messages in channels" enabled?

@andrewmcd7
Copy link
Author

I believe they are enabled. I just briefly looked and cannot see any settings for that. Where would it be? And should they be enabled or disabled.

@thomst08
Copy link
Owner

It can be enabled or disabled, I was wondering if that is the culprit, but from testing I don't think so, it's in the "Chat client" section.

It appears to be with the building of the slash commands when the bot is reset, my guess there could be a network issue and this might be part of the issue as it's not reconnecting to discord... Do you have any special network setting for your containers? And do you have any roles setup in the "Chat clients" section?

@andrewmcd7
Copy link
Author

Let me find out about the special network settings.

No special roles.

I invite the bot to discord, add the channel ID TO REQUESTRR and everything is good for about a day before it starts erroring.

@andrewmcd7
Copy link
Author

We do have I think 5 or 6 requestrrs on 1 host. All named differently. Req1, req2, req3, etc.

I have reached out to the guy who sets them up to see if there’s any weird network settings he’s using.

@andrewmcd7
Copy link
Author

No weird network settings on the containers in portainer.

@andrewmcd7
Copy link
Author

Here’s some more logs on a different container. Same issue:

     at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
     at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
     at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
     at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
     at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
     at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
     at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
     at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
     at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
     at Requestrr.WebApi.RequestrrBot.DownloadClients.Radarr.RadarrClientV3.HttpGetAsync(HttpClient client, RadarrSettings settings, String url) in D:\Users\thoms\Documents\GitHub\requestrr\Requestrr.WebApi\RequestrrBot\DownloadClients\Radarr\RadarrClientV3.cs:line 431
     at Requestrr.WebApi.RequestrrBot.DownloadClients.Radarr.RadarrClientV3.FindExistingMovieByMovieDbIdAsync(Int32 theMovieDbId) in D:\Users\thoms\Documents\GitHub\requestrr\Requestrr.WebApi\RequestrrBot\DownloadClients\Radarr\RadarrClientV3.cs:line 405

fail: Requestrr.WebApi.RequestrrBot.DownloadClients.Radarr.RadarrClient[0]
An error occurred finding an existing movie by tmdbId "37541" with Radarr: Name or service not known (radarr:7878)
System.Net.Http.HttpRequestException: Name or service not known (radarr:7878)
---> System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Requestrr.WebApi.RequestrrBot.DownloadClients.Radarr.RadarrClientV3.HttpGetAsync(HttpClient client, RadarrSettings settings, String url) in D:\Users\thoms\Documents\GitHub\requestrr\Requestrr.WebApi\RequestrrBot\DownloadClients\Radarr\RadarrClientV3.cs:line 431
at Requestrr.WebApi.RequestrrBot.DownloadClients.Radarr.RadarrClientV3.FindExistingMovieByMovieDbIdAsync(Int32 theMovieDbId) in D:\Users\thoms\Documents\GitHub\requestrr\Requestrr.WebApi\RequestrrBot\DownloadClients\Radarr\RadarrClientV3.cs:line 405

@thomst08
Copy link
Owner

Hmmm, they are interesting errors, I will do some more testing and try to find a reason for this.

Where the containers working perfectly on version 2.1.2 of requestrr? (about two months back)

@andrewmcd7
Copy link
Author

No. We have been plagued by this for a while.

@thomst08
Copy link
Owner

This might take me a while to pin point.
I will keep thinking about this and try to come up with an answer.

Just to double check, are all the services you connect requestrr too all on the one server?
Is there any known network issues such as drop-outs with the server?

@andrewmcd7
Copy link
Author

Any progress? Not sure what else we can try and do.

@thomst08
Copy link
Owner

Hey @andrewmcd7,

Not at the moment, this is still in the back of my mind and I am looking for issues but haven't seen any links to what you have been expecting.

There is a few things to note however, I have a few long term plans, one of which is to setup requestrr to allow to handle multiple sources (multiple radarr connections and using categories to control each one) and to also change how the slash commands are created.

Both of these are going to be a while and might help your setup, but also don't solve the underlying issue your having, at the moment.

I was still wondering if any of the traffic was through a vpn setup on the network that might be part of the issue, but I'm not sure.
I'll keep exploring, if I find something I will let you know.

@thomst08
Copy link
Owner

Hey @andrewmcd7,

Sorry this has taken me a while, I think I have some solutions to your problem, I don't think this is an issue with the program, but it might be the setup...

First, the second issue you mentioned, this issue has been fixed in the last update, this is because some movies or shows get removed from the TVDB or TMDB and then requestrr can no longer match them, the newest update removes these form the notifications, this should not have been stopping the program, but does fill the logs.

The first issue you noted...
I haven't found an EXACT reasons, but this might be to do with your settings or a miss configuration.
I managed to emulate this issue by modifying my language file, this didn't give me your exact issue, but something close. If you are using a custom language file, please recreate it from one of the latest version as new properties have been added.
If that's not the case, there is a chance your settings might be causing the issue as some in your config will get loaded into the commands, this could cause the issue, it might be worth clearing the settings and rebuilding them, or tweaking each setting and removing weird characters if you have any ()[],
That could upset the building of the commands.

Let me know if any of this helped, I will keep exploring our options if you are still having issues.

@andrewmcd7
Copy link
Author

Thank you! I'm not sure what you are referencing wrt language file, we haven't modified the config. If it helps, without touching any settings restarting it a few times seems to push past the error. Is it possible this error is some sort of discord IP or bot rate limit?

@thomst08
Copy link
Owner

That's ok, in short, you can setup custom language files for you bot, allowing you to setup for another language if needed, however, if you are just using English, then this can be ignored.
It is possible this is a rate limit yes, I forgot to mention that sorry, I haven't found a way to trigger this yet, but there is a fix coming.
DSharp is releasing a new update soon that fixes a rate limit issue, when it's out, I will update the program, we can see if that helps.

Again, it could be a discord error still, but from the errors, it appears to be a building error of the slash commands or at least when its sent to Discord it is rejected.

@edrock200
Copy link

Just to add to this, seeing similar errors and restarting several times or proxying seems to resolve, which lends itself to the theory that a rate limit is occuring. One thing I've noticed is right at the start of requestrr, in addition to registering the bot, it attempts to push out any backlogged notifications in rapid succession. Some of them are for users that no longer exist. Deleting notifications.json resolved this, but I'm wondering if combination of the two is also causing rate limiting. Would it be possible to delay the notifications and prune notifications that return "user not found?" @thomst08

@thomst08
Copy link
Owner

Hey @edrock200,
Thanks for your input, that is a good idea, I will look into that part.
I'm not sure a delay is the right step, but handling missing users should be something to look into....
I'll definitely dig into this

@edrock200
Copy link

Hey @edrock200, Thanks for your input, that is a good idea, I will look into that part. I'm not sure a delay is the right step, but handling missing users should be something to look into.... I'll definitely dig into this

Thank you! Delay was the wrong word. I meant a throttle of notification alerts, e.g. no more than xx a minute.

@thomst08
Copy link
Owner

Hey everyone, just a small update.

Still testing and make a link, I looked over the notifications part, and while that can trigger a lot of requests, I have found that if a user is no longer on your server, it should remove the request once its active, if there is more then 10~ requests it will rate limit, report it in the long and try again in another minute, This part seems to be handling fine and isn't taking down the main bot as it seems to still be functioning. I don't think it in the notification side causing a limit problem in this case by I will keep testing here in other situations to best work it out...

I have, however, started to look over the main resetting of the bot and setup, it seems if there is a break in the slash commands the bot should restart and try to rebuild itself, however, there's a chance this doesn't always happen. So, I am looking into a way of check if the bot broken down and failed to reset and reset itself if this happens, there was no check if DSharp's library fails.
I have been keeping an eye on DSharps releases and still waiting for their next release that will have some changes to the rate limiter and a few other things to check if the bot is connected. I might wait until this is released before updating the program, but I will keep testing and try to come up with a solution that works for everyone.

TLDR: Tested notifications, found some cases where there could be an issue, not confident enough in the fix, needs more testing, working on a fix.

@edrock200
Copy link

Thank you!

@thomst08
Copy link
Owner

thomst08 commented Mar 6, 2024

Small update, looking into this a bit more.
The main error when registering the slash commands is building the commands, then sending it off to DSharp to send it to Discord, while I can attach error handling functions, none of them seem to catch this error in DSharp, once DSharp errors, this seems to crash DSharps slash register completely, to the point if the bot is reset in program it does not re-register, it appears to not even respond or attempt to register the commands again...

Hopefully when the next update is out this might correct this issue, but DSharp v5 will allow registering dynamic slash commands during run time witch will hopefully solve the root of this issue. (I hope to not have to wait too long to resolve this)

@edrock200
Copy link

Small update, looking into this a bit more. The main error when registering the slash commands is building the commands, then sending it off to DSharp to send it to Discord, while I can attach error handling functions, none of them seem to catch this error in DSharp, once DSharp errors, this seems to crash DSharps slash register completely, to the point if the bot is reset in program it does not re-register, it appears to not even respond or attempt to register the commands again...

Hopefully when the next update is out this might correct this issue, but DSharp v5 will allow registering dynamic slash commands during run time witch will hopefully solve the root of this issue. (I hope to not have to wait too long to resolve this)

Thanks for the info and hard work! In the meantime is their a specific phrase we can grep for in the logs when it found, to trigger a log clear and restart?

@thomst08
Copy link
Owner

thomst08 commented Mar 6, 2024

@edrock200, good question.
I would look for "There was an error registering application commands".
Once this happens, you loss your slash commands or the bot becomes unresponsive.

@thomst08
Copy link
Owner

thomst08 commented Mar 14, 2024

Hey @andrewmcd7 and @edrock200,

Just giving you both a heads up, I have just released version 2.1.6, this contains a few things that link back to these issues.
First, there are a few tweaks in how the bot builds its commands and retrying, these changes may help correct your issues or at lease help prevent others and maybe give us more context.

I have also changed where the temp files are created when building the commands, again, this might help prevent your issue.

I don't expect this version to solve your issues as it does not solve the underlining issue, but it might help, please let me know if there are any changes and post any new logs you have.

@edrock200
Copy link

Hey @andrewmcd7 and @edrock200,

Just giving you both a heads up, I have just released version 2.1.6, this contains a few this that link back to these issues. First, there are a few tweaks in how the bot builds its commands and retrying, this changes may help correct your issues or at lease help prevent others and maybe give us more context.

I have also changed where the temp files are created when building the commands, again, this might help prevent your issue.

I don't expect this version to solve your issues as it does not solve the underlining issue, but it might help, please let me know if there are any changes and post any new logs you have.

Thank you so much!

@CapnClunge
Copy link

I've been getting the same error pretty consistently for a few weeks now; only about 1 in every 5 requests actually works. Restarting the container doesn't seem to help, nor does restarting the whole server. It just starts working again seemingly at random. This also occurs whether I request in a channel or through direct message. Below is the resulting log after attempting to request the same movie both by title and by TMDB:

fail: Requestrr.WebApi.RequestrrBot.ChatBot[0]
      Error while searching movies with title the one : Not found: 404
      DSharpPlus.Exceptions.NotFoundException: Not found: 404
         at DSharpPlus.Net.DiscordApiClient.CreateInteractionResponseAsync(UInt64 interaction_id, String interaction_token, InteractionResponseType type, DiscordInteractionResponseBuilder builder)
         at Requestrr.WebApi.RequestrrBot.SlashCommands.RequestGroup.MovieByTitle99999(InteractionContext ctx, String movieTitle)
fail: Requestrr.WebApi.RequestrrBot.ChatBot[0]
      Error while searching movies by TmDb Id 10796 : Not found: 404
      DSharpPlus.Exceptions.NotFoundException: Not found: 404
         at DSharpPlus.Net.DiscordApiClient.CreateInteractionResponseAsync(UInt64 interaction_id, String interaction_token, InteractionResponseType type, DiscordInteractionResponseBuilder builder)
         at Requestrr.WebApi.RequestrrBot.SlashCommands.RequestGroup.ByTmdb99999(InteractionContext ctx, Int64 theMovieDbId)

@thomst08
Copy link
Owner

thomst08 commented Jul 2, 2024

These are good points, from my testing, I've found it seems to be a fail in D# when connecting to Discord. The issue is I could find a way of knowing when an error occurred and catching it to restart the setup process again.
What your describing here does add up.

Currently the bot will watch the settings for any changes and then resets all the commands, it also looks out for new servers and ones dropping off, this will also trigger a reset. Commands are also build dynamically into a temp file and used to connect to each server, if the command file fails to build, it will restart the process.
There's a delay in-between the checks of about 60 seconds, but if there is a number of servers and maybe a number of commands, this could be hitting the rate limit in D# and Discord, that, or Discord fails receiving the commands.

I was waiting for D# v5 to come out (and for the documentation to be updated, looks like there is a migration guide now) as it seems to have a few new things I can use and some changes to the rate limits and dynamic command building. What I might do is turn my attention back to this issue and see if I can implement the nightly builds into a version of Requestrr, I could put it into a docker container for anyone to test and see if that helps resolve the issue.

Would anyone be OK to test such a build?

@edrock200
Copy link

These are good points, from my testing, I've found it seems to be a fail in D# when connecting to Discord. The issue is I could find a way of knowing when an error occurred and catching it to restart the setup process again. What your describing here does add up.

Currently the bot will watch the settings for any changes and then resets all the commands, it also looks out for new servers and ones dropping off, this will also trigger a reset. Commands are also build dynamically into a temp file and used to connect to each server, if the command file fails to build, it will restart the process. There's a delay in-between the checks of about 60 seconds, but if there is a number of servers and maybe a number of commands, this could be hitting the rate limit in D# and Discord, that, or Discord fails receiving the commands.

I was waiting for D# v5 to come out (and for the documentation to be updated, looks like there is a migration guide now) as it seems to have a few new things I can use and some changes to the rate limits and dynamic command building. What I might do is turn my attention back to this issue and see if I can implement the nightly builds into a version of Requestrr, I could put it into a docker container for anyone to test and see if that helps resolve the issue.

Would anyone be OK to test such a build?

I'd be happy to!

@thomst08
Copy link
Owner

thomst08 commented Jul 7, 2024

Hey guys, I've looked over DSharps latest nightly build and have got it running, however, it is bit unstable at the moment...
Basically, the bot cannot restart on a setting change and fails completely, the only way forward with that is to ether change the handling of a restart and/or wait for a more later build.
As it's still in development I will leave it alone for the time being and wait for a full release or a newer build to dev on, as I will need to rebuild the slash commands in their new library (that will require a bit more work).
Testing however did give me some insight on some other areas, I still have yet to complete the Lidarr integration and its nearing its completion.

My intent is to come back to this bug and find a solution, I'm sorry this issue is taking a while. I will comment here when a build is ready to test, if not fix the problem.

@andrewmcd7
Copy link
Author

Has there been any progress on this? I’m getting it failing super often now.

[2024-09-16 14:10:22 +03:00] [0 / ] [Crit ] There was an error registering application commands
System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.List1.set_Capacity(Int32 value) at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<<RegisterCommands>b__0>d.MoveNext() [2024-09-16 14:10:22 +03:00] [0 / ] [Crit ] There was an error registering application commands System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray') at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable) at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection)
at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<b__0>d.MoveNext()
[2024-09-16 14:10:22 +03:00] [0 / ] [Crit ] There was an error registering application commands
System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.List1.set_Capacity(Int32 value) at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<<RegisterCommands>b__0>d.MoveNext() [2024-09-16 14:10:22 +03:00] [0 / ] [Crit ] There was an error registering application commands System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray') at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable) at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection)
at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<b__0>d.MoveNext()
warn: Requestrr.WebApi.RequestrrBot.ChatBot[0]
Bot has been restarted.

@thomst08
Copy link
Owner

Hey @andrewmcd7,

Partly, the latest version has an updated version of Dsharp+, I haven't looked much more into this issue as I wanted to complete lidarr support first, but I will begin researching this issue again.

Has this happened since updating to the new version?

@andrewmcd7
Copy link
Author

andrewmcd7 commented Sep 16, 2024 via email

@andrewmcd7
Copy link
Author

andrewmcd7 commented Sep 16, 2024 via email

@thomst08
Copy link
Owner

@andrewmcd7,

I think I have something!
I did a bit more digging and found some errors matching yours, it seems when you have a number of servers with commands, DSharp+ has issues dealing with this kind of setup if you assign the same commands to all the guilds, the answer is to set them up as global commands.

So.... knowing this now and having a look, in its current form, Requestrr assigns the same commands to all servers with each of their guild numbers.... So I quickly setup a version that only assigns the commands as global commands, the draw back is you can only have 100 of them. So as long as you don't have over 98~ categories, you should be ok....

I have setup a docker container for you to try this new version.
thomst08/requestrr:dev
Give this container a try and let me know if that solves the issue then we can go from there.
Anyone else who is having this issue is also welcome to try too.

@kevle1
Copy link

kevle1 commented Sep 17, 2024

I was also getting the following exception, and yep I did have the bot in multiple servers -

[Crit ] There was an error registering application commands
System.ArgumentException: Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')
   at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
   at System.Collections.Generic.List`1.set_Capacity(Int32 value)
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at DSharpPlus.SlashCommands.SlashCommandsExtension.<>c__DisplayClass39_0.<<RegisterCommands>b__0>d.MoveNext()

Trying out the dev tag container it looks good, commands are registered again. Cheers!

@edrock200
Copy link

@andrewmcd7,

I think I have something! I did a bit more digging and found some errors matching yours, it seems when you have a number of servers with commands, DSharp+ has issues dealing with this kind of setup if you assign the same commands to all the guilds, the answer is to set them up as global commands.

So.... knowing this now and having a look, in its current form, Requestrr assigns the same commands to all servers with each of their guild numbers.... So I quickly setup a version that only assigns the commands as global commands, the draw back is you can only have 100 of them. So as long as you don't have over 98~ categories, you should be ok....

I have setup a docker container for you to try this new version. thomst08/requestrr:dev Give this container a try and let me know if that solves the issue then we can go from there. Anyone else who is having this issue is also welcome to try too.

You sir, are a genius! Thank you!

@thomst08
Copy link
Owner

Hey @andrewmcd7,
Just checking in to see if you had a chance to test the dev container, just want some feedback to see if that resolves the issue.
If it does, then I will start to find how we can fix this issue going forward and if the next version of DSharp will have this issue.

Now for the bad news with the fix, after setting it up and thinking about it for a while, this is setting a global list of commands for the bot, the issue for that is it will now mean each command is usable with a private message directly to the bot, that wouldn't be too bad, but it also means permissions are ignored in this case, so this isn't the best fix going forward, just so anyone using the dev container is aware of this exploit.

If this does fix the bot form crashing and its working fine, then I will do the following.

  1. Talk with DSharp support and find out how/what we can do to solve this issue and if global commands are the only way.
  2. Depending on the result of that, ether make the bot auto change or have a setting to toggle global vs private servers, and to block direct communications if not desired.
  3. Keep you guys updated on the progress and deploy the fix when ready.

But first, I need to know if this 100% fixed the issue, still has the issue or somewhere in between.

@andrewmcd7
Copy link
Author

andrewmcd7 commented Sep 18, 2024 via email

@thomst08
Copy link
Owner

@andrewmcd7,

Ok then, I will keep looking, just double checking, do you have "Enable requesting via a private message" enabled?

@andrewmcd7
Copy link
Author

andrewmcd7 commented Sep 18, 2024 via email

@edrock200
Copy link

@thomst08 I think something else might be going on here. I believe your fix did resolve the initial error. I think one of our hosts is having some sort of network issue though, so we are moving the bots to another host to test. Please don't spend time on this for now until we can see if it happens on the new host. Apologies for the inconvenience.

@thomst08
Copy link
Owner

Hey @edrock200,

That's all good, thanks for the feedback.
I've reached out to DSharpPlus support to see what more they suggest anyway, if something comes up I'll post it back here.
Hope you get the network issues resolved.

@edrock200
Copy link

So far, since moving, things seem good. The host we moved to was getting the errors before the dev version too, but not after. The host still affected I think is being plagued by a docker bug.in which random containers keep dropping packets on their vnics, which is fixed in a a docker update not available to bionic. So I need to rebuild it to jammy.

@thomst08
Copy link
Owner

@edrock200, interesting, thanks for the feedback :)

@edrock200
Copy link

@edrock200, interesting, thanks for the feedback :)

It's been a few days since moving and no errors. Also, restarting the host doesn't cause the errors mentioned in the initial post since moving to the dev version, so I think you got it. :)

@Znuff
Copy link

Znuff commented Sep 23, 2024

I can confirm that at least on initial start, the :dev build seems to work fine.

@thomst08
Copy link
Owner

I can confirm that at least on initial start, the :dev build seems to work fine.

Nice, thanks for the feedback, let me know if this changers

@thomst08
Copy link
Owner

Hey guys, I've reached to to DSharps support but still waiting for a response... So I am a little bit stuck in which direction we go from here atm.
@andrewmcd7 did the dev version help in anyway or the problem is still persisting after giving it some more time? I'm not sure if you moved back to the latest version or stayed with dev after giving it a try?

@edrock200
Copy link

Hey guys, I've reached to to DSharps support but still waiting for a response... So I am a little bit stuck in which direction we go from here atm. @andrewmcd7 did the dev version help in anyway or the problem is still persisting after giving it some more time? I'm not sure if you moved back to the latest version or stayed with dev after giving it a try?

The dev version appears to have resolved (I am working with @andrewmcd7 as well.) thank you!

@thomst08
Copy link
Owner

Oh, excellent!
Glad to hear that :)

On another note, I reached out to DSharp support again on my original post, after commenting on this issue a dev got back to me and confirmed what we have discovered, the commands are best setup as global commands, when not setup this way and setup for a number of guilds/servers, it can cause issues like we saw (due to a few things it seems like rate limiting).
So...I will have to work out the best way to deal with this going forward and update Requestrr to use global commands instead as default, but block and restrict access when permissions are given and/or when direct messaging is not desired.

Once I have something more here, I will update the dev branch and we can go from there.
I will keep you guys posted.

@rascal2018
Copy link

Also encountering this issue. I will test with the dev branch and report back.

@Spillebulle
Copy link

I am having this same issue I believe on the latest dev build. For me it's been going on for a few weeks.

Someone messages me that the bot is down, I restart the container, and it's back up for a few more days before going down again

A request went through without issue on the 10th at 16:05 then not working the 11th 23:20 (might need to add 1 hr for timezone)

Requestrr_logs.txt

@thomst08
Copy link
Owner

Hey @Spillebulle,
I'll have to have a look into this, it just seems that your bot is losing connection to Discord, are you running it through a VPN by any chance? I've seen something like this happens when a VPN is blocking or losing connection.
It seems like this might be not 100% related to this issue.

@Spillebulle
Copy link

@thomst08 Not running through a VPN. My network had no downtime during that time according to my other services

@thomst08
Copy link
Owner

Hmmm, ok, I'll have to have a think and toy then, I am working on slowly changing to DSharpPlus v5, I have noticed it has some other info when it comes to feedback, when I have it in a state that's manageable, I plan to push it to the dev branch to test, I'll post here when that's done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

9 participants