Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not relaying Matrix->IRC after performance issue #520

Closed
mrboatsman opened this issue Nov 17, 2017 · 8 comments
Closed

Not relaying Matrix->IRC after performance issue #520

mrboatsman opened this issue Nov 17, 2017 · 8 comments
Labels
X-External-Issue Something upstream of the bridge (library, homeserver, ircd) has caused an issue.

Comments

@mrboatsman
Copy link

mrboatsman commented Nov 17, 2017 •
edited
Loading

#I had some performance issues due to low memory on the server and after that the appservice does not relay Matrix -> IRC, the other way around goes well. Restarted the Appservice without any success and then I can see issues like:
HTTP 400 Error: {"errcode":"M_USER_IN_USE","error":"User ID already taken."}.
But that seems to be an IRC-> Matrix issue so I can't understand why Matrix -> IRC relaying does not work.

It is like the appservice just ignoring everything which is being written on Matrix, can't see anything in the logs when i sends a message on Matrix group chat.

Error LOG

2017-11-17 02:11:59 ERROR:IrcBridge [-] POST https://example.com/_matrix/client/r0/register (AS) HTTP 504 Error: "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx/1.10.3 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n"

...


2017-11-17 07:46:15 ERROR:IrcBridge [-] POST https:/example.com/_matrix/client/r0/register (AS) HTTP 400 Error: {"errcode":"M_USER_IN_USE","error":"User ID already taken."}
2017-11-17 07:46:15 ERROR:IrcBridge [-] POST https:/example.com/_matrix/client/r0/register (AS) HTTP 400 Error: {"errcode":"M_USER_IN_USE","error":"User ID already taken."}
2017-11-17 07:46:15 ERROR:IrcBridge [-] POST https:/example.com/_matrix/client/r0/register (AS) HTTP 400 Error: {"errcode":"M_USER_IN_USE","error":"User ID already taken."}
2017-11-17 07:46:15 ERROR:IrcBridge [-] POST https:/example.com/_matrix/client/r0/register (AS) HTTP 400 Error: {"errcode":"M_USER_IN_USE","error":"User ID already taken."}
2017-11-17 07:46:15 ERROR:IrcBridge [-] POST https:/example.com/_matrix/client/r0/register (AS) HTTP 400 Error: {"errcode":"M_USER_IN_USE","error":"User ID already taken."}

Debug log

2017-11-17 07:46:15 INFO:req [l1nxi2e6utc0w] [I->M] No mapped matrix rooms for IRC channel #ircChannel with origin = [join,alias]
2017-11-17 07:46:15 INFO:IrcBridge [l1nxi2e6utc0w] [I->M] SUCCESS  (40ms)
2017-11-17 07:46:15 INFO:MemberListSyncer updateIrcMemberList: Leaving 0 users as they are not in #ircChannel.
2017-11-17 07:46:15 INFO:req [5ggswd2xl8cgo] [I->M] Joining room !vTW3ScEazaAzqmOhZz:example.com and setting presence to online
2017-11-17 07:46:15 INFO:IrcBridge [-] POST https://example.com/_matrix/client/r0/register (AS) Body: {"auth":{},"username":"ircNetwork_ircNick"}
2017-11-17 07:46:15 INFO:IrcBridge [-] PUT https://example.com/_matrix/client/r0/presence/%40ircNetwork_ircNick%3Aexample.com/status (@ircNetwork_ircNick:example.com) Body: {"presence":"online"}
2017-11-17 07:46:15 ERROR:IrcBridge [-] POST https://example.com/_matrix/client/r0/register (AS) HTTP 400 Error: {"errcode":"M_USER_IN_USE","error":"User ID already taken."}

STDOUT from matrix-appservice-irc

2017-11-17 07:56:15 INFO:req [s10z3tn1chcs] [I->M] Joining room !vTW3ScEazaAzqmOhZz:example.com and setting presence to online
2017-11-17 07:56:15 INFO:IrcBridge [-] POST https://example.com/_matrix/client/r0/register (AS) Body: {"auth":{},"username":"ircNetwork_ircNick"}
2017-11-17 07:56:15 INFO:IrcBridge [-] PUT https://example.com/_matrix/client/r0/presence/%40ircNetwork_ircNick%3Aexample.com/status (@ircNetwork_ircNick:example.com) Body: {"presence":"online"}
2017-11-17 07:56:15 ERROR:IrcBridge [-] POST https://example.com/_matrix/client/r0/register (AS) HTTP 400 Error: {"errcode":"M_USER_IN_USE","error":"User ID already taken."}
@mrboatsman
Copy link
Author

Found an issue which looks like the issue I have but with Discord, the result is the same that Matrix->(bridged network) does not work. And the same symptom with "User ID already taken.". Have no idea if it is the same root cause.
matrix-org/matrix-appservice-discord#37

@Mikaela Mikaela added the bug label Nov 17, 2017
@mrboatsman
Copy link
Author

This is quite interesting, I left the Synapse server and the app-service running just to be able to do some debugging later, it started to work exactly 7 days later without me touching it, since last restart. Everything I had posted in the Matrix server got relayed to IRC. This might give some clues where it might be problems.

@tidux
Copy link

tidux commented Jan 18, 2018

I'm seeing this bug on the latest git master Synapse and IRC bridge. This may be yet another case of Synapse being a bloated pig-whale and the Python GIL biting Matrix users.

@FrobtheBuilder
Copy link

I have this exact issue as well.

@TimePath
Copy link

I had this issue, dug into synapse and noticed that there's a queuing mechanism for sending events to appservices. In particular:

SELECT                                                          
  e.stream_ordering,
  e.event_id
FROM events AS e
WHERE
  (SELECT stream_ordering
   FROM appservice_stream_position)
  < e.stream_ordering
  AND e.stream_ordering <= $current_id
ORDER BY e.stream_ordering ASC
LIMIT 10

I then ran

SELECT                                                          
  e.stream_ordering,
  e.event_id
FROM events AS e
ORDER BY e.stream_ordering DESC
LIMIT 10

Took the highest stream_ordering and updated appservice_stream_position manually:

update appservice_stream_position set stream_ordering = $max_stream_ordering;

After a synapse restart (unsure if really needed), events started going through from M>I in minutes

@zyphlar
Copy link

zyphlar commented May 11, 2018

I was having this / similar problems, I ended up deleting the synapse database file and appservice database files. Painful because all the users/rooms/settings/names/IDs changed and had to be re-joined/set, but at least it works now. (The above queries produced nothing for me, that wasn't my problem I guess.)

@Half-Shot
Copy link
Contributor

I believe this is a result of matrix-org/synapse#1834, so there isn't much we can do as the AS.

@Half-Shot Half-Shot added X-External-Issue Something upstream of the bridge (library, homeserver, ircd) has caused an issue. and removed bug labels May 12, 2018
@Half-Shot
Copy link
Contributor

Closing as isn't an IRC specific issue

@snyk-bot snyk-bot mentioned this issue Jan 19, 2022
Open
@snyk-bot snyk-bot mentioned this issue Aug 29, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
X-External-Issue Something upstream of the bridge (library, homeserver, ircd) has caused an issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@zyphlar @Mikaela @mrboatsman @tidux @Half-Shot @FrobtheBuilder @TimePath