User directory missing local users and contains remote users with federation disabled

[NiTRoeSE]

Description

Hi, our synapse homeserver is private and did not federate with matrix.org.

I wonder about that clients can't find local users from our homeserver.
So i took a look in database select * from user_directory_search;.
Surprisingly in this table there are a few users from matrix.org but no user from our own homeserver.
I don't know why this is the case, but i think it's strange. ?

The Homeserver configuration for user_directory is:

user_directory:
    enabled: true
    search_all_users: true
    prefer_local_users: true

Steps to reproduce

I have no clue how to reproduce this.

Homeserver

private homeserver

Synapse Version

Synapse 1.91.2

Installation Method

Docker (matrixdotorg/synapse)

Database

Postgres SQL , not migrated from SQLite , not restored any backup.

Workers

Single process

Platform

docker-compose stack with following containers:

• synapse
• psql_synapse
• redis_synapse

Configuration

No response

Relevant log output

did not find any relevant in logs

Anything else that would be useful to know?

Is there any way to rebuild the user_directory ?

I tried to run UPDATE user_directory_stream_pos SET stream_id = NULL;
Followed by a restart of synapse.
This didn't work for me.

So i Tried also:

curl --header "Authorization: Bearer TOKEN" -X POST 'http://10.1.1.20:8008/_synapse/admin/v1/background_updates/start_job' -d '{"job_name": "regenerate_directory"}'

But this also didn't work, user_directory remains the same.

I hope someone can clarify or help, i don't know if its a bug.

[DMRobertson]

Is there any way to rebuild the user_directory ?

I tried to run UPDATE user_directory_stream_pos SET stream_id = NULL;

You should NOT manually make edits to the Synapse database at all. Doing so without knowing what you're doing is highly unsafe and we cannot support anyone who has done so without our guidance.

So i Tried also:

curl --header "Authorization: Bearer TOKEN" -X POST 'http://10.1.1.20:8008/_synapse/admin/v1/background_updates/start_job' -d '{"job_name": "regenerate_directory"}'

But this also didn't work, user_directory remains the same.

Can you sanity check this succeeded by searching the logs for regenerate_directory?

Hi, our synapse homeserver is private and did not federate with matrix.org.

Are you sure about this? What config have you set up that enforces this? What do

• SELECT count(*) FROM destination_rooms WHERE destination = 'matrix.org'; and
• SELECT * FROM destinations WHERE destination = 'matrix.org';

return?

I wonder about that clients can't find local users from our homeserver.

That does seem odd and probably needs further investigation.

[NiTRoeSE]

@DMRobertson

Hi, thanks for investigate in this.

You should NOT manually make edits to the Synapse database at all. Doing so without knowing what you're doing is highly unsafe and we cannot support anyone who has done so without our guidance.

Shure, thats right. I found this in another issue and wanted to do all i can before i open a new issue.

Are you sure about this? What config have you set up that enforces this? What do

• SELECT count(*) FROM destination_rooms WHERE destination = 'matrix.org'; and
• SELECT * FROM destinations WHERE destination = 'matrix.org';

return?

Can you sanity check this succeeded by searching the logs for regenerate_directory?

If i run this command curl --header "Authorization: Bearer TOKEN" -X POST 'http://10.1.1.20:8008/_synapse/admin/v1/background_updates/start_job' -d '{"job_name": "regenerate_directory"}'
The only log output i get is:

synapse-server-mz         | 2023-09-07 12:03:40,164 - synapse.storage.background_updates - 512 - INFO - background_updates-1 - Not starting on bg update populate_user_directory_cleanup until populate_user_directory_process_users is done
synapse-server-mz         | 2023-09-07 12:03:41,165 - synapse.storage.background_updates - 544 - INFO - background_updates-1 - Starting update batch on background update 'populate_user_directory_createtables'
synapse-server-mz         | 2023-09-07 12:03:41,193 - synapse.storage.background_updates - 587 - INFO - background_updates-1 - Running background update 'populate_user_directory_createtables'. Processed 1 items in 26ms. (total_rate=0.038461538461538464/ms, current_rate=0.038461538461538464/ms, total_updated=1, batch_size=100)
synapse-server-mz         | 2023-09-07 12:03:41,194 - synapse.storage.background_updates - 512 - INFO - background_updates-1 - Not starting on bg update populate_user_directory_cleanup until populate_user_directory_process_users is done
synapse-server-mz         | 2023-09-07 12:03:42,195 - synapse.storage.background_updates - 544 - INFO - background_updates-1 - Starting update batch on background update 'populate_user_directory_process_rooms'
synapse-server-mz         | 2023-09-07 12:03:42,356 - synapse.storage.background_updates - 587 - INFO - background_updates-1 - Running background update 'populate_user_directory_process_rooms'. Processed 109 items in 160ms. (total_rate=0.68125/ms, current_rate=0.68125/ms, total_updated=109, batch_size=100)
synapse-server-mz         | 2023-09-07 12:03:43,357 - synapse.storage.background_updates - 544 - INFO - background_updates-1 - Starting update batch on background update 'populate_user_directory_process_rooms'
synapse-server-mz         | 2023-09-07 12:03:43,457 - synapse.storage.background_updates - 587 - INFO - background_updates-1 - Running background update 'populate_user_directory_process_rooms'. Processed 73 items in 99ms. (total_rate=0.7027027027027027/ms, current_rate=0.7041152263374485/ms, total_updated=182, batch_size=68)
synapse-server-mz         | 2023-09-07 12:03:44,458 - synapse.storage.background_updates - 544 - INFO - background_updates-1 - Starting update batch on background update 'populate_user_directory_process_rooms'
synapse-server-mz         | 2023-09-07 12:03:44,563 - synapse.storage.background_updates - 587 - INFO - background_updates-1 - Running background update 'populate_user_directory_process_rooms'. Processed 72 items in 103ms. (total_rate=0.7016574585635359/ms, current_rate=0.7024867889337892/ms, total_updated=254, batch_size=70)
synapse-server-mz         | 2023-09-07 12:03:45,564 - synapse.storage.background_updates - 544 - INFO - background_updates-1 - Starting update batch on background update 'populate_user_directory_process_rooms'
synapse-server-mz         | 2023-09-07 12:03:45,667 - synapse.storage.background_updates - 587 - INFO - background_updates-1 - Running background update 'populate_user_directory_process_rooms'. Processed 72 items in 100ms. (total_rate=0.7056277056277056/ms, current_rate=0.7069827741123919/ms, total_updated=326, batch_size=70)
synapse-server-mz         | 2023-09-07 12:03:46,637 - synapse.metrics._gc - 120 - INFO - sentinel - Collecting gc 1
synapse-server-mz         | 2023-09-07 12:03:46,668 - synapse.storage.background_updates - 544 - INFO - background_updates-1 - Starting update batch on background update 'populate_user_directory_process_rooms'
synapse-server-mz         | 2023-09-07 12:03:46,694 - synapse.storage.background_updates - 587 - INFO - background_updates-1 - Running background update 'populate_user_directory_process_rooms'. Processed 16 items in 25ms. (total_rate=0.702258726899384/ms, current_rate=0.7025241162264995/ms, total_updated=342, batch_size=70)
synapse-server-mz         | 2023-09-07 12:03:47,695 - synapse.storage.background_updates - 544 - INFO - background_updates-1 - Starting update batch on background update 'populate_user_directory_process_rooms'
synapse-server-mz         | 2023-09-07 12:03:47,700 - synapse.storage.background_updates - 587 - INFO - background_updates-1 - Running background update 'populate_user_directory_process_rooms'. Processed 1 items in 3ms. (total_rate=0.7/ms, current_rate=0.6992762872951765/ms, total_updated=343, batch_size=70)
synapse-server-mz         | 2023-09-07 12:03:47,702 - synapse.storage.background_updates - 512 - INFO - background_updates-1 - Not starting on bg update populate_user_directory_cleanup until populate_user_directory_process_users is done
synapse-server-mz         | 2023-09-07 12:03:48,704 - synapse.storage.background_updates - 544 - INFO - background_updates-1 - Starting update batch on background update 'populate_user_directory_process_users'
synapse-server-mz         | 2023-09-07 12:03:48,709 - synapse.storage.background_updates - 587 - INFO - background_updates-1 - Running background update 'populate_user_directory_process_users'. Processed 16 items in 4ms. (total_rate=4.0/ms, current_rate=4.0/ms, total_updated=16, batch_size=100)
synapse-server-mz         | 2023-09-07 12:03:49,709 - synapse.storage.background_updates - 544 - INFO - background_updates-1 - Starting update batch on background update 'populate_user_directory_process_users'
synapse-server-mz         | 2023-09-07 12:03:49,713 - synapse.storage.background_updates - 587 - INFO - background_updates-1 - Running background update 'populate_user_directory_process_users'. Processed 1 items in 2ms. (total_rate=2.8333333333333335/ms, current_rate=2.75/ms, total_updated=17, batch_size=400)
synapse-server-mz         | 2023-09-07 12:03:50,715 - synapse.storage.background_updates - 544 - INFO - background_updates-1 - Starting update batch on background update 'populate_user_directory_cleanup'
synapse-server-mz         | 2023-09-07 12:03:50,727 - synapse.storage.background_updates - 587 - INFO - background_updates-1 - Running background update 'populate_user_directory_cleanup'. Processed 1 items in 10ms. (total_rate=0.1/ms, current_rate=0.1/ms, total_updated=1, batch_size=100)
synapse-server-mz         | 2023-09-07 12:03:50,729 - synapse.storage.background_updates - 418 - INFO - background_updates-1 - No more background updates to do. Unscheduling background update task.

Are you sure about this? What config have you set up that enforces this?

We use the whitelist option for federation in homeserver.yml :
This option points to a "prevent.federation.de" fake fqdn.

federation_domain_whitelist:
  - block.federation.de

I hope that helps, thanks in advanced !

Additional information:

As far i can remember the time we setup the matrix server.
I think in the beginning and testing while setup we had a really short period of time enabled federation, maybe the entries in user_directory_search came from this.
But that does not explain why the user_directory_search don't gets updated / refreshed.

[DMRobertson]

That table should definitely get truncated:

synapse/synapse/storage/databases/main/user_directory.py

Line 826 in dac9764

txn.execute(f"{truncate} user_directory_search")

One thing you could do is to turn on DEBUG logging for SQL (see here) and rerun the regenerate_directory job. That should show you want SQL is getting executed and with which values. Note that this will make the logger output PII like user IDs to the database; don't paste it here without carefully redacting sensitive information first.

[NiTRoeSE]

That table should definitely get truncated:

synapse/synapse/storage/databases/main/user_directory.py

Line 826 in dac9764

txn.execute(f"{truncate} user_directory_search")

One thing you could do is to turn on DEBUG logging for SQL (see here) and rerun the regenerate_directory job. That should show you want SQL is getting executed and with which values. Note that this will make the logger output PII like user IDs to the database; don't paste it here without carefully redacting sensitive information first.

Ok, thanks i have the logs ...
Is it ok to send you the logs direct ? I can of course change the user ids , but I'm not sure if there is more than this sensitive information ?

[DMRobertson]

Is it ok to send you the logs direct ?

Yes, please email me via davidr@element.io or matrix @dmrobertson:matrix.org.

[DMRobertson]

Unfortunately those logs didn't have all the info I was hoping for (c.f. #16281).

My guess is that the Matrix.org users are showing up in your directory due to

I think in the beginning and testing while setup we had a really short period of time enabled federation, maybe the entries in user_directory_search came from this.

I don't fully understand why your own users aren't showing up in the directory. Is this something you can observe as an end-user searching for users in the client?

To confirm/investigate this, some more questions:

-Do you have any application services configured? If so, what users are the responsible for? (Application service users do not show up in the user directory, which may explain why you're not seeing your local users).

• Can you run the following queries and email me the results?
  • SELECT name, user_type, deactivated FROM users;
  • SELECT * FROM user_directory;
  • SELECT * FROM user_directory_search;
  • SELECT * FROM room_stats_current WHERE joined_members != local_users_in_room;

[clokep]

I think in the beginning and testing while setup we had a really short period of time enabled federation, maybe the entries in user_directory_search came from this.

It could be worth purging any rooms that were from matrix.org, maybe?

[DMRobertson]

Thank you for the logs. We saw that most of the users are marked as support users.

At the beginning i thought it was a good idea to flag users with a type ..
[...]
Maybe i have not seen the hints to this in documentation or it is not written down there that flagging a user with a type prevents them to be shown in search results.

I don't think we strongly advertise anywhere what changing the user_type means. The only place I can see where we even mention this is possible is in the admin API docs.

Is there a good way to remove the user type from the users ?

I suggest running

UPDATE users
SET user_type = NULL
WHERE user_type = 'support'

then trying to regenerate the user directory. That should make your local users appear.

Additionally, there are three rooms according to the query on room_stats_current with non-local members. As Patrick says, I suggest using the delete room admin API to purge those rooms from your homeserver, then rebuild the user directory again.

[NiTRoeSE]

UPDATE users
SET user_type = NULL
WHERE user_type = 'support'

Hello,

thanks a lot for troubleshooting.
All recommended steps to fix this are working.
I set the user_type for every user to NULL and deleted all rooms from matrix.org on our server.
After this i rebuild the user_directory_search with the background job.
Now all is fine, i can find all users and the matrix users are gone.
Databases tables looks good.

Thanks again, have a nice day!

/closed

[DMRobertson]

Thanks for confirming; glad to help.