Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

History purge failed with: ValueError: max() arg is an empty sequence #10669

Closed
Kokokokoka opened this issue Aug 21, 2021 · 6 comments
Closed

History purge failed with: ValueError: max() arg is an empty sequence #10669

Kokokokoka opened this issue Aug 21, 2021 · 6 comments
Labels
good first issue Good for newcomers S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.

Comments

@Kokokokoka
Copy link
Contributor

Description

On history purge, this happened:

2021-08-20 21:25:22,304 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626834204073 => stream_or
dering 6232155)
2021-08-20 23:17:30,415 - synapse.handlers.pagination - 286 - ERROR - _purge_history-5673 - [purge] failed
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/pagination.py", line 279, in _purge_history
    await self.storage.purge_events.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/purge_events.py", line 54, in purge_history
    state_groups = await self.stores.main.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 49, in purge_history
    return await self.db_pool.runInteraction(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 686, in runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 791, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
    raise exception.with_traceback(traceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 786, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 554, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 105, in _purge_history_txn
    max_depth = max(row[1] for row in rows)
ValueError: max() arg is an empty sequence
2021-08-20 23:17:30,443 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626841050432 => stream_or
dering 6232211)
2021-08-20 23:17:30,453 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626841050443 => stream_or
dering 6232211)
2021-08-20 23:19:03,094 - synapse.handlers.pagination - 286 - ERROR - _purge_history-5691 - [purge] failed
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/pagination.py", line 279, in _purge_history
    await self.storage.purge_events.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/purge_events.py", line 54, in purge_history
    state_groups = await self.stores.main.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 49, in purge_history
    return await self.db_pool.runInteraction(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 686, in runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 791, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
    raise exception.with_traceback(traceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 786, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 554, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 105, in _purge_history_txn
    max_depth = max(row[1] for row in rows)
ValueError: max() arg is an empty sequence
2021-08-20 23:19:15,824 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626841155788 => stream_or
dering 6232211)

Then there's no connection with element (crash?)

Version information

Synapse: 1.40,
on Debian 11,
using matrix repo.
This is a migration from 1.0 version
@erikjohnston erikjohnston added good first issue Good for newcomers S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. labels Aug 23, 2021
@erikjohnston
Copy link
Member

I think the fix here is to handle the case of the DB query returning no rows.

@Kokokokoka Kokokokoka mentioned this issue Aug 25, 2021
Merged
@anoadragon453
Copy link
Member

What caused the purge out of interest? Do you have the experimental room retention feature enabled, or was the room purged manually?

@Kokokokoka
Copy link
Contributor Author

I answered in a PR reply, sorry, I'll copy it here:
I would love to find out the cause of this issue. The problem is, that we migrated from 1.0->1.41. Rooms, messages, files got cleared by some mystique scripts before (maybe this is the cause of the bug, i'm trying to find this out in the db).
Now there is message retention policy, it helps a lot. I don't know if there's a policy for empty rooms/rooms with only deleted accounts which weren't accessed for x. I had to manually implement local files|thumbnails cleaner (which checks what elems have last access ts > x, and then purge the elems which aren't in the db.

For instance, how could you have a room graph that doesn't have any leaves?

Can I see this in the db somehow?
(For now I did:
empty rooms:
select count(*) from rooms where not exists(select from room_memberships where room_memberships.room_id = rooms.room_id); to find empty rooms (26)
and:
rooms that become empty if deactivated|ppl with activity > x ago are substracted:


select count(*) from rooms where room_id not in(select room_id from room_memberships where
user_id not in (select name from users where name in (select user_id from (select  
user_id, max(timestamp) ts from user_daily_visits group by user_id) x where ts < 
1627215050807) or deactivated = 1));
count
-------
 1421
(1 row))

rooms with 1 active:

select count(*) from (select room_id, count(user_id) cn from room_memberships where 
user_id not in (select name from users where name in (select user_id from (select  
user_id, max(timestamp) ts from user_daily_visits group by user_id) x where ts < 
1627215050807) or deactivated = 1) group by room_id)x where cn =1;
count
796
(1 row)

So, I don't really know whether this room was purged manually, all I know that there was some script that managed local_media+should have managed messages and we had syn v1.0

@richvdh
Copy link
Member

richvdh commented Sep 23, 2021

There are a few known reasons why we could end up with a room without any forward-extremities: see #8094 which is another symptom of such a situation. Some examples include:

  • There was a background update introduced in Synapse 1.10 (Backgroud update to clean out rooms from current state #6802) which removes the forward-extremities for any room where there are no active members.
  • Ill-advised database surgery, including running various scripts that can be found on the internet.

richvdh pushed a commit that referenced this issue Sep 24, 2021
@Kokokokoka
In _purge_history_txn, ensure that txn.fetchall has elements before…
e704cc2 
… accessing rows (#10690)

This change adds a check for row existence before accessing row element, this should fix issue #10669
Signed-off-by: Vasya Boytsov vasiliy.boytsov@phystech.edu
@callahad
Copy link
Contributor

callahad commented Oct 5, 2021

Now that #10690 is merged, can we close this?

@Kokokokoka
Copy link
Contributor Author

I think so, if there's no objections, i'll gladly do this.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
good first issue Good for newcomers S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@callahad @anoadragon453 @richvdh @erikjohnston @Kokokokoka and others