Backfill remote event fetched by MSC3030 so we can paginate from it later

[MadLittleMods]

Backfill remote event fetched by MSC3030 /timestamp_to_event so we can paginate from it later

• Depends on Update get_pdu to return the original, pristine EventBase #13320

Complement tests: matrix-org/complement#406

We could use the same method to backfill for /context as well in the future, see #3848

Dev notes

COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestJumpToDateEndpoint

COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestJumpToDateEndpoint/parallel/federation/can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint

docker build -t matrixdotorg/synapse -f docker/Dockerfile .

Random snippets

For asdf_get_debug_events_in_room_ordered_by_depth to dump the events from the database in a room, see the dev notes in #12370. And poetry add tabulate

For a version without dependencies, see the dev notes for asdf_get_debug_events_in_room_ordered_by_depth in matrix-org/matrix-react-sdk#8354

---

logger.info("backfill_event event=%s", pformat(vars(event)))
logger.info("backfill_event eventAsdf=%s", pformat(vars(eventAsdf)))

def get_debug_internal_metadata(ev: EventBase) -> dict[str, str]:
    dict = ev.internal_metadata.get_dict()
    return {
        "outlier": ev.internal_metadata.outlier,
        "stream_ordering": ev.internal_metadata.stream_ordering,
        "out_of_band_membership": dict.get("out_of_band_membership"),
        "send_on_behalf_of": dict.get("send_on_behalf_of"),
        "recheck_redaction": dict.get("recheck_redaction"),
        "soft_failed": dict.get("soft_failed"),
        "proactively_send": dict.get("proactively_send"),
        "redacted": dict.get("redacted"),
        "txn_id": dict.get("txn_id"),
        "token_id": dict.get("token_id"),
        "historical": dict.get("historical"),
    }

logger.info(
    "backfill_event event.internal_metadata=%s",
    pformat(get_debug_internal_metadata(event)),
)
logger.info(
    "backfill_event eventAsdf.internal_metadata=%s",
    pformat(get_debug_internal_metadata(eventAsdf)),
)

via https://stackoverflow.com/a/54653137/796832

def exception_with_traceback(message):
    import sys
    import types

    tb = None
    depth = 0
    while True:
        try:
            frame = sys._getframe(depth)
            depth += 1
        except ValueError as exc:
            break

        tb = types.TracebackType(tb, frame, frame.f_lasti, frame.f_lineno)

    return Exception(message).with_traceback(tb)

asdf_exc = exception_with_traceback("asdf")
logger.error(
    "_update_outliers_txn event_id=%s outlier_persisted=%s",
    event.event_id,
    outlier_persisted,
    exc_info=(type(asdf_exc), asdf_exc, asdf_exc.__traceback__),
)

import traceback

logger.info(
    "persist_events _event_persist_queue event_id=%s backfilled=%s stack=%s",
    event.event_id,
    backfilled,
    traceback.format_stack(),
)

Call stacks

persist_events or persist_event
_EventPeristenceQueue ->
_persist_event_batch
_persist_events_and_state_updates
_persist_events_txn
_update_outliers_txn ("Updating state for ex-outlier event")

_get_event_cache

_invalidate_get_event_cache

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Pull request includes a sign off
• Code style is correct
  (run the linters)

[richvdh]

I'm starting to find this rather hard to follow. Could you separate the updates to get_pdu and the assertions from the MSC3030-related changes?

[richvdh]

This is much clearer now that #13320 is factored out, thanks!

lgtm other than the below

[MadLittleMods]

@richvdh Thanks for the review and prompt to split things up! 🦏