Lots of Olm sessions for the same device

[richvdh]

For some reason Element is repeatedly creating new sessions for another of my devices.

So far it is up to 73 of them.

[weeman1337]

@richvdh how bad is this? Can you provide some snipped to get the session count for my devices? Just to test with some of my accounts, so that I can triage this issue better.

[richvdh]

I ran this in a browser console to dump the devices with a large number of sessions:

async function sessionsPerDevice() {
    const db = await new Promise((resolve) => {
        const req = indexedDB.open('matrix-js-sdk:crypto');
        req.onsuccess = () => resolve(req.result);
    });

    // fetch all the known Olm sessions
    const rows = await new Promise((resolve) => {
        const tx1 = db.transaction(["sessions"], "readonly");
        const idx = tx1.objectStore("sessions").index("deviceKey");
        const req = idx.getAll();
        req.onsuccess = () => resolve(req.result);
    });

    // collect the sessions by device ID
    const deviceMap = new Map();
    for (const row of rows) {
        const {deviceKey} = row;
        const m = deviceMap.get(deviceKey) ?? 0;
        deviceMap.set(deviceKey, m+1);
    }

    // sort by descending number of sessions
    const sorted = Array.from(deviceMap.entries());
    sorted.sort(([,a], [,b]) => b-a);

    return sorted;
}

const deviceList = mxMatrixClientPeg.get().crypto.deviceList;
for (const [key, count] of (await sessionsPerDevice()).slice(0, 5)) {
    console.log(`${key} [${deviceList.getUserByIdentityKey("m.megolm.v1.aes-sha2", key)}]: ${count}`);
}

[richvdh]

Having looked at the rageshakes, what happened is a double failure:

• EA device sends lots (about 70?) of encrypted to-device messages. [This actually happened twice.]
• Those to-device messages get reordered before decryption at EW. We only keep 40 message keys, so end up dropping a load of keys.
• Once the messages that we have dropped keys for turn up, we decide that the olm session is wedged, and force a new olm session.
• New-olm-session-creation is supposed to be rate-limited, but it is racy, so if lots of requests for new olm sessions happen at once, the rate limit doesn't work.

So, the problems here are:

1. To-device-messages are being reordered. They shouldn't be reordered at all, though Element tries to process them in parallel, which may be part of the problem.
2. New-olm-session-creation-rate-limiting is racy.

[richvdh]

Have opened #25723 to track the reordering. Suggest we use this issue to track "New-olm-session-creation-rate-limiting is racy."