E2E Messaging

[Half-Shot]

This is the issue for looking at implementing https://matrix.org/git/olm in the SDK. It looks like the lib includes bindings for python so it should be a case of staring at the js-sdk's implementation and trying to fit that into our code.

This is likely going to be a long and tedious task so it might be months before this gets ticked off.

[pik]

Looking into this - probably olm should be wrapped into a distro supported package for this to be accessible as well?

[pik]

Outbound sessions for the Olm (m.olm.v1.curve25519-aes-sha2) double ratchet (not megaolm) are working via. https://github.com/pik/matrix-python-sdk/tree/e2e.

Here is a code snippet to get started if anyone wants to play around with it:

OLM_BASE_DIR = '/home/pik/git/olm'
MY_DEVICE_ID = 'MY_MATRIX_PYTHON_SDK_DEVICE'
MY_USER_ID = '@pik:pik-test'
MY_PASSWORD = '1234'
HOMIE_USER_ID = '@homie:pik-test'
import os
import sys

sys.path.append(os.path.join(OLM_BASE_DIR, 'python'))

from matrix_client import client, api, crypto

cli = client.MatrixClient('http://localhost:8008')
olm_device = crypto.OlmDevice.load_or_create_olm_device(cli.api, MY_USER_ID, MY_DEVICE_ID)
# Pass to the device_id to the login method otherwise Synapse will auto-generate
# a new one for us
cli.login_with_password(MY_USER_ID, MY_PASSWORD, device_id=MY_DEVICE_ID)
sessions = olm_device.create_outbound_sessions_to_user(HOMIE_USER_ID)
def get_first_session(d):
    return get_first_session(list(d.values())[0]) if isinstance(d, dict) else d
outbound_session = get_first_session(sessions)
# room should be a room with encryption enabled via. the m.room.encryption event
# to do this create a room and an m.room_encryption event e.g.
room = cli.create_room('homes_are_invited')
room.invite_user(HOMIE_USER_ID) # Accept Invite
cli.api.send_state_event(room.room_id, 'm.room.encryption',
    {'algorithm': 'm.olm.v1.curve25519-aes-sha2'}
)
olm_device.send_encrypted_message_to_session(room.room_id, outbound_session, 'Hello via. e2e on matrix-python-sdk')

[AlexDaniel]

News? :)

[rocket-pig]

If anyone comes here after me, and wants to play and not try to deduce what's necessary to do that:

get pik's branch:
wget wget https://github.com/pik/matrix-python-sdk/archive/e2e.zip
get OLM master:
wget https://git.matrix.org/git/olm/snapshot/olm-master.zip
unzip both
change OLM_BASE_DIR in above example to point to where you unzipped olm-master to.

I'm yet unwilling to learn the entire git lexicon, so I'm sure for those in deep git cover there's a different way. but, above works for the novitiate.

Edit
....I've spent half a day trying to get this to work, to no avail. This functionality is wanted! I and Im sure others would like to have bots in encrypted channels. Not quite junior-scout ready though.
Where it breaks is the last line, sending a message:
olm_device.send_encrypted_message_to_session(room.room_id, outbound_session, 'Hello via. e2e on matrix-python-sdk')
..returning
TypeError: _send_encrypted_message_to_sessions() takes 4 positional arguments but 5 were given
...no matter how i try to compose the request.

[gsantner]

any news on this since october? sadly cannot find any SDK that has working e2e yet, other than android/ios

[ara4n]

there's a GSOC project this year to finish this. Meanwhile, matrix-js-sdk is the reference for working e2e.

[gsantner]

wow, glad to hear that 👍

[Zil0]

Hi, I do the GSoC project :)
I will report here when I reach important milestones, meanwhile general development info can be found at https://github.com/Zil0/matrix-python-sdk/wiki

[gsantner]

thanks for responding here! Thanks too for working on it and good luck! ;)

[mariuszcyranowski]

Good luck !

[Zil0]

All the PRs for basic E2E support are now opened! Here are some instructions in order to try it out (and help track down eventual bugs).

Disclaimer

The implementation is mostly unreviewed, and shouldn't be trusted.
There are still missing features compared to the JS SDK, such as encrypted attachments, device verification and key sharing.

Installation

The Olm C library is needed. It can be installed from source, or with a package manager (Archlinux, Debian...).
Then, use the branch https://github.com/Zil0/matrix-python-sdk/tree/e2e_beta_1 (there's a new one, read below), where I'll only push fixes in the future. You will also have to get the optional E2E dependencies. For example, using pip (preferably inside a virtualenv): pip install -e 'git+https://github.com/Zil0/matrix-python-sdk@e2e_beta_1#egg=matrix-python-sdk-e2e[e2e]' --process-dependency-links

Using encryption

In order to enable encryption support, the only modification needed is to pass encryption=True when instantiating MatrixClient.
However, you should also keep track of the device id of the client, otherwise a new one will be generated by Synapse on every launch.

Sample code

Assuming we can join the existing room #test:localhost:

from matrix_client.client import MatrixClient

user_id = "@test:localhost"
password = "test"
device_id = "device_id"
room_id = "#test:localhost"

client = MatrixClient("http://localhost:8008", encryption=True)
client.login(username=user_id, password=password, device_id=device_id)
room = client.join_room(room_id)

if not room.encrypted:
    room.send_text("Unencrypted!")
    room.enable_encryption()
    room.send_text("Encrypted, if we had the required power level.")
client.logout()

client = MatrixClient("http://localhost:8008", encryption=True)
client.login(username=user_id, password=password, device_id=device_id)
room = client.join_room(room_id)
room.send_text("Reusing the same device ID should allow good performance and keep Riot "
               "from complaining.")
client.logout()

Finding and reporting bugs

Most bugs will include undecryptable sent or received messages. There are a lot of different possible causes, which might be hard to track down if they are race conditions.
When a bug is mostly reproducible, I welcome a report along with a log at least of INFO level, and explanations on how to trigger it. An issue with those info can be opened against my fork.

[warp1337]

Do you have any example of how to decrypt a message using the client module?

Nevermind :D I was on the wrong branch.

[Zil0]

The end of my GSoC has come. I'll stay around, get everything merged and fix bugs, but this is a good opportunity to post an update!

The new branch https://github.com/Zil0/matrix-python-sdk/tree/e2e_beta_2 is available for anyone wanting to test E2E.

The instructions are alike to those above, except there is a new feature that spares having to keep track of device ID. Simply do client = MatrixClient(host, encryption=True, restore_device_id=True) to enable it.

All the missing features are also out since the last update (encrypted attachments, device verification, key sharing, import/export of room keys). There might be some rough edges, let me know!

Last but not least, I've written up a file which explains more of what can be done with E2E. It is present at the root of the repo as E2E_overview.rst.

[warp1337]

Are there any news on sending encrypted files, e.g., sending an encrypted PDF?

[Zil0]

@warp1337 the missing bit was that the m.file event would not be sent encrypted. Just added a commit to allow this. Now you just need to use the code sample you've already seen at #244 (comment) and everything should work as expected.

This solution is most likely temporary while more E2E code gets reviewed and while we think about a better upload process in general :)

[gordon-quad]

Any news why PRs aren't merged yet?

[Zil0]

They are still awaiting review by someone from Matrix core team. Last ETA given by @uhoreg is that it should start happening in a month, and no one knows how long the whole process will take, since I may or may not be busy with other things at that time.

[vurpo]

It's currently very awkward to try to store the same access token across restarts (specifically it's difficult to create a new MatrixClient capable of E2EE from an existing access token), if you want to make a program that keeps the session across restarts and doesn't store the account password. I think this would need to be solved before E2EE is ready for production.

[gsantner]

should start happening in a month

: smiley :

[Half-Shot]

FYI. you folks may want to read https://github.com/matrix-org/matrix-python-sdk#project-status

[ara4n]

all the GSoC work here eventually got used to implement E2E in matrix-nio, which has ended up with excellent E2E support (it powers github.com/matrix-org/pantalaimon etc).

I'm inclined to close this wontfix?

[Half-Shot]

Yeah, there is no work being done here. Folks wishing to continue there E2E adventures will want to checkout matrix-nio.