ElectronPlatform: Add support for a event index using Seshat. #11125

poljar · 2019-10-11T15:05:34Z

This is the platform specific part of our E2EE search support. The ElectronPlatform is extended to make use of Seshat.

t3chguy · 2019-10-11T15:21:35Z

The flow type annotations in ElectronPlatform are a bit un-useful with {}

jryans

Thanks, looks like a good starting point!

If you'd like to work towards merging this before we have a build process ready for native modules, then we should do something like:

Test for the native node module(s) needed with runtime checks
Do not add the native node module as an app dependency (instead record what should be added in a doc)

The push-to-talk PR (which is in a similar boat) has a draft of docs around native node modules, so perhaps we can describe the manual steps needed in something like that.

As for the seshat backend itself, as you've seen, I filed various issues over there, but most are smaller nits that can be addressed whenever. I would like to see the following resolved in some way or another before merging integration into Riot:

jryans · 2019-11-06T12:08:32Z

electron_app/package.json

@@ -5,11 +5,17 @@
  "version": "1.4.2",
  "description": "A feature-rich client for Matrix.org",
  "author": "New Vector Ltd.",
+  "scripts": {
+    "build": "electron-build-env --electron 6.0.3 neon build seshat-node --release",
+    "postinstall": "yarn build"


Hmm, so this is just to call the line above...?

jryans · 2019-11-06T12:29:04Z

electron_app/package.json

@@ -5,11 +5,17 @@
  "version": "1.4.2",
  "description": "A feature-rich client for Matrix.org",
  "author": "New Vector Ltd.",
+  "scripts": {
+    "build": "electron-build-env --electron 6.0.3 neon build seshat-node --release",


We should really try hard to find a way to build that doesn't duplicate the Electron version in another place, as we'll surely forget to keep them synchronised. Maybe we need a wrapper script to pull it out of the main package.json?

The thing doesn't even work with the version passed as is. It seems to be only necessary if you need to rebuild seshat for some reason (e.g. if you're linking it from your dev folder) anyways.

Since we're adding run-time checks for seshat anyways it will be safe to remove it as well.

jryans · 2019-11-06T12:46:46Z

electron_app/package.json

-    "png-to-ico": "^1.0.2"
+    "png-to-ico": "^1.0.2",
+    "make-dir": "^3.0.0",
+    "seshat-node": "^0.2.0"


It would be quite nice to have this renamed to something without the redundant node instead. See matrix-org/seshat#9.

jryans · 2019-11-06T12:51:51Z

electron_app/src/electron-main.js

+                let p = path.normalize(path.join(eventStorePath, args[0]));
+                try {
+                    await makeDir(p);
+                    eventIndex = new seshat(p);


A common JS convention is for new-able things to start with a capital letter. I suppose we may not have a linting rule enforcing that at the moment, but I suggest changing the require to call it Seshat instead, unless there's a strong reason to always use the lowercase name.

jryans · 2019-11-06T12:54:06Z

electron_app/src/electron-main.js

@@ -149,6 +155,17 @@ autoUpdater.on('update-downloaded', (ev, releaseNotes, releaseName, releaseDate,
 ipcMain.on('ipcCall', async function(ev, payload) {
    if (!mainWindow) return;

+    const send_error = (id, e) => {


Use camel-case naming:

Suggested change

const send_error = (id, e) => {

const sendError = (id, e) => {

(We should have linting rules for this, but they seem not to have fired here... 🤔)

jryans · 2019-11-06T12:58:10Z

electron_app/src/electron-main.js

@@ -39,6 +39,8 @@ const { migrateFromOldOrigin } = require('./originMigrator');

 const windowStateKeeper = require('electron-window-state');
 const Store = require('electron-store');
+const seshat = require('seshat-node');


As part of attempting to land this without a path to build it, let's test at runtime for its existence with a try / catch.

poljar · 2019-11-13T16:26:42Z

This should be ready for another look. The thing that's missing is switching from seshat-node to matrix-seshat, enabling encryption, and removing the stuff from package.json. Will be a trivial fix, once I get matrix-seshat published.

poljar · 2019-11-14T15:23:55Z

The logout logic, and therefore event index deletion required a bit more changes here since I wrongly scoped our event index to a user id.

Should now be ready.

jryans

Thanks for the continued work here. It looks reasonable enough, but I'd like to review the final form with dep removed, documentation to enable, etc. so I can check the approach, see how it aligns with the push-to-talk PR, etc.

If you'd like another review Thursday - Friday while I am away, feel free to request review from the riot-web team, or else we can wait until I am back next week.

jryans

I realise you copied some of this text from the Push-to-talk PR. I hadn't reviewed it in depth over there yet, as it was still unclear how it would all come together...

Since it looks like you'll be the first to merge with native modules, that means the detailed text review is happening here. Apologies for all my copy edits, but I want make sure we convey the right message in these docs.

docs/native-node-modules.md

jryans · 2019-11-25T16:59:35Z

docs/native-node-modules.md

+If for some reason recompilation of Seshat is needed, e.g. when using a
+development version of Seshat using `yarn link`, or if the initial compilation was
+done for the wrong electron version, Seshat can be recompiled with the
+`electron-build-env` tool. Again from the `electron_app/` directory:
+
+    yarn add electron-build-env
+
+Recompiling Seshat itself can be done like so:
+
+    yarn run electron-build-env -- --electron 6.1.1 -- neon build matrix-seshat --release`
+
+Please make sure to include all the `--` as well as the `--release` command line
+switch at the end. Modify your electron version accordingly depending on the
+version that is installed on your system.


Hmm, so is electron-build-env needed even for the first installation of the matrix-seshat Node module, or is only a rebuild helper?

If it's only used for rebuilds, I wonder if we can find some way to repeat the build that happens at install time without this extra tool somehow, such as (cd node_modules/matrix-seshat; yarn run install) or similar.

I also noticed that the Electron docs suggest electron-rebuild, is that of any use here?

I believe it's only needed to rebuild it, but I'm not 100% on that.

The neon docs state this:

We are working on adding Neon support to electron-rebuild, so you'll be able to just drop Neon dependencies into your app like any other. For now, there's a tool called electron-build-env that you can use for building any Neon dependencies of your Electron app.

As far as I can tell this has not changed as of yet.

After looking around a bit, it's quite difficult to follow what's actually recommended and also matches what we need here. 😓 I don't want to block all this work on sorting it out though, so I filed a separate issue to improve this in a future step.

package.json

…ive modules.

jryans · 2019-11-26T11:08:03Z

riot.im/develop/config.json

@@ -27,7 +27,8 @@
        "feature_sas": "labs",
        "feature_room_breadcrumbs": "labs",
        "feature_state_counters": "labs",
-        "feature_many_integration_managers": "labs"
+        "feature_many_integration_managers": "labs",
+        "feature_event_indexing": "labs"


Please also add a short explanation of this flag in the labs docs as per the guide.

docs/native-node-modules.md

jryans · 2019-11-26T16:44:57Z

docs/native-node-modules.md

+If for some reason recompilation of Seshat is needed, e.g. when using a
+development version of Seshat using `yarn link`, or if the initial compilation was
+done for the wrong electron version, Seshat can be recompiled with the
+`electron-build-env` tool. Again from the `electron_app/` directory:
+
+    yarn add electron-build-env
+
+Recompiling Seshat itself can be done like so:
+
+    yarn run electron-build-env -- --electron 6.1.1 -- neon build matrix-seshat --release`
+
+Please make sure to include all the `--` as well as the `--release` command line
+switch at the end. Modify your electron version accordingly depending on the
+version that is installed on your system.


After looking around a bit, it's quite difficult to follow what's actually recommended and also matches what we need here. 😓 I don't want to block all this work on sorting it out though, so I filed a separate issue to improve this in a future step.

jryans

As always, thanks for working through all the nits and copy edits here! 😁

Looks like a few conflicts to resolve, but should be ready to merge. Assuming GH will let you, it's fine to merge without another review once those are resolved.

jryans · 2019-11-26T17:31:33Z

docs/labs.md

@@ -49,3 +49,11 @@ That's it. Now should see your new counter under the header.
 ## Multiple integration managers (`feature_many_integration_managers`)

 Exposes a way to access all the integration managers known to Riot. This is an implementation of [MSC1957](https://github.com/matrix-org/matrix-doc/pull/1957).
+
+## Event indexing or E2EE search support using Seshat (`feature_event_indexing`)


Probably "and" is more natural since Seshat supports both.

Suggested change

## Event indexing or E2EE search support using Seshat (`feature_event_indexing`)

## Event indexing and E2EE search support using Seshat (`feature_event_indexing`)

@kerryarchibald

* Remove `feature_favourite_messages` as it is has been abandoned for now ([\#11097](matrix-org/matrix-react-sdk#11097)). Fixes element-hq#25555. * Don't setup keys on login when encryption is force disabled ([\element-hq#11125](matrix-org/matrix-react-sdk#11125)). Contributed by @kerryarchibald. * OIDC: attempt dynamic client registration ([\element-hq#11074](matrix-org/matrix-react-sdk#11074)). Fixes element-hq#25468 and element-hq#25467. Contributed by @kerryarchibald. * OIDC: Check static client registration and add login flow ([\element-hq#11088](matrix-org/matrix-react-sdk#11088)). Fixes element-hq#25467. Contributed by @kerryarchibald. * Improve message body output from plain text editor ([\element-hq#11124](matrix-org/matrix-react-sdk#11124)). Contributed by @alunturner. * Disable encryption toggle in room settings when force disabled ([\element-hq#11122](matrix-org/matrix-react-sdk#11122)). Contributed by @kerryarchibald. * Add .well-known config option to force disable encryption on room creation ([\element-hq#11120](matrix-org/matrix-react-sdk#11120)). Contributed by @kerryarchibald. * Handle permalinks in room topic ([\element-hq#11115](matrix-org/matrix-react-sdk#11115)). Fixes element-hq#23395. * Add at room avatar for RTE ([\element-hq#11106](matrix-org/matrix-react-sdk#11106)). Contributed by @alunturner. * Remove new room breadcrumbs ([\element-hq#11104](matrix-org/matrix-react-sdk#11104)). * Update rich text editor dependency and associated changes ([\element-hq#11098](matrix-org/matrix-react-sdk#11098)). Contributed by @alunturner. * Implement new model, hooks and reconcilation code for new GYU notification settings ([\element-hq#11089](matrix-org/matrix-react-sdk#11089)). Contributed by @justjanne. * Allow maintaining a different right panel width for thread panels ([\element-hq#11064](matrix-org/matrix-react-sdk#11064)). Fixes element-hq#25487. * Make AppPermission pane scrollable ([\element-hq#10954](matrix-org/matrix-react-sdk#10954)). Fixes element-hq#25438 and element-hq#25511. Contributed by @luixxiul. * Integrate compound design tokens ([\element-hq#11091](matrix-org/matrix-react-sdk#11091)). Fixes vector-im/internal-planning#450. * Don't warn about the effects of redacting state events when redacting non-state-events ([\element-hq#11071](matrix-org/matrix-react-sdk#11071)). Fixes element-hq#8478. * Allow specifying help URLs in config.json ([\element-hq#11070](matrix-org/matrix-react-sdk#11070)). Fixes element-hq#15268. * Fix error when generating error for polling for updates ([\element-hq#25609](element-hq#25609)). * Fix spurious notifications on non-live events ([\element-hq#11133](matrix-org/matrix-react-sdk#11133)). Fixes element-hq#24336. * Prevent auto-translation within composer ([\#11114](matrix-org/matrix-react-sdk#11114)). Fixes element-hq#25624. * Fix caret jump when backspacing into empty line at beginning of editor ([\#11128](matrix-org/matrix-react-sdk#11128)). Fixes element-hq#22335. * Fix server picker not allowing you to switch from custom to default ([\element-hq#11127](matrix-org/matrix-react-sdk#11127)). Fixes element-hq#25650. * Consider the unthreaded read receipt for Unread dot state ([\element-hq#11117](matrix-org/matrix-react-sdk#11117)). Fixes element-hq#24229. * Increase RTE resilience ([\element-hq#11111](matrix-org/matrix-react-sdk#11111)). Fixes element-hq#25277. Contributed by @alunturner. * Fix RoomView ignoring alias lookup errors due to them not knowing the roomId ([\element-hq#11099](matrix-org/matrix-react-sdk#11099)). Fixes element-hq#24783 and element-hq#25562. * Fix style inconsistencies on SecureBackupPanel ([\element-hq#11102](matrix-org/matrix-react-sdk#11102)). Fixes element-hq#25615. Contributed by @luixxiul. * Remove unknown MXIDs from invite suggestions ([\element-hq#11055](matrix-org/matrix-react-sdk#11055)). Fixes element-hq#25446. * Reduce volume of ring sounds to normalised levels ([\element-hq#9143](matrix-org/matrix-react-sdk#9143)). Contributed by @JMoVS. * Fix slash commands not being enabled in certain cases ([\element-hq#11090](matrix-org/matrix-react-sdk#11090)). Fixes element-hq#25572. * Prevent escape in threads from sending focus to main timeline composer ([\element-hq#11061](matrix-org/matrix-react-sdk#11061)). Fixes element-hq#23397.

poljar added 2 commits October 11, 2019 16:05

ElectronPlatform: Add support for a event index using Seshat.

1dbdd0a

ElectronPlatform: Fix lint errors.

71023ae

dbkr added the cross-signing-sprint label Oct 24, 2019

This was referenced Oct 24, 2019

Rethink release process so we can support native node modules #11227

Closed

As a user I want to find old messages by searching, in encrypted and unencrypted rooms element-hq/user-stories#5

Open

ilmaisin mentioned this pull request Oct 24, 2019

Declaring End-to-end Encryption stable and turning it on by default for private rooms. #6779

Closed

14 tasks

jryans requested a review from a team October 28, 2019 11:08

t3chguy mentioned this pull request Nov 4, 2019

'Files' view in e2e-encrypted rooms does not show files #4959

Closed

turt2live requested review from bwindels and jryans and removed request for a team November 4, 2019 17:22

jryans removed the request for review from bwindels November 6, 2019 12:05

jryans requested changes Nov 6, 2019

View reviewed changes

lampholder added phase:5 labels Nov 7, 2019

poljar added 5 commits November 8, 2019 13:45

electron-main: Use camle-case for the send_error method.

94196eb

electron-main: Use a capital letter for the seshat import.

a6839af

ElectronPlatform: Implement the EventIndexManager for Seshat.

c3c5756

electron-main: Add a missing break.

449eca6

electron-main: Check for seshat existence instead of erroring out.

437c59f

poljar requested a review from jryans November 13, 2019 16:27

poljar added 4 commits November 14, 2019 12:12

electron-main: Switch to matrix-seshat.

e9352fc

electron-main: Enable encryption for Seshat.

b90a94b

ElectronPlatform: Don't scope the event index per user.

7147af8

electron-main: Rework the event index initialization and deletion.

dd2c210

jryans requested changes Nov 20, 2019

View reviewed changes

poljar added 2 commits November 21, 2019 10:51

electron_app: Remove Seshat from the dependencies.

2f2cbad

docs: Add documentation explaining how to enable Seshat support.

d0b5391

poljar requested a review from jryans November 25, 2019 09:39

jryans requested changes Nov 25, 2019

View reviewed changes

poljar added 10 commits November 26, 2019 09:56

package.json: Remove the unneeded Neon/Seshat dependencies.

e96c44c

native-node-modules: Add a header level to the title.

4c629e8

native-node-modules: Don't mention the riot version that supports nat…

da4b403

…ive modules.

native-node-modules: Add a section about cross compilation.

b52141d

native-node-modules: Reword the second paragraph.

5f6636e

native-node-modules: Explain the packaging situation a bit.

b1aff29

native-node-modules: Expand the Seshat subtitle a bit.

40f2648

native-node-modules: Capitalize some project names.

f0fe968

native-node-modules: Explain how to install Rust and link to its docs.

5b8e918

native-node-modules: Remove a spurious and.

1869350

poljar requested a review from jryans November 26, 2019 09:50

jryans mentioned this pull request Nov 26, 2019

Clean up docs on native Node modules #11517

Closed

jryans requested changes Nov 26, 2019

View reviewed changes

poljar added 2 commits November 26, 2019 18:02

native-node-modules: Mention that Seshat requires SQLCipher.

b0783a8

labs: Document the event indexing labs feature.

f28f27a

poljar requested a review from jryans November 26, 2019 17:18

jryans approved these changes Nov 26, 2019

View reviewed changes

labs: Clarify that the event indexing feature supports E2EE search.

e5956de

poljar merged commit e5956de into develop Nov 26, 2019

arthurlutz mentioned this pull request Dec 16, 2019

Feature request: export chat logs #2630

Closed

jryans added the phase:1 label Feb 10, 2020

worldofpeace mentioned this pull request Feb 18, 2020

riot-{web,desktop}: 1.5.8 -> 1.5.9 NixOS/nixpkgs#80357

Merged

10 tasks

poljar deleted the poljar/seshat-pr branch June 21, 2020 17:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ElectronPlatform: Add support for a event index using Seshat. #11125

ElectronPlatform: Add support for a event index using Seshat. #11125

poljar commented Oct 11, 2019

t3chguy commented Oct 11, 2019

jryans left a comment

jryans Nov 6, 2019

jryans Nov 6, 2019

poljar Nov 8, 2019

jryans Nov 6, 2019

jryans Nov 6, 2019

jryans Nov 6, 2019

jryans Nov 6, 2019

poljar commented Nov 13, 2019

poljar commented Nov 14, 2019

jryans left a comment

jryans left a comment

jryans Nov 25, 2019

jryans Nov 25, 2019

poljar Nov 25, 2019

jryans Nov 26, 2019

jryans Nov 26, 2019

jryans Nov 26, 2019

jryans left a comment

jryans Nov 26, 2019

	const send_error = (id, e) => {
	const sendError = (id, e) => {

	## Event indexing or E2EE search support using Seshat (`feature_event_indexing`)
	## Event indexing and E2EE search support using Seshat (`feature_event_indexing`)

ElectronPlatform: Add support for a event index using Seshat. #11125

ElectronPlatform: Add support for a event index using Seshat. #11125

Conversation

poljar commented Oct 11, 2019

t3chguy commented Oct 11, 2019

jryans left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

poljar commented Nov 13, 2019

poljar commented Nov 14, 2019

jryans left a comment

Choose a reason for hiding this comment

jryans left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jryans left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment