Multiple user collaboration for data section #10712

aptkingston · 2023-05-26T09:17:07Z

Edit - do not merge. Still have issues with conflicts when attempting to acquire usage quota docs which is being investigated, although I don't believe that's specifically part of this feature, but is highlighted by this feature.

2nd edit - may as well merge this anyway and see what the conflicts are like in a real env.

Description

This PR allows multiple users to collaborate on the same app. For now, the extent of that collaboration is limited to the data section only. Secondary users will not be able to see or edit anything in the design or automation sections.

Brief summary of changes:

Multiple users can open the same app
Any changes made will be received by other users in real time
All existing websockets (and this new one) now work in scalable, multi node environments
"Lock" mechanism improved so that the lock is instantly lost once stopping working on the builder. This means as soon as one user is done, another user can immediately take full control of the app.
Standardised usage of user avatars across the whole platform. Users colour and their initials are the same everywhere now.
Technical stuff can be read at the bottom
Small enhancement to close context menu when pressing escape in grids
FIx issues with users table due to it returning a different response format than all other tables

Editing an app as the first users

If you're the first user to edit an app, nothing changes. You can open it as normal and work away. The only visual change for this first user is that you'll see your own user avatar displayed in the top right of the builder:

Editing an app when someone else is already editing it

Previously if someone was editing an app, we displayed a lock symbol in the app list page and we prevented other users from clicking it. The lock has been removed, but we still indicate if another user is using your app. As a secondary user looking at the app list, it will now look like this:

We can still click into it, but we show that another user is currently editing it. It's important to note here - the user we show is the "primary" user that has full control of the app. There could be 10 other users also inside the app, but we only show the person who currently has control.

Clicking into the app now, we are presented with a cut-down version of the builder:

The immediate visual differences are:

The top nav has been replaced with a message informing the user that someone else is editing your screens and automations
You are not able to navigate to any other part of the builder, manage users, publish, update the app version or do anything else outside of the data section
You do not have access to the command palette (this would easily allow circumventing our restrictions)
You can see all the users in the top right

The primary user will see that a user has connected, and a new avatar will be displayed in the top right for them too.

Grid collaboration

If you're using the same grid as another user, you'll be able to see what they're doing:

Opening multiple tabs

You're able to open as many tabs as you like. Every tab will receive changes made by other users and function exactly the same. You won't show up multiple times in the avatar list though, because you're the same user.

The only time you'll see a visual difference with multiple tabs, is if you edit the same grid in multiple tabs. In this scenario everyone (including yourself) will be able to see all your selected cells. e.g. here's the same user with 3 tabs open, all looking at the same grid:

From this other user's perspective, there is one other user editing the same app, but they have 3 sessions open and have selected 3 different cells.

From your own perspective, you'll be able to see all the other cells selected by you:

When you have multiple tabs open, you'll retain control as the "primary" builder until all tabs are closed.

Avatar standardisation

Previously we used multiple different avatars across the platform. These all showed different colours and different initials, so there was no consistency at all. I've standardised everything to use the same new component now, which ensures that no matter where you are inside the builder, the same user will always have the same avatar. Their colour will always be the same, and so will the initials inside the avatar.

Global users will appear identically to their app user equivalent as well.

Technical stuff

A new "builder" websocket is being used to propagate changes in real time. This means there are now 3 websockets used altogether:

Builder (currently now only used for users in the builder, and data section changes). Users are grouped by what app they're in.
Grid (used for data changes). Users are grouped by what table they're editing.
Client (used for plugin updates). All users connect to the same socket.

All sockets continue to be readonly. The API is used as normal to make changes, and those changes are piped down the websocket to all users.

The new building websocket handles all CRUD events for:

Tables
Views
Datasources

I've improved the performance of all sockets so that they now actually send down the new changed docs, as opposed to simply sending a notification that a certain doc ID changed, causing all clients to fetch the new version. This change is more scalable and prevents API load spikes whenever someone else makes a change.

Socket.io (the library we use for websockets) offers a redis adapter which can be used to proxy messages between multiple server instances. This is what I'm using to ensure compatibility with scalable environments. I've been unable to test this locally, so I'll only really know if it works once we get this up on QA (and we might need to manually scale up the number of server pods just to test this). I've confirmed that redis pub/sub messages are indeed being sent so I'm assuming this is working.

There's also some unknowns regarding how websockets will perform in prod, considering the multiple layers of proxies and load balancers. We'll have to see what happens.

Clarification of the lock system changes

The lock system is still in use. Instead of being used to control who the only editor of an app is, it's now used to control who the "primary" user is. The lock is still acquired in middleware, and there's still a timeout to ensure idle people still get kicked. The only change made to the lock system is a faster removal of the lock when a user disconnects. The new builder websocket means we can immediately detect when a user stops editing an app, for any reason. When we detect this socket disconnecting, we check if that was their only session in the app, and if so we remove the app lock. This is much nicer than having to wait 10 minutes and explain to other users what was happening.

Addresses:

(LInear) BUDI-6772
Escape Key should dismiss cell context menu #10705

…ry builder or not

…ng loading states

…prevent access to the design and automate pages

…ion-multidev

… grid column labels. Allow deletion of duplicated user columns

…ion-multidev

aptkingston · 2023-05-26T09:18:31Z

packages/backend-core/src/redis/utils.ts

@@ -27,6 +27,7 @@ export enum Databases {
  GENERIC_CACHE = "data_cache",
  WRITE_THROUGH = "writeThrough",
  LOCKS = "locks",
+  SOCKET_IO = "socket_io",


This redis namespace is never used because socket.io generates it's own key names, so I've literally only added this to be consistent with other usages of constructing instances of our redis wrapper, as technically we could still manually use that client instance. So I think it's still prudent to reserve a dedicated namespace.

With this - is there anyway we could get the socket.io layer to use its own Redis database? This way its in a completely separate keyspace and won't show up in any key/scan lists, could help with performance of certain features which enumeration a lot of keys.

Since it's only using redis for pub/sub, I'm not sure anything it actually sets is persistent. It's all just ephemeral messages which are sent to subscribers but then dropped. At least I'm pretty sure that's how it works.

ah I see - so it doesn't show up in the keyspace at all, thats perfect then it won't affect anything else in terms of performance!

aptkingston · 2023-05-26T09:20:29Z

packages/server/src/api/controllers/row/index.ts

@@ -12,21 +13,9 @@ function pickApi(tableId: any) {
  return internal
 }

-function getTableId(ctx: any) {


Just moved this to utils so I can reuse it elsewhere.

aptkingston · 2023-05-26T09:22:00Z

packages/server/src/api/controllers/view/index.ts

@@ -48,18 +49,16 @@ export async function save(ctx: Ctx) {
  if (!view.meta.schema) {
    view.meta.schema = table.schema
  }
-  table.views[viewName] = view.meta
+  table.views[viewName] = { ...view.meta, name: viewName }


There were some weird inconsistencies with where the name of a view was stored, especially in the frontend code that calls this endpoint. I've rewritten most of that, and now I'm also just ensuring that the view name is properly saved the view itself.

aptkingston · 2023-05-26T09:22:53Z

packages/server/src/app.ts

@@ -61,7 +61,6 @@ if (env.isProd()) {

 const server = http.createServer(app.callback())
 destroyable(server)
-initialiseWebsockets(app, server)


Websocket initialisation has been moved to startup.ts, as it needs to be done after redis has been initialised, so that we can use the socket.io redis adapter.

aptkingston · 2023-05-26T09:23:36Z

packages/server/src/middleware/builder.ts

-  // they do have lock, update it
-  await updateLock(appId, ctx.user)
+  // If this user already owns the lock, then update it
+  if (await doesUserHaveLock(appId, ctx.user)) {


Now that we allow multiple users, we don't want to error if the user doesn't have the lock. If they don't have the lock, we just ignore them.

If the user does own the lock, then we update the expiration time.

codecov-commenter · 2023-05-26T09:23:51Z

Codecov Report

Merging #10712 (322c7b3) into develop (554a94d) will not change coverage.
The diff coverage is n/a.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@           Coverage Diff            @@
##           develop   #10712   +/-   ##
========================================
  Coverage    50.03%   50.03%           
========================================
  Files          153      153           
  Lines         5144     5144           
  Branches      1028     1028           
========================================
  Hits          2574     2574           
  Misses        2354     2354           
  Partials       216      216

Impacted Files	Coverage Δ
packages/backend-core/src/redis/utils.ts	`61.90% <ø> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

aptkingston · 2023-05-26T09:25:15Z

packages/server/src/utilities/redis.ts


-// we init this as we want to keep the connection open all the time
+// We need to maintain a duplicate client for socket.io pub/sub
+let socketSubClient: any


The socket.io redis adapter requires two separate redis clients, one for pub and one for sub. I can't use another instance of our redis wrapper class for this, as they reuse the same underlying redis clients depending on the "selectDb" flag. The easiest way to handle this was simply maintain a duplicate of the core redis client here, and ensure it's also cleaned up when appropriate.

aptkingston · 2023-05-26T09:27:11Z

Added a bunch of comments to explain the various server changes, but would appreciate a review in case anything should be moved around or done differently 🙏

mike12345567

LGTM! Looking forward to playing with this.

Rory-Powell · 2023-05-30T08:17:59Z

packages/builder/src/builderStore/websocket.js

+    tables.replaceTable(id, table)
+  })
+
+  // Table events


Suggested change

// Table events

// Datasource events

?

Rory-Powell · 2023-05-30T08:19:19Z

packages/builder/src/components/start/AppRow.svelte

  export let lockedAction

+  $: editing = app?.lockedBy != null
+  $: initials = helpers.getUserInitials(app?.lockedBy)


Rory-Powell · 2023-05-30T08:29:43Z

packages/server/src/api/controllers/table/index.ts

@@ -77,6 +78,7 @@ export async function save(ctx: UserCtx) {
  ctx.eventEmitter &&
    ctx.eventEmitter.emitTable(`table:save`, appId, savedTable)
  ctx.body = savedTable
+  builderSocket.emitTableUpdate(ctx, savedTable)


It would be beneficial to update the tests to allow us to mock builderSocket and confirm it's called with the correct data. The same applies for datasource, rows, etc.

I'm not sure of the performance impact of setting up multiple sockets, but it'd be preferable to avoid this in unit tests

Rory-Powell · 2023-05-30T08:32:58Z

packages/server/src/websockets/grid.ts

          socket.leave(currentRoom)
        }

        // Join new room
        currentRoom = tableId
        socket.join(currentRoom)
-        socket.to(currentRoom).emit("user-update", socket.data.user)
+        socket.to(currentRoom).emit("user-update", user)


It would be nice to enumerate our socket events, for discovery and reuse.

aptkingston added 23 commits May 12, 2023 13:55

Remove single user restriction and notify users if they are the prima…

b4babfe

…ry builder or not

Update top bar when not the primary builder and prevent flashing duri…

4e32ffb

…ng loading states

Disable command palette and tour for non lock holders and forcefully …

1ac5524

…prevent access to the design and automate pages

Update grid websocket to send actual changes down to reduce API load

a305d21

Restore proper functionality for showing other users inside grids

4eff61d

Fix visual issues with other users selecting cells

d042917

Fix cell colours with multiple users

33a6936

Fix validation error position for new rows

271fe9e

Improve builder socket

01c0822

Merge branch 'develop' of github.com:Budibase/budibase into data-sect…

1ca774b

…ion-multidev

Merge branch 'develop' of github.com:Budibase/budibase into data-sect…

fc56925

…ion-multidev

Simplify websocket comms for grid and improve builder socket

9f00e8d

Handle table, query and datasource events using builder socket

a0000d9

Update comments

f749f7d

Merge branch 'develop' of github.com:Budibase/budibase into data-sect…

2a2d8b9

…ion-multidev

Merge branch 'develop' of github.com:Budibase/budibase into data-sect…

cef0287

…ion-multidev

Merge branch 'develop' of github.com:Budibase/budibase into data-sect…

0cb35b7

…ion-multidev

Use socker.io redis adapter to broadcast events to all server instances

4314ec4

Fix issue with schema mutation in grids. Use displayName to represent…

fbb9ef6

… grid column labels. Allow deletion of duplicated user columns

Ensure text cells account for being used for number cells too

017145a

Update lock icon and remove logs

f117e4c

Standardise usage of user avatars and colours across the entire platform

f7da0b3

Merge branch 'develop' of github.com:Budibase/budibase into data-sect…

1ff2b5c

…ion-multidev

aptkingston commented May 26, 2023

View reviewed changes

aptkingston requested review from mike12345567 and Rory-Powell May 26, 2023 09:26

aptkingston added 3 commits May 26, 2023 10:29

Lint

0f20115

Close context menu on escape keypress

5a2801b

Handle users tables edges cases when working with grids

322c7b3

aptkingston added do not merge PR is not ready to be merged - generally the PR description should say why and removed do not merge PR is not ready to be merged - generally the PR description should say why labels May 26, 2023

mike12345567 approved these changes May 30, 2023

View reviewed changes

aptkingston merged commit 0e3368a into develop May 30, 2023

aptkingston deleted the data-section-multidev branch May 30, 2023 08:08

github-actions bot locked and limited conversation to collaborators May 30, 2023

Rory-Powell reviewed May 30, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple user collaboration for data section #10712

Multiple user collaboration for data section #10712

aptkingston commented May 26, 2023 •

edited

Loading

aptkingston May 26, 2023 •

edited

Loading

mike12345567 May 26, 2023 •

edited

Loading

aptkingston May 26, 2023

mike12345567 May 26, 2023

aptkingston May 26, 2023

aptkingston May 26, 2023 •

edited

Loading

aptkingston May 26, 2023

aptkingston May 26, 2023

codecov-commenter commented May 26, 2023 •

edited

Loading

aptkingston May 26, 2023 •

edited

Loading

aptkingston commented May 26, 2023

mike12345567 left a comment

Rory-Powell May 30, 2023

Rory-Powell May 30, 2023

Rory-Powell May 30, 2023

Rory-Powell May 30, 2023

Multiple user collaboration for data section #10712

Multiple user collaboration for data section #10712

Conversation

aptkingston commented May 26, 2023 • edited Loading

Description

Editing an app as the first users

Editing an app when someone else is already editing it

Grid collaboration

Opening multiple tabs

Avatar standardisation

Technical stuff

Clarification of the lock system changes

aptkingston May 26, 2023 • edited Loading

Choose a reason for hiding this comment

mike12345567 May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aptkingston May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented May 26, 2023 • edited Loading

Codecov Report

aptkingston May 26, 2023 • edited Loading

Choose a reason for hiding this comment

aptkingston commented May 26, 2023

mike12345567 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aptkingston commented May 26, 2023 •

edited

Loading

aptkingston May 26, 2023 •

edited

Loading

mike12345567 May 26, 2023 •

edited

Loading

aptkingston May 26, 2023 •

edited

Loading

codecov-commenter commented May 26, 2023 •

edited

Loading

aptkingston May 26, 2023 •

edited

Loading