Structured Change Detection for Mutation Events and Schema Compliance

[joshuakarp]

Specification

Whilst considering vault schemas, it became apparent that it would be extremely beneficial to implement some kind of filesystem event-watching API integrated into the EFS, such that we can track changes before and after they're made directly to the files in the EFS. That is, before and after changes are made to files, we can generate an accumulating list of these changes. Then, we can provide hooks for these pre-edit and post-edit changes.

This would allow us to solve 3 problems in one:

• enforce the vault schema before a file change is made (i.e. pre-edit vs. having to do a post-edit check)
• easier automatic commit message generator
• accurate elimination of dirty commits

The latter 2 problems are currently being solved by a hacky, post-edit solution through recursive scans over the EFS (see the top-level comment from @scottmmorris here).

Additional context

• vault schemas epic Vault and File Schema - Ingress and Egress Schemas Polykey#222

Some further discussion of pre-edit and post-edit from the vaults refactoring MR:

• https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/205#note_687323248
• https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/205#note_687486709

Tasks

1. ...
2. ...
3. ...

[CMCDragonkai]

@scottmmorris can you expand on the relevant items in the vaultsrefactoring MR https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/205 and why is this issue important.

[joshuakarp]

There's been some further discussion of this with VaultInternal here https://vimeo.com/manage/videos/652700328

Specifically, we'd be able to significantly reduce the complexity of VaultInternal.commit by implementing this.

Currently, commit performs a recursive search over all files within the vault directory (in the EFS), utilising the iso-git-provided statusMatrix to deduce the changes made. This would be completely unnecessary with the addition of this event-watching API.

It would also allow us to solve MatrixAI/Polykey#260 (single digit modification is not detected by isomorphic-git's status call) because we'd no longer need to rely on the statusMatrix (which is most likely the cause of this "mystery bug").

[CMCDragonkai]

Interesting usage of git for finding changed files: https://github.com/facebook/jest/tree/main/packages/jest-changed-files.

Relies on shelling out to the git command. Obviously pre-edit and post-edit change events is more efficient, but interesting to compare.

[tegefaulkes]

Linear marked this as stale so I'm reopening.