Enable lazy tangle updates in the Account

[PhilippGackstatter]

Description of change

Give users of the Account more control over when to publish to the tangle. For example, if an identity is created and a service and a method are added subsequently, this would currently result in three messages being published to the tangle. With this PR, all changes can be applied to a locally stored document, and then published in a single message. That increases the overall speed, since there are fewer network interactions and less Proof-of-Work, and reduces the used storage space on nodes.

Implementation

The Account already organizes the updates to a document in Commits. Each Commit is identified by a Generation (an incrementing u32). The basic idea is to store which generation was last published to the tangle, and when updating, publishing all commits that have been added since.

One potential use-case might require setting the autopublish configuration not just when building the Account, but also on the instance itself. Users could then switch between the modes very flexibly. I would probably not add this for now, though, and see if the requirement actually comes up in practice. This is mainly to be in line with the current immutability of the account config, but would be open to discuss this. It would be fairly simple to add this through an AtomicBool in the Config.

Testing

Testing this really only makes sense if there is a tangle. As far as I'm aware, no tangle interaction is currently tested, except "passively" in the examples (they do run in CI now, but only once per day, not per PR). So we could do a couple of things:

1. We could skip testing the network aspects of this feature, too, which is basically everything in that case.
2. Another idea is to use a private tangle in CI and test it that way. That means we're not reliant on the mainnet or testnet, and tests should be faster. This would probably run per PR, and would also test the private tangle support as a side bonus. Would add a barrier to local testing though, as we need to run a private tangle locally, or skip these tests - neither is ideal.
3. We could actually start using the main- or testnet, increasing testing time, but requiring no special private tangle configuration and being always available, from both CI and locally.
4. We could add an example that includes assertions. This is quite common in Rust in general, but we don't have any assertions in our examples right now, so it would break our style a bit.

I've chosen the third option and added an example that tests directly on the testnet. The test runs in 5 seconds; not adding significantly to the CI execution time.

Links to any relevant issues

part of #325

The other tasks of the issue will be completed in separate PRs.

Type of change

Add an x to the boxes that are relevant to your changes.

• Bug fix (a non-breaking change which fixes an issue)
• Enhancement (a non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Fix

How the change has been tested

The stronghold implementation was tested by modifying the stronghold example. The added test uses the MemStore.

Change checklist

Add an x to the boxes that are relevant to your changes.

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[cycraig]

Can we change get_lock to lazily create locks as-needed instead of creating them all at once here and on insert? That would be a performance benefit too since one would not need locking at all if not using update_identity / publish_changes, and then only for those identities being updated.

I am a bit concerned with how the IdentityIndex is used by the Storage where it is cloned (in MemStore) or serialized then de-serialized (Stronghold) that one could circumvent the locks by using different copies of them. It may be better to store the lock map in the Account struct proper to avoid these issues altogether?

[PhilippGackstatter]

You're right, this is better. I was previously worried about contention on the RwLock<IdentityIndex> in the account, since get_lock would then modify the index and require the lock to be acquired as a write lock rather than just a read one. But most likely the performance is better this way then adding locks for a large number of identities.

Yes theoretically possible, but prevent by the design of the account. The index is only cloned when returned by storage, and added when the account is created, and never modified as a whole. So for any given account no cloning happens, and thus this should be fine.

Having the locks in the account requires more locks, as you would have RwLock<IdentityIndex> and also a separate RwLock<HashMap<IdentityId, Arc<RwLock<IdentityId>>>>, which I wanted to avoid. One possibility is to remove the custom Clone impl and add a derive, in which case the Arc around the lock would be cloned, and thus your concern of getting around the locks when cloning would be alleviated.

[cycraig]

I think using something like Dashmap<IdentityId, Arc<RwLock<IdentityId>>> in the account wouldn't be too terrible.

Clone is only used by MemStore, the index is (de)serialized by the Stronghold storage so it would still suffer from that. I think in general this may not be a problem since, as you indicated, the Account itself is not cloned. I also do not see any cases where an Account re-loads the index from storage after the initial configuration but may be incorrect.

Overall the locking seems fine. I'll take another look as there may be places where a read lock is appropriate.

[cycraig]

Looks good to me! Minor comments.

I would still prefer the lock map to be in the Account itself rather than the index to avoid any potential index cloning, storage issues and workarounds for e.g. PartialEq but this still solves the race condition.