Preparing for Testnet Deployment

[tegefaulkes]

Description

Note that this has been migrated from #326.

Once #310 has been merged, we'll finally be able to move onto the deployment of our testnet into AWS.

I foresee this to be achievable in a few stages:

1. Deploy Polykey onto AWS and ensure we have a working remote keynode (Testnet Node Deployment (testnet.polykey.io) #194)
2. Create the testnet seed nodes and store these securely somewhere (Testnet securely maintain a pool of recovery codes #285)
3. Use these seed nodes for our domain name (Update testnet.polykey.io to point to the list of IPs running seed keynodes #177)

• Relates to NAT-Traversal Testing with testnet.polykey.io #159
• Relates to Create automated tests for establishing connection via hole-punch signalling message #161
• Migrated from Testnet Deployment #326
• Relates to Asynchronous Promise Cancellation with Cancellable Promises, AbortController and Generic Timer #297

Issues Fixed

• Fixes Support host (IP address) in parseSeedNodes #324 being completed in Extracting Node Connection Management out of NodeManager to NodeConnectionManager #310

Testnet deployment:

• Relates to Testnet Node Deployment (testnet.polykey.io) #194
• Relates to Testnet securely maintain a pool of recovery codes #285
• Relates to Update testnet.polykey.io to point to the list of IPs running seed keynodes #177

NodeGraph Structure:

• Relates Support read-after-write consistency for NodeGraph bucket operations #244
• Fixes Test communications when root key pair is renewed and NodeID is changed #317
• Fixes Tree-based Kademlia Data Structure - k-bucket library #158
• Fixes Setting a Node involves NodeManager and NodeGraph, where NodeManager handles bucket overflow policy #359

Node Adding Policies:

• Fixes Improve node search over k-buckets (getClosestLocalNode) #212
• Fixes Investigate means of authenticating a node ID when adding to NodeGraph #322
• Fixes Seed node not adding details of connecting node to its NodeGraph #344
• Fixes Adding timeout overrides to NodeConnectionManager methods #363

Node Removal Policies:

• Fixes Remove "revoked" node ID -> node address mappings in NodeGraph #150
• Fixes Expunge any added "invalid" nodes from k-buckets #226
• Fixes Supporting the "refresh" operation for NodeGraph buckets #345

Tasks

• [ ] 2. Complete Support host (IP address) in parseSeedNodes #324 - being completed in Extracting Node Connection Management out of NodeManager to NodeConnectionManager #310
• 1. Read through and revisit all previous issues
• [ ] 3. Complete Testnet Node Deployment (testnet.polykey.io) #194 - Extracted out, should be finished after this PR merges
  • Ultimately provides us with a deployment of Polykey on AWS!
  • Task list there should be sufficient to follow process.
• [ ] 4. Complete Testnet securely maintain a pool of recovery codes #285 - Extracted out, should be finished after this PR merges
• [ ] 5. Complete Update testnet.polykey.io to point to the list of IPs running seed keynodes #177 - Extracted out, should be finished after this PR merges
• [ ] 6. Created automated testing that utilises the testnet - to be done in Tests for NAT-Traversal and Hole-Punching #357
  • [ ] - NAT-Traversal Testing with testnet.polykey.io #159
  • [ ] - Create automated tests for establishing connection via hole-punch signalling message #161
• 7. Add agent service nodes tests Testnet Node Deployment (testnet.polykey.io) #194 (comment)
  • - nodesChainDataGet
  • - nodesClosestLocalNode
  • - nodesHolePunchMessage
• 8. Changed Cmd to EntryPoint because PK only has 1 executable and makes docker run ... easier.
• 9. The --seed-nodes='<defaults>;...' will now mean that any specified seed nodes overrides the defaults rather than the other way around
• 10. If seed nodes contains the own NodeID, the own NodeId will be automatically filtered.
• 11. Make pk agent start return status information such as nodeId and not just recoveryCode
• 12. All PK commands can contact a remote agent (except agent start and the like), it needs to be made more obvious whether we are contacting the local agent or remote agent Testnet Deployment #326 (comment)
  • This is done by making the required client connection parameters of node ID, client host and client port all or nothing when set. This means if they are partially set, this results in a new exception ErrorCLIClientOptions.
  • If they are all set, that means a connection to an agent not specified by the local status.
  • If none are set, this means relying on the local status.
• 13. The seed node should be adding the details of any node connecting to it to its own NodeGraph: see issue Seed node not adding details of connecting node to its NodeGraph #344
  • currently the seed node does not add an entry in its NodeGraph for a connecting node
  • this is essential for the Kademlia implementation to start to function, and for the node's details to eventually propagate out into the wider Polykey network
  • see Testnet Deployment #326 (comment)
• 14. Investigate and fix encountered Polykey errors/problems during testnet testing Testnet Deployment #326 (comment):
  • [ ] a. No output provided on pk identities trust? However, I understand that we've previously shied away from providing "success" output on every single one of our commands. It can be tricky to diagnose problems when this is the case though. - moved this to https://github.com/MatrixAI/js-polykey/issues/334#issuecomment-1043779027
  • [ ] b. pk identities trust seemingly not adding a node to the gestalt graph when it doesn't already exist in GestaltGraph/NodeGraph - see Testnet Deployment #326 (comment) and https://matrixai.slack.com/archives/CEAUUV5QX/p1645069875382019 - moved this to https://github.com/MatrixAI/js-polykey/issues/334#issuecomment-1043779027
  • c. SyntaxError: Unexpected end of data on an "invalid" node ID when using decodeNodeId - rebased on master has new js-id 3.3.1 that catches syntax errors when decoding multibase encoded strings
• 15. Refactor NodeGraph structure
  • The new structure will use 3 nested sublevels to achieve Support read-after-write consistency for NodeGraph bucket operations #244
    • buckets sublevel will contain each bucket, where each bucket sublevel contains NodeId to NodeData
    • meta sublevel contains each bucket, where each bucket sublevel is a config structure, currently only count
    • lastUpdated or index sublevel that contains each bucket, and each bucket sublevel contains lexi(lastUpdated)-NodeId to NodeId, the key is a compound index, allowing us to efficiently acquire the most up-to-date or least up-to-date node entry
  • New methods like getNodes and getBuckets will allow us to interrogate the state of the NodeGraph efficiently, by streaming data out of the NodeGraph, this will be important for debugging, and later analytics
  • Integration of DBTransaction and new iterator feature of sublevels that allow us to maintain a snapshot of the DB at a point in time
  • Reviewing kademlia trie structure and bringing ideas in - reviewed and discovered that trie structure properties are not necessary in our NodeGraph, our design is already efficient and optimal
• 16. Optimise getClosestNode method based on new NodeGraph structure
• 17. Relationship between NodeGraph and NodeManager should be tightly coupled due to the adding and removal policies of nodes, maintaining the bucket limit must involving pinging old nodes
• 18. tests for changing root keys and connections.
  • Check how active connections work during a keychange.
  • Check if new connections use new TLS config
  • check if a renew/reset key event propigates to where it is needed.
• 19. cleaning up and small fixes to nodesUtils parser functions
• 20. adding force and ping flags to nodes add command

Final checklist

• Domain specific tests
• Full tests
• Updated inline-comment documentation
• Lint fixed
• Squash and rebased
• Sanity check the final build

[tegefaulkes]

This was migrated from #326 since we needed to change to a feature branch. Work from this PR/branch from now on.

[CMCDragonkai]

@tegefaulkes please remember to tick off anything above in case they are done, or cross out things that are no longer relevant in this PR.

[CMCDragonkai]

Change the base branch to target #374 branch.

[tegefaulkes]

Starting rebase on feature-dep-upgrades.

[tegefaulkes]

I've done the first blush rebase but there are some parts that I have to fix. Most of it is just the NodeGraph since It had a major refactor in this PR. So I need to go over it and apply the locking and DB changes. On top of that there are likely a bunch of bugs that were introduced that I have to review and fix.

[tegefaulkes]

Finished with the re-base + initial fixes. Everything is building and linting fine now. I'm going to start fixing the tests.

[tegefaulkes]

I can't get the iterator to work properly with the gt or lt options. I'm not sure if I'm using it wrong or if it's broken. Looking at the js-db tests we don't seem to be testing if they work so it might be broken.

[CMCDragonkai]

This PR will need to target staging and rebase on top. However @emmacasolin is still pending some changes to the CI/CD so we can hold off on that for now. There are changes from #374 that this should rebase on, because of all the fixes to logging.

[CMCDragonkai]

@tegefaulkes is this PR's description updates to point to abort controller changes?

[CMCDragonkai]

Also please clean up the task list, and start adding ETAs into it as well.

What is the final task list?

[tegefaulkes]

Re-based and targeting staging now.

[emmacasolin]

The wip commits have now been squashed and renamed. All of the tests should still be passing, in which case this can be merged into staging and the CI commits moved on top of this. There shouldn't be anything here that relies on those changes.

[CMCDragonkai]

Ok merging into staging, afterwards @emmacasolin you can rebase and move the CI commits above this merge commit.

[CMCDragonkai]

Many of the issues didn't autoclose because PK's default branch is still master. I've closed relevant issues manually, but we will switch to staging as the default branch once the CI/CD is all fixed up in the staging branch.

[CMCDragonkai]

Some remaining issues that come out of this PR:

• Support host (IP address) in parseSeedNodes #324 - need @tegefaulkes to confirm if this is done, or @emmacasolin can confirm
• Test communications when root key pair is renewed and NodeID is changed #317 - this is done here, however @tegefaulkes needs to create new issue for general keypair checks - be as comprehensive as possible
• Support read-after-write consistency for NodeGraph bucket operations #244 - this is waiting on SI to be done
• Testnet Node Deployment (testnet.polykey.io) #194, Testnet securely maintain a pool of recovery codes #285, Update testnet.polykey.io to point to the list of IPs running seed keynodes #177 are all procedures to be done when staging is ready to be used and docker containers are built (I still need to review the entire codebase in staging).