WIP - Improved HA stability #9854
Commits on Oct 31, 2022
-
Detect distributed setup early in activate process.
Split loading of enabled server plugins and starting of plugins to allow presence of a distributed server manager to be detected prior to network listeners being established and storage open. This allows guarding of constructs that require the distributed plugin to be present and running, which currently experience a race condition between the network listeners starting and the distributed plugin fully starting.
-
-
Add distinct openInternal operation that bypasses online checks.
The current usages of openNoAuthenticate include cases (like DB delta/full syncs) that need to bypass not only auth checks but distributed online status.
-
-
-
-
Add tracing support to executors.
Errors in unbound tasks in executors that are launched from common points (e.g. OrientDBEmbedded#execute) are hard to trace. This change allows a task ID to be associated with each execution, which will be reported on any exception, and if debug logging is enabled, a full stack trace identifying the launching call site will be attached.
-
-
Add a global executor for use instead of general one-off threads.
This allows improved logging and tracing consistency over general use of new Thread()
-
Defer setting running to end of distributed plugin startup.
This prevents storage tasks that require the distributed status to be online from accessing distributed lifecycle objects that have not yet been set up (which shows up as NPEs during execution).
-
Defer installation of databases until distributed plugin is online.
This avoids accesses to uninitialised distributed state during initial database setup from cluster.
-
Fix waiting for last task in ViewManager close.
Prefer attempting to cancel task before execution before waiting. Also removes double logging of execution exception, and avoids problem where get cannot be called after cancel.
-
Use tracing executor service in OrientDBEmbedded
Provide tracing overrides to aid in tracking async errors.
-
Use internal open for ViewManager init
Allows registering live updates to succeed when distributed plugin not online.
-
Make ViewManager updates resilient to offline DB status.
View update uses distributed state, which can break if view update occurs during a distributed state change, breaking the update loop.
-
-
Move DistributedDatabase registration out of constructor.
ODistributedDatabaseImpl construction registered the instance, leaking the this reference, and shut down the previous instance if present. The previous instance may not have been constructed fully however, so shutdown could NPE, resulting in the construction of the current instance aborting with uninitialised state, which would then be picked up by other threads finding it registered in the message service. This change externalises the construction into an atomic operation in the message service, and makes the state in the distributed database impl final. The warning about needing registration because of use "further in the call chain" appears to be spurious.
-
-
-
-
Guard database create to avoid partially initialised storage being cr…
…eated. If the plugin isn't online, initialisation of newly created database will fail, resulting in a partially initialised database that will break when used (usually because the schema hasn't been loaded).
-
DEV: Expand HazelcastPlugin startup time to widen window to expose ra…
…ce conditions on startup.
