tsm: cache: race condition exists between compactor and backup

[jonseymour]

A second race condition exists in the implementation of the TSM cache system.

The issue exists because no lock exists that prevents two goroutines attempting to execute Engine.WriteSnapshots() at the same time. When this happens, the second thread to start executing Cache.Snapshots() can interfere with the first thread which is executing Cache.Deduplicate().

An example of how this could occur in real life is that the period compaction happens at the same time that a BACKUP is issued.

This race condition is implied by the lack of synchronization between two invocations of WriteSnapshot executing on different threads; it is demonstrated by a test case that will be submitted along with a proposed fix.

(A proposed fix for this was rolled up into #5689 but since that PR has been discard in favour of #5789, this issue has been opened to separately document, explain, demonstrate and fix this issue.)

The issue is easily fixed by introducing a mutex that guards the execution that occurs between Snapshot() and ClearSnapshot(),

[jonseymour]

Closed via #5807