Skip to content
Jingyu Zhou edited this page Feb 1, 2023 · 3 revisions

Notes on failure injection workloads

Failure injection workloads are registered with FailureInjectorFactory, including:

  • MachineAttrition
  • DiskFailureInjection
  • RandomClogging
  • MoveKeys
  • Rollback

They can be enabled or disabled in the following ways:

  • Explicitly added in the toml file
  • Certain workloads can claim they dislike some/all of the failure injection workloads, e.g. LowLatency hates Attrition. (and this is defined in the workload file)
  • The workload itself can prevent its injection, base on the use of the database, the count of the existing same workloads, and/or a random number. (and this is defined in tester.actor.cpp, by default; or overridden in the actual workload .actor.cpp file)
  • A compound workload can also prevent creating certain type of workloads, by defining disabledFailureInjectionWorkloads in the toml file. See https://github.com/apple/foundationdb/pull/9240

Workloads

ApiCorrectness.actor.cpp

ApiWorkload.actor.cpp

AsyncFileCorrectness.actor.cpp

AsyncFileRead.actor.cpp

AsyncFileWrite.actor.cpp

AtomicOps.actor.cpp

Atomic operations.

AtomicOpsApiCorrectness.actor.cpp

AtomicRestore.actor.cpp

AtomicSwitchover.actor.cpp

BackgroundSelectors.actor.cpp

BackupAndParallelRestoreCorrectness.actor.cpp

Backup the database and use FastRestore to restore cluster data.

BackupCorrectness.actor.cpp

Backup the database and restore the cluster data.

BackupToBlob.actor.cpp

BackupToDBAbort.actor.cpp

BackupToDBCorrectness.actor.cpp

BackupToDBUpgrade.actor.cpp

BulkLoad.actor.cpp

Cache.actor.cpp

ChangeConfig.actor.cpp

ClientTransactionProfileCorrectness.actor.cpp

CommitBugCheck.actor.cpp

ConfigureDatabase.actor.cpp

ConflictRange.actor.cpp

ConsistencyCheck.actor.cpp

CpuProfiler.actor.cpp

Cycle test (Cycle.actor.cpp): failing with error "Cycle got longer/shorter"

This means there is data corruption in the DB. The cycle test has a loop: e.g., a->b->c->...->a, and the test random choose 5 nodes and switch their pointers, but keep the loop structure. In the end of the test, check the number of nodes remains the same.

DDBalance.actor.cpp

DDMetrics.actor.cpp

DDMetricsExclude.actor.cpp

DataDistributionMetrics.actor.cpp

DifferentClustersSameRV.actor.cpp

DiskDurability.actor.cpp

DiskDurabilityTest.actor.cpp

Downgrade.actor.cpp

DummyWorkload.actor.cpp

ExternalWorkload.actor.cpp

FastTriggeredWatches.actor.cpp

FileSystem.actor.cpp

FuzzApiCorrectness.actor.cpp

GetRangeStream.actor.cpp

HealthMetricsApi.actor.cpp

Increment.actor.cpp

IncrementalBackup.actor.cpp

IndexScan.actor.cpp

Inventory.actor.cpp

KVStoreTest.actor.cpp

KillRegion.actor.cpp

LocalRatekeeper.actor.cpp

LockDatabase.actor.cpp

LockDatabaseFrequently.actor.cpp

LogMetrics.actor.cpp

LowLatency.actor.cpp

MachineAttrition.actor.cpp

Mako.actor.cpp

MemoryLifetime.actor.cpp

MetricLogging.actor.cpp

ParallelRestore.actor.cpp

Performance.actor.cpp

Ping.actor.cpp

PopulateTPCC.actor.cpp

ProtocolVersion.actor.cpp

PubSubMultiples.actor.cpp

QueuePush.actor.cpp

RYWDisable.actor.cpp

RYWPerformance.actor.cpp

RandomClogging.actor.cpp

RandomMoveKeys.actor.cpp

RandomSelector.actor.cpp

ReadAfterWrite.actor.cpp

ReadHotDetection.actor.cpp

ReadWrite.actor.cpp

RemoveServersSafely.actor.cpp

ReportConflictingKeys.actor.cpp

RestoreBackup.actor.cpp

RestoreFromBlob.actor.cpp

Rollback.actor.cpp

RyowCorrectness.actor.cpp

SaveAndKill.actor.cpp

SelectorCorrectness.actor.cpp

Serializability.actor.cpp

Sideband.actor.cpp

SimpleAtomicAdd.actor.cpp

SlowTaskWorkload.actor.cpp

SnapTest.actor.cpp

SpecialKeySpaceCorrectness.actor.cpp

StatusWorkload.actor.cpp

Storefront.actor.cpp

StreamingRead.actor.cpp

SubmitBackup.actor.cpp

SuspendProcesses.actor.cpp

TPCC.actor.cpp

TagThrottleApi.actor.cpp

TargetedKill.actor.cpp

TaskBucketCorrectness.actor.cpp

ThreadSafety.actor.cpp

Throttling.actor.cpp

Throughput.actor.cpp

TimeKeeperCorrectness.actor.cpp

TriggerRecovery.actor.cpp

UDPWorkload.actor.cpp

UnitPerf.actor.cpp

UnitTests.actor.cpp

Unreadable.actor.cpp

VersionStamp.actor.cpp

WatchAndWait.actor.cpp

Watches.actor.cpp

WatchesSameKeyCorrectness.actor.cpp

WorkerErrors.actor.cpp

WriteBandwidth.actor.cpp

WriteDuringRead.actor.cpp

WriteTagThrottling.actor.cpp

Clone this wiki locally