Demo Steps

Reenable DD

Status: We have added split points to the test TOML file to specify shards. When DB starts at the first generation, the master recruits each storage server for a unique team and we have disabled DD from creating new teams or moving shards. The one SS per team is due a limitation that a SS can only peek one team (mapped to one TLog group).

Goal: reenable DD, starts DB without split points With split points, we are assigning a shard to one SS, not respecting the replication factor.

Dependency:

• SS supports peeking from multiple teams
• The transaction system needs to handle adding a new team and removing an old team.
  1. Adding a new team: Proxy assigns the new team to a new tlog group, and should notify tlogs in that groups.
  2. Removing an old team: similarly, proxy should notify the TLogs of the corresponding group, and the notification can be done by adding information to TLogCommit Messages.
• SS becomes aware of such team assignment changes, which are private mutations delivered by a special team that only contains this SS.

Broadcast vs. Unicast, controlled by knob TLOG_NEW_INTERFACE:

• Merge cursor supports broadcast, i.e., each version is sent all TLog groups
• Merge cursor supports unicast: TLogCommit message needs to include information about how many teams each SS expects for a version

Remove split points in TOML file:

• Master recruits an initial team, DD creates more teams
• Create special team for each SS (can't use UID(0, 0) as teamID), which could be mapped from the SS tag

Demo Steps

TOML -> shards (PR #4843)

[configuration]
config = 'triple'
storageEngineType = 3
processesPerMachine = 3
coordinators = 3
machineCount = 15
splits = "a:b:c"

=> TestConfig => SimulationConfig => DatabaseConfiguration: shards

newSeedServers() add shard info into InitializeStorageRequest

storageServer() adds req.storageTeamId and req.shardInfo

modify storage server by adding req.shardInfo

Demo limitations

• Proxy broadcasts to all TLog groups due to private mutations (knob BROADCAST_TLOG_GROUPS=true).

  • Move private mutation generation to Resolvers PR 5969
  • Assign private mutations to TLog groups and track group versions at Resolvers. Then sending these private mutations to commit proxies along with TLogGroup's Previous Commit Versions (TPCVs). Note if there are shard changes, then we fall back to broadcast, because proxy's pre-computed (before resolution) TLog Group set is no longer accurate.

• 1 SS : 1 team (1 TLog group)

  • SS tag is mapped to a team at ApplyMetadataMutation: git commit
  • 1 SS : n teams -- needs merge cursor support

• Recruit SS with teamID, and shard info Team can be dynamically assigned

• 1 TLog group : 1 or many? TLog

Proxy -> TLog TLogCommitRequest req.reply.send(logData->durableKnownCommittedVersion);

Proxy

Proxy -> master ReportRawCommittedVersionRequest max()

Note "keySplits" was introduced for demo purpose and predefining splits can become a feature in the future, so that loading data into DB can be faster----without load balancing across storage servers later.

Seed

r -r simulation --crash --logsize 1024MB --knob_tlog_new_interface=1 -f src/foundationdb/tests/ptxn/CycleTest.toml -s 100 -b off

cmr | grep -E 'SSTeam' | s | less cmr | grep StorageMetrics | s | sed -e 's/Elapse.BytesInput=/Input=/' -e 's/BytesFetche.//' | less TestSetupComplete|TestComplete|CheckingResults|TestFetchingMetrics

Type="ServerTag" Machine="2.0.1.3:1:tls" ID="98995bbcca0a0877" Server="345ac520e5ba8396" Tag="0:4"