mc mirror claims objects copied under network packet loss

Connect to source/client

ssh -p 20431 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Connect to target

ssh -p 20673 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Create data for testing

mc ls source/
mc rb --force source/simple
mc mb source/simple
mc rb --force target/simple
mc mb target/simple
mc version enable source/simple
mc version enable target/simple

rm -rf ~/simple && mkdir ~/simple && cd ~/simple
dd if=/dev/urandom bs=10M count=4 iflag=fullblock of=object
split -b 10MiB ~/source/object object

Save a backup

mc cp --recursive ~/simple source/simple/original
mc mirror --json source/simple target/simple/original

Make multiple versions

for i in {1..10}; do mc cp --recursive ~/simple source/simple; done

Repeat as needed. Observe that mc failures and claims that objects were copied. This is not the case.

In terminal 1

sudo lxc exec mc-mirror-source -- sudo tc qdisc add dev eth0 root netem loss 90% && sleep 10 && sudo lxc exec mc-mirror-source -- sudo tc qdisc del dev eth0 root

Immediately in terminal 2

mc mirror --json source/simple target/simple/copy

Verify if objects were copied

mc ls --recursive --summarize --versions source/simple/original | grep "Total"
mc ls --recursive --summarize --versions target/simple/original | grep "Total"
mc ls --recursive --summarize --versions target/simple/copy | grep "Total"