Fix issue that allowed multiple simultaneous snapshots to be allowed

[brandond]

Proposed Changes

Replace 1-weight semaphore on snapshots with simple mutex

Fixes an issue where the semaphore wasn't permanently initialized until a scheduled snapshot was taken, allowing multiple on-demand snapshots to be taken until the first scheduled snapshot was triggered.

Types of Changes

bugfix

Verification

When fixed, you should see:

root@k3s-server-1:~# k3s etcd-snapshot save & k3s etcd-snapshot save; sleep 5
FATA[0000] see server log for details: Internal error occurred: etcd-snapshot error ID 40905
INFO[0000] Snapshot on-demand-k3s-server-1-1718755362 saved.
[1]+  Done(1)                    k3s etcd-snapshot save

server log:

INFO[0055] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-k3s-server-1-1718755362
{"level":"info","ts":"2024-06-19T00:02:41.544644Z","logger":"etcd-client","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-k3s-server-1-1718755362.part"}
{"level":"info","ts":"2024-06-19T00:02:41.546515Z","logger":"etcd-client.client","caller":"v3@v3.5.13-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-06-19T00:02:41.546559Z","logger":"etcd-client","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2024-06-19T00:02:41.546709Z","caller":"v3rpc/maintenance.go:126","msg":"sending database snapshot to client","total-bytes":2859008,"size":"2.9 MB"}
ERRO[0055] etcd-snapshot error ID 40905: snapshot save already in progress
ERRO[0055] Sending HTTP 500 response to 127.0.0.1:35044: etcd-snapshot error ID 40905
{"level":"info","ts":"2024-06-19T00:02:41.5617Z","caller":"v3rpc/maintenance.go:166","msg":"sending database sha256 checksum to client","total-bytes":2859008,"checksum-size":32}
{"level":"info","ts":"2024-06-19T00:02:41.561741Z","caller":"v3rpc/maintenance.go:175","msg":"successfully sent database snapshot to client","total-bytes":2859008,"size":"2.9 MB","took":"now"}
{"level":"info","ts":"2024-06-19T00:02:41.561852Z","logger":"etcd-client.client","caller":"v3@v3.5.13-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2024-06-19T00:02:41.571993Z","logger":"etcd-client","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"2.9 MB","took":"now"}
{"level":"info","ts":"2024-06-19T00:02:41.572068Z","logger":"etcd-client","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-k3s-server-1-1718755362"}
INFO[0055] Reconciling ETCDSnapshotFile resources
I0619 00:02:41.576558      54 event.go:389] "Event occurred" object="local-on-demand-k3s-server-1-1718755362-e95d32" fieldPath="" kind="ETCDSnapshotFile" apiVersion="k3s.cattle.io/v1" type="Normal" reason="ETCDSnapshotCreated" message="Snapshot on-demand-k3s-server-1-1718755362 saved on k3s-server-1"
INFO[0055] Reconciliation of ETCDSnapshotFile resources complete

Testing

Linked Issues

• Multiple simultaneous snapshots result in silent failure and/or corruption of at least one snapshot #10371

User-Facing Change

Further Comments

[codecov]

Codecov Report

Attention: Patch coverage is 20.00000% with 8 lines in your changes missing coverage. Please review.

Project coverage is 41.62%. Comparing base (b4d4ed8) to head (0a64ef9).

Files	Patch %	Lines
pkg/etcd/snapshot_handler.go	0.00%	5 Missing ⚠️
pkg/etcd/snapshot.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10372      +/-   ##
==========================================
- Coverage   47.97%   41.62%   -6.36%     
==========================================
  Files         177      177              
  Lines       14807    14802       -5     
==========================================
- Hits         7104     6161     -943     
- Misses       6363     7463    +1100     
+ Partials     1340     1178     -162     

Flag	Coverage Δ
e2etests	36.48% <20.00%> (-10.06%)	⬇️
inttests	19.96% <20.00%> (-17.10%)	⬇️
unittests	11.36% <20.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.