*: keep tombstone if revision == compactAtRev #18274

fuweid · 2024-07-03T10:33:18Z

Before this patch, the tombstone can be deleted if its revision is equal compacted revision. It causes that the watch subscriber won't get this DELETE event. Based on Compact API [1], we should keep tombstone revision if it's not less than the compaction revision.

CompactionRequest compacts the key-value store up to a given revision.
All superseded keys with a revision less than the compaction revision
will be removed.

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

To Reviewers:

During writing tests, I found that the compact doesn't always remove smaller revision data.
For example,

➜  bin/etcdctl put hello world -w json
{"header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":2,"raft_term":2}}

➜  bin/etcdctl put hello-v2 world -w json
{"header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":3,"raft_term":2}}

➜  bin/etcdctl put hello world-v2 -w json
{"header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":4,"raft_term":2}}

➜  bin/etcdctl get --rev 2 hello
hello
world

➜  bin/etcdctl compact 3 --physical=true
compacted revision 3

➜  bin/etcdctl get --rev 2 hello
{"level":"warn","ts":"2024-07-03T20:12:39.086991+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002105a0/127.0.0.1:2379","method":"/etcdserverpb.KV/Range","attempt":0,"error":"rpc error: code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"}
Error: etcdserver: mvcc: required revision has been compacted

➜  bin/tools/etcd-dump-db iterate-bucket ./default.etcd/member/snap/db key --decode
rev={Revision:{Main:4 Sub:0} tombstone:false}, value=[key "hello" | val "world-v2" | created 2 | mod 4 | ver 2]
rev={Revision:{Main:3 Sub:0} tombstone:false}, value=[key "hello-v2" | val "world" | created 3 | mod 3 | ver 1]
rev={Revision:{Main:2 Sub:0} tombstone:false}, value=[key "hello" | val "world" | created 2 | mod 2 | ver 1]

Not sure we should fix it in the follow-up. Just want to highlight it.

k8s-ci-robot · 2024-07-03T10:33:20Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

fuweid · 2024-07-03T12:17:05Z

ping @ahrtr @serathius @siyuanfoundation @chaochn47 ~ PTAL, thanks

ahrtr · 2024-07-03T13:14:11Z

➜  bin/tools/etcd-dump-db iterate-bucket ./default.etcd/member/snap/db key --decode
rev={Revision:{Main:4 Sub:0} tombstone:false}, value=[key "hello" | val "world-v2" | created 2 | mod 4 | ver 2]
rev={Revision:{Main:3 Sub:0} tombstone:false}, value=[key "hello-v2" | val "world" | created 3 | mod 3 | ver 1]
rev={Revision:{Main:2 Sub:0} tombstone:false}, value=[key "hello" | val "world" | created 2 | mod 2 | ver 1]

Not sure we should fix it in the follow-up. Just want to highlight it.

I think it's expected behavior. Compaction always keeps the latest revision K, which is <= compaction_rev.

ahrtr · 2024-07-03T13:56:13Z

The e2e test cases, especially the mix-version test cases, as mentioned in #18162 (comment) are important. What's the plan to implement the e2e test cases?

fuweid · 2024-07-03T14:23:24Z

I think it's expected behavior. Compaction always keeps the latest revision K, which is <= compaction_rev.

ETCD server prevents requests from fetching revisions which are smaller than compacted revision. It looks fine to me.
However, based on the compact API, if the revision is not latest and smaller than the compaction revision, that revision should be deleted.

The e2e test cases, especially the mix-version test cases, as mentioned in #18162 (comment) are important.

For the mix-version part, it's hard to finish it in one request. I'm trying to use patch mode in this pull request, like https://github.com/etcd-io/etcd/blob/main/tests/robustness/patches/beforeSendWatchResponse/watch.patch.

ahrtr · 2024-07-03T15:39:09Z

ETCD server prevents requests from fetching revisions which are smaller than compacted revision. It looks fine to me.
However, based on the compact API, if the revision is not latest and smaller than the compaction revision, that revision should be deleted

It sounds reasonable, but we still have two options, either keep it as it's or fix it.

The rev, which is smaller than compaction rev, isn't accessible via the get/range API, but it's still in the db file. I agree it's an issue, but it seems harmless. Fixing it may break user experience as well.

If we decide to fix it (let's go for this direction if there is NO objection),

we should fix it in one PR instead of two.
evaluate the effort & impact.

For the mix-version part, it's hard to finish it in one request.

It's OK to do it in followup PR, but we should manually verify it before we merge this PR. Please also feedback if you need help from others.

server/storage/mvcc/key_index.go

server/storage/mvcc/index_test.go

fuweid · 2024-07-04T10:08:51Z

ping @ahrtr I updated the pull request with new e2e cases. The TestVerifyHashKVAfterCompactMixVersion is to test this change in a mix-version cluster (1 x main process + 2 x v3.5.14 process). One case will cause inconsistent hash value and I left comment in the case. Please take a look when you have time. Thanks

client/pkg/testutil/leak.go

server/storage/mvcc/key_index.go

server/storage/mvcc/key_index_test.go

tests/e2e/watch_test.go

ahrtr · 2024-07-10T12:04:24Z

Please put all hashKV related e2e cases into a separate file, e.g. hashkv_test.go

ahrtr · 2024-07-11T15:08:07Z

Discussed with @fuweid today. The existing implementation of HashByRev and Compaction are tightly coupled, and also they don't meet the expected behavior (see below).

The draft expected behavior is recorded in the doc. See summary below as well. Please comment in the doc to ensure we have consensus.

HashByRev (rev)

All revisions up to $(rev) should be included/calculated. For example, HashByRev(9), all revisions, which <=9, should be included/calculated.
All members should always return the same hash, no matter if it's a mix-version cluster or not.

Concern
Compatibility is a big concern. it's hard to ensure all members always return the same hash in a mix-version cluster.

Solution: use cluster-level feature-gate to resolve this concern. The fix/change will not be applied until all members have enabled the feature.

Compaction

When compacting revision X,

compact all history up to X (exclusive), but we always keep the revision X.
For all revisions which are < X, always keep the latest non-tombstone revision for each key.

It means that users can start to read Revision X and watch starting from X after the compaction. Reading or Watching X-1 will get an ErrCompacted error.

For example, compact (9),

Revision 9 will be kept;
For all revisions, which < 9
- If it’s the latest non-tombstone revision for a key, then keep it;
- Otherwise, compact the revisions

Next step

Resolve *: keep tombstone if revision == compactAtRev #18274 (comment) in a separate PR;
Ensure we have consensus on the expected behavior. Please comment on the doc;
Make change per the agreed expected behavior, but it should can be enabled or disabled by a flag. Once the cluster-level feature is ready, migrate the flag to the cluster-level feature.

fuweid · 2024-08-09T08:42:39Z

Can we close/mark as addressed all the comments opened about the code?

Done. PTAL，thanks

serathius · 2024-08-15T13:34:10Z

tests/e2e/watch_test.go

+	})
+}
+
+func testStartWatcherFromCompactedRevision(t *testing.T, performCompactOnTombstone bool) {


This function would benefit from splitting it up and cleanup, but let's not block fix on test.

ahrtr

LGTM

Nice work! @fuweid

k8s-ci-robot · 2024-08-16T07:16:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, fuweid, serathius

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahrtr,serathius]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ahrtr · 2024-08-16T07:16:56Z

@fuweid could you backport #18369 and this PR to 3.5 and 3.4? thx

fuweid · 2024-08-16T12:50:26Z

@fuweid could you backport #18369 and this PR to 3.5 and 3.4? thx

@ahrtr Yes, I will do the backport. Thanks for the review! @ahrtr @serathius

ahrtr · 2024-12-17T11:04:48Z

I see the failure below in
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/etcd-io_etcd/19077/pull-etcd-e2e-amd64/1868930443156393984 .

But unfortunately, I couldn't reproduce it locally. I also manually verified the steps, and couldn't reproduce it.

/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd (TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-2) (61865): {"level":"info","ts":"2024-12-17T08:23:04.293653Z","caller":"embed/serve.go:258","msg":"serving client traffic securely","traffic":"grpc+http","address":"127.0.0.1:20010"}
    hashkv_test.go:210: Writing keys: [key0], value: 1
    hashkv_test.go:210: Writing keys: [key0], value: 2
    hashkv_test.go:210: Writing keys: [key0], value: 3
    hashkv_test.go:210: Writing keys: [key0], value: 4
    hashkv_test.go:210: Writing keys: [key0], value: 5
    hashkv_test.go:210: Writing keys: [key0], value: 6
    hashkv_test.go:210: Writing keys: [key0], value: 7
    hashkv_test.go:210: Writing keys: [key0], value: 8
    hashkv_test.go:210: Writing keys: [key0], value: 9
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd (TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-2) (61865): {"level":"info","ts":"2024-12-17T08:23:04.324070Z","caller":"etcdserver/server.go:518","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"72e86eb5e356a737","forward-ticks":8,"forward-duration":"800ms","election-ticks":10,"election-timeout":"1s","active-remote-members":2}
    hashkv_test.go:196: Deleting key=key0
    hashkv_test.go:210: Writing keys: [key0], value: 11
    hashkv_test.go:210: Writing keys: [key0], value: 12
    hashkv_test.go:210: Writing keys: [key0], value: 13
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd-last-release (TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-1) (61866): {"level":"info","ts":"2024-12-17T08:23:04.332Z","caller":"etcdserver/server.go:767","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"5ac8aa22f1eb4c8f","forward-ticks":8,"forward-duration":"800ms","election-ticks":10,"election-timeout":"1s","active-remote-members":2}
    hashkv_test.go:210: Writing keys: [key0], value: 14
    hashkv_test.go:210: Writing keys: [key0], value: 15
    hashkv_test.go:210: Writing keys: [key0], value: 16
    hashkv_test.go:210: Writing keys: [key0], value: 17
    hashkv_test.go:210: Writing keys: [key0], value: 18
    hashkv_test.go:210: Writing keys: [key0], value: 19
    hashkv_test.go:196: Deleting key=key0
    hashkv_test.go:210: Writing keys: [key0], value: 21
    hashkv_test.go:210: Writing keys: [key0], value: 22
    hashkv_test.go:210: Writing keys: [key0], value: 23
    hashkv_test.go:210: Writing keys: [key0], value: 24
    hashkv_test.go:210: Writing keys: [key0], value: 25
    hashkv_test.go:210: Writing keys: [key0], value: 26
    hashkv_test.go:210: Writing keys: [key0], value: 27
    hashkv_test.go:210: Writing keys: [key0], value: 28
    hashkv_test.go:210: Writing keys: [key0], value: 29
    hashkv_test.go:196: Deleting key=key0
    hashkv_test.go:210: Writing keys: [key0], value: 31
    hashkv_test.go:210: Writing keys: [key0], value: 32
    hashkv_test.go:210: Writing keys: [key0], value: 33
    hashkv_test.go:210: Writing keys: [key0], value: 34
    hashkv_test.go:210: Writing keys: [key0], value: 35
    hashkv_test.go:210: Writing keys: [key0], value: 36
    hashkv_test.go:210: Writing keys: [key0], value: 37
    hashkv_test.go:210: Writing keys: [key0], value: 38
    hashkv_test.go:210: Writing keys: [key0], value: 39
    hashkv_test.go:196: Deleting key=key0
    hashkv_test.go:91: COMPACT on rev=16
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd-last-release (TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-0) (61864): {"level":"info","ts":"2024-12-17T08:23:04.411Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":16}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd-last-release (TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-0) (61864): {"level":"info","ts":"2024-12-17T08:23:04.411Z","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":16,"took":"66.101µs"}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd-last-release (TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-1) (61866): {"level":"info","ts":"2024-12-17T08:23:04.411Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":16}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd-last-release (TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-1) (61866): {"level":"info","ts":"2024-12-17T08:23:04.411Z","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":16,"took":"59.131µs"}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd (TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-2) (61865): {"level":"info","ts":"2024-12-17T08:23:04.411560Z","caller":"mvcc/index.go:194","msg":"compact tree index","revision":16}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd (TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-2) (61865): {"level":"info","ts":"2024-12-17T08:23:04.412101Z","caller":"mvcc/kvstore_compaction.go:71","msg":"finished scheduled compaction","compact-revision":16,"took":"454.515µs","hash":2489252793,"current-db-size-bytes":24576,"current-db-size":"25 kB","current-db-size-in-use-bytes":24576,"current-db-size-in-use":"25 kB"}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd (TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-2) (61865): {"level":"info","ts":"2024-12-17T08:23:04.412168Z","caller":"mvcc/hash.go:157","msg":"storing new hash","hash":2489252793,"revision":16,"compact-revision":-1}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--cacert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/ca.crt_--cert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.crt_--key=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.key.insecure_--endpoints=https://localhost:20000,https://localhost:20005,https://localhost:20010_compact_16_--physical) (61917): compacted revision 16
    hashkv_test.go:223: HashKV on rev=16
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--cacert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/ca.crt_--cert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.crt_--key=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.key.insecure_--endpoints=https://localhost:20000,https://localhost:20005,https://localhost:20010_endpoint_hashkv_--rev_16_-w_json) (61928): {"level":"warn","ts":"2024-12-17T08:23:04.442409Z","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000344780/localhost:20000","method":"/etcdserverpb.Maintenance/HashKV","attempt":0,"error":"rpc error: code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--cacert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/ca.crt_--cert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.crt_--key=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.key.insecure_--endpoints=https://localhost:20000,https://localhost:20005,https://localhost:20010_endpoint_hashkv_--rev_16_-w_json) (61928): Failed to get the hash of endpoint https://localhost:20000 (etcdserver: mvcc: required revision has been compacted)
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--cacert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/ca.crt_--cert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.crt_--key=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.key.insecure_--endpoints=https://localhost:20000,https://localhost:20005,https://localhost:20010_endpoint_hashkv_--rev_16_-w_json) (61928): {"level":"warn","ts":"2024-12-17T08:23:04.458105Z","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0003c01e0/localhost:20005","method":"/etcdserverpb.Maintenance/HashKV","attempt":0,"error":"rpc error: code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--cacert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/ca.crt_--cert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.crt_--key=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.key.insecure_--endpoints=https://localhost:20000,https://localhost:20005,https://localhost:20010_endpoint_hashkv_--rev_16_-w_json) (61928): Failed to get the hash of endpoint https://localhost:20005 (etcdserver: mvcc: required revision has been compacted)
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--cacert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/ca.crt_--cert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.crt_--key=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.key.insecure_--endpoints=https://localhost:20000,https://localhost:20005,https://localhost:20010_endpoint_hashkv_--rev_16_-w_json) (61928): [{"Endpoint":"https://localhost:20010","HashKV":{"header":{"cluster_id":16074511605466142234,"member_id":8279989642402899767,"revision":41,"raft_term":2},"hash":2489252793,"compact_revision":-1,"hash_revision":16}}]
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--cacert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/ca.crt_--cert=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.crt_--key=/home/prow/go/src/github.com/etcd-io/etcd/tests/fixtures/server.key.insecure_--endpoints=https://localhost:20000,https://localhost:20005,https://localhost:20010_endpoint_hashkv_--rev_16_-w_json) (61928): Error: etcdserver: mvcc: required revision has been compacted
    hashkv_test.go:227: 
        	Error Trace:	/home/prow/go/src/github.com/etcd-io/etcd/tests/e2e/hashkv_test.go:227
        	            				/home/prow/go/src/github.com/etcd-io/etcd/tests/e2e/hashkv_test.go:96
        	Error:      	"1" is not greater than "1"
        	Test:       	TestVerifyHashKVAfterCompact/compactedOnTombstone=false_-_quorum-last-version_-_Keys=[key0]
    logger.go:146: 2024-12-17T08:23:04.480Z	INFO	closing test cluster...
    logger.go:146: 2024-12-17T08:23:04.480Z	INFO	stopping server...	{"name": "TestVerifyHashKVAfterCompactcompactedOnTombstonefalse_-_quorum-last-version_-_Keyskey0-test-0"}

fuweid · 2024-12-17T16:40:28Z

ping @ahrtr

I think it's related to github service. The test was using v3.5.0 which is not expected.

'release' started at Tue Dec 17 08:08:05 UTC 2024
fatal: unable to access 'https://github.com/etcd-io/etcd.git/': The requested URL returned error: 503
Found latest release: .
              
fallback to v3.5.0
Downloading etcd-v3.5.0-linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

 76 18.4M   76 14.1M    0     0  22.1M      0 --:--:-- --:--:-- --:--:-- 22.1M
100 18.4M  100 18.4M    0     0  27.1M      0 --:--:-- --:--:-- --:--:--  101M
etcd-v3.5.0-linux-amd64/Documentation/
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/apispec/
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/apispec/swagger/
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/apispec/swagger/v3lock.swagger.json
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/apispec/swagger/v3election.swagger.json
etcd-v3.5.0-linux-amd64/Documentation/dev-guide/apispec/swagger/rpc.swagger.json
etcd-v3.5.0-linux-amd64/Documentation/README.md
etcd-v3.5.0-linux-amd64/README-etcdutl.md
etcd-v3.5.0-linux-amd64/READMEv2-etcdctl.md
etcd-v3.5.0-linux-amd64/README-etcdctl.md
etcd-v3.5.0-linux-amd64/README.md
etcd-v3.5.0-linux-amd64/etcdutl
etcd-v3.5.0-linux-amd64/etcdctl
etcd-v3.5.0-linux-amd64/etcd
'release' PASSED and completed at Tue Dec 17 08:08:07 UTC 2024

The following script can reproduce this.

EXPECT_DEBUG=true MANUAL_VER=v3.5.0 TESTCASE='TestVerifyHashKVAfterCompactionOnLastTombstone_MixVersions' GO_TEST_FLAGS="-v -count=1" make test-e2e-release

ahrtr · 2024-12-17T16:51:07Z

I think it's related to github service. The test was using v3.5.0 which is not expected.

script issue?

etcd/scripts/test.sh

Lines 598 to 599 in b4450d5

    
           UPGRADE_VER="v3.5.0" 
        
           log_warning "fallback to" ${UPGRADE_VER}

fuweid · 2024-12-17T16:56:30Z

It's github service issue.

'release' started at Tue Dec 17 08:08:05 UTC 2024
fatal: unable to access 'https://github.com/etcd-io/etcd.git/': The requested URL returned error: 503
Found latest release: .

The git service returns 503 so that we fallback to v3.5.0

ahrtr · 2024-12-17T17:27:16Z

I know that github wasn't stable at that time. I meant that we should fail the workflow check immediately so that it's well aware that it's github issue. cc @ivanvc

fuweid · 2024-12-17T20:23:14Z

+1 to abort test if there is something wrong during fetching latest version

k8s-ci-robot added do-not-merge/work-in-progress area/testing size/XL labels Jul 3, 2024

fuweid force-pushed the fix-event-loss-caused-by-compact branch from bb203a1 to a89b240 Compare July 3, 2024 10:35

fuweid marked this pull request as ready for review July 3, 2024 12:08

k8s-ci-robot removed the do-not-merge/work-in-progress label Jul 3, 2024

ahrtr self-requested a review July 3, 2024 12:48

siyuanfoundation reviewed Jul 3, 2024

View reviewed changes

server/storage/mvcc/key_index.go Outdated Show resolved Hide resolved

server/storage/mvcc/key_index.go Outdated Show resolved Hide resolved

server/storage/mvcc/key_index.go Outdated Show resolved Hide resolved

siyuanfoundation reviewed Jul 3, 2024

View reviewed changes

server/storage/mvcc/index_test.go Show resolved Hide resolved

fuweid force-pushed the fix-event-loss-caused-by-compact branch 4 times, most recently from abdda43 to 2f0a9e1 Compare July 4, 2024 10:01

fuweid commented Jul 4, 2024

View reviewed changes

client/pkg/testutil/leak.go Outdated Show resolved Hide resolved

fuweid force-pushed the fix-event-loss-caused-by-compact branch from 2f0a9e1 to 1143e2a Compare July 5, 2024 14:22

fuweid mentioned this pull request Jul 5, 2024

client/pkg/testutil: update interestingGoroutines #18287

Merged

ahrtr reviewed Jul 8, 2024

View reviewed changes

server/storage/mvcc/key_index.go Outdated Show resolved Hide resolved

server/storage/mvcc/key_index.go Show resolved Hide resolved

ahrtr reviewed Jul 8, 2024

View reviewed changes

server/storage/mvcc/key_index.go Outdated Show resolved Hide resolved

fuweid force-pushed the fix-event-loss-caused-by-compact branch from 1143e2a to c19beea Compare July 9, 2024 15:15

ahrtr reviewed Jul 10, 2024

View reviewed changes

server/storage/mvcc/key_index_test.go Outdated Show resolved Hide resolved

ahrtr reviewed Jul 10, 2024

View reviewed changes

serathius approved these changes Aug 15, 2024

View reviewed changes

k8s-ci-robot added the approved label Aug 15, 2024

serathius approved these changes Aug 15, 2024

View reviewed changes

serathius reviewed Aug 15, 2024

View reviewed changes

ahrtr approved these changes Aug 16, 2024

View reviewed changes

ahrtr merged commit 1d0a6b3 into etcd-io:main Aug 16, 2024
45 checks passed

ahrtr added backport/v3.4 backport/v3.5 labels Aug 16, 2024

fuweid deleted the fix-event-loss-caused-by-compact branch August 16, 2024 12:50

ahrtr mentioned this pull request Jan 13, 2025

Missing delete event on watch opened on same revision as compaction request #19179

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: keep tombstone if revision == compactAtRev #18274

*: keep tombstone if revision == compactAtRev #18274

fuweid commented Jul 3, 2024 •

edited

Loading

k8s-ci-robot commented Jul 3, 2024

fuweid commented Jul 3, 2024

ahrtr commented Jul 3, 2024

ahrtr commented Jul 3, 2024

fuweid commented Jul 3, 2024

ahrtr commented Jul 3, 2024 •

edited

Loading

fuweid commented Jul 4, 2024

ahrtr commented Jul 10, 2024

ahrtr commented Jul 11, 2024

fuweid commented Aug 9, 2024

serathius Aug 15, 2024

ahrtr left a comment

k8s-ci-robot commented Aug 16, 2024

ahrtr commented Aug 16, 2024

fuweid commented Aug 16, 2024

ahrtr commented Dec 17, 2024

fuweid commented Dec 17, 2024 •

edited

Loading

ahrtr commented Dec 17, 2024

fuweid commented Dec 17, 2024

ahrtr commented Dec 17, 2024

fuweid commented Dec 17, 2024

*: keep tombstone if revision == compactAtRev #18274

*: keep tombstone if revision == compactAtRev #18274

Conversation

fuweid commented Jul 3, 2024 • edited Loading

k8s-ci-robot commented Jul 3, 2024

fuweid commented Jul 3, 2024

ahrtr commented Jul 3, 2024

ahrtr commented Jul 3, 2024

fuweid commented Jul 3, 2024

ahrtr commented Jul 3, 2024 • edited Loading

fuweid commented Jul 4, 2024

ahrtr commented Jul 10, 2024

ahrtr commented Jul 11, 2024

HashByRev (rev)

Compaction

Next step

fuweid commented Aug 9, 2024

serathius Aug 15, 2024

Choose a reason for hiding this comment

ahrtr left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 16, 2024

ahrtr commented Aug 16, 2024

fuweid commented Aug 16, 2024

ahrtr commented Dec 17, 2024

fuweid commented Dec 17, 2024 • edited Loading

ahrtr commented Dec 17, 2024

fuweid commented Dec 17, 2024

ahrtr commented Dec 17, 2024

fuweid commented Dec 17, 2024

fuweid commented Jul 3, 2024 •

edited

Loading

ahrtr commented Jul 3, 2024 •

edited

Loading

fuweid commented Dec 17, 2024 •

edited

Loading