roachtest: engine/switch/encrypted/nodes=3 failed #55362
Assignees
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
release-blocker
Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Milestone
Comments
cockroach-teamcity
added
branch-master
@jbowens This is the sibling test of the failure in #55361. This failure is more concerning. Is this possibly a problem on 20? |
|
(roachtest).engine/switch/nodes=3 failed on master@d816ce972c4d28fcde15925ebc04e7c3522dbf20: More
Artifacts: /engine/switch/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/encrypted/nodes=3 failed on master@d816ce972c4d28fcde15925ebc04e7c3522dbf20: More
Artifacts: /engine/switch/encrypted/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/encrypted/nodes=3 failed on master@e824da5a2b33168fa4ef93c83295c8205acfadb7: More
Artifacts: /engine/switch/encrypted/nodes=3 See this test on roachdash |
|
The latest failure looks like an infrastructural issue. But the previous one again had this disconcerting failure: @itsbilal Since Jackson is off this week, can you take a look at this? Specifically, I'm hoping you can verify that the problem reproduces and then see if it reproduces with the new WAL handling behavior disabled. |
|
For what it's worth, I tried reproducing this on the 20.2 branch and didn't reproduce it in 20 runs. On master it reproduces readily. I wonder if something else is going on. |
Oh, that's good to know. My main concern this week is whether this is a bug in 20.2. |
|
(roachtest).engine/switch/nodes=3 failed on master@9b97bda17396aabc4519b9fa272c39dfe5e851da: More
Artifacts: /engine/switch/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/encrypted/nodes=3 failed on master@9b97bda17396aabc4519b9fa272c39dfe5e851da: More
Artifacts: /engine/switch/encrypted/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/nodes=3 failed on master@ea42b57cbd348864b7d389cb1cf3cd70fff61bfa: More
Artifacts: /engine/switch/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/encrypted/nodes=3 failed on master@ea42b57cbd348864b7d389cb1cf3cd70fff61bfa: More
Artifacts: /engine/switch/encrypted/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/encrypted/nodes=3 failed on master@05826e3a748d3bf7634130d49667627e687d5875: More
Artifacts: /engine/switch/encrypted/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/nodes=3 failed on master@80e7127197f76ef35c1f6ec3984c4d49d4afde7f: More
Artifacts: /engine/switch/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/encrypted/nodes=3 failed on master@80e7127197f76ef35c1f6ec3984c4d49d4afde7f: More
Artifacts: /engine/switch/encrypted/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/encrypted/nodes=3 failed on master@47044feed11ec0c0390989bf8f44e777ec3eb00d: More
Artifacts: /engine/switch/encrypted/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/nodes=3 failed on master@47044feed11ec0c0390989bf8f44e777ec3eb00d: More
Artifacts: /engine/switch/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/encrypted/nodes=3 failed on master@b1abf9c8dfb5880fce69dfc7240e593f077bf77c: More
Artifacts: /engine/switch/encrypted/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/nodes=3 failed on master@b1abf9c8dfb5880fce69dfc7240e593f077bf77c: More
Artifacts: /engine/switch/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/encrypted/nodes=3 failed on master@d752fa2bd9afad255e8c655de9c7edc6dad14486: More
Artifacts: /engine/switch/encrypted/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/nodes=3 failed on master@d752fa2bd9afad255e8c655de9c7edc6dad14486: More
Artifacts: /engine/switch/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/encrypted/nodes=3 failed on master@ab503e2fd708541e5e9ebb9a6f2651eda506f2ef: More
Artifacts: /engine/switch/encrypted/nodes=3 See this test on roachdash |
|
(roachtest).engine/switch/nodes=3 failed on master@ab503e2fd708541e5e9ebb9a6f2651eda506f2ef: More
Artifacts: /engine/switch/nodes=3 See this test on roachdash |
itsbilal
added a commit
to itsbilal/cockroach
that referenced
this issue
Oct 19, 2020
Currently, doing direct mutations on a RocksDB instance bypasses custom batching / syncing logic that we've built on top of it. This, or something internal to RocksDB, started leading to some bugs when all direct mutations started passing in WriteOptions.sync = true (see cockroachdb#55240 for when this change went in). In this change, direct mutations still commit the batch with sync=true to guarantee WAL syncing, but they go through the batch commit pipeline too, just like the vast majority of operations already do. Fixes cockroachdb#55362. Release note: None.
itsbilal
added a commit
to itsbilal/cockroach
that referenced
this issue
Oct 19, 2020
Currently, doing direct mutations on a RocksDB instance bypasses custom batching / syncing logic that we've built on top of it. This, or something internal to RocksDB, started leading to some bugs when all direct mutations started passing in WriteOptions.sync = true (see cockroachdb#55240 for when this change went in). In this change, direct mutations still commit the batch with sync=true to guarantee WAL syncing, but they go through the batch commit pipeline too, just like the vast majority of operations already do. Fixes cockroachdb#55362. Release note: None.
This was referenced Oct 20, 2020
itsbilal
added a commit
to itsbilal/cockroach
that referenced
this issue
Oct 20, 2020
Currently, doing direct mutations on a RocksDB instance bypasses custom batching / syncing logic that we've built on top of it. This, or something internal to RocksDB, started leading to some bugs when all direct mutations started passing in WriteOptions.sync = true (see cockroachdb#55240 for when this change went in). In this change, direct mutations still commit the batch with sync=true to guarantee WAL syncing, but they go through the batch commit pipeline too, just like the vast majority of operations already do. Fixes cockroachdb#55362. Release note: None.
craig bot
pushed a commit
that referenced
this issue
Oct 20, 2020
55708: storage: Use batches for direct RocksDB mutations r=itsbilal a=itsbilal Currently, doing direct mutations on a RocksDB instance bypasses custom batching / syncing logic that we've built on top of it. This, or something internal to RocksDB, started leading to some bugs when all direct mutations started passing in WriteOptions.sync = true (see #55240 for when this change went in). In this change, direct mutations still commit the batch with sync=true to guarantee WAL syncing, but they go through the batch commit pipeline too, just like the vast majority of operations already do. Fixes #55362. Release note: None. 55734: server: skip unit'ed engines in tombstone storage r=irfansharif a=tbg Empty (i.e. uninitialized engines) could receive tombstones before being initialized. Initialization checks that the engine is empty (save for the cluster version) and thus failed. Simply don't write tombstones to uninitialized engines, which is fine since by the time the callbacks fire, at least one is initialized anyway, and besides, this mechanism is best effort. The alternatives would have been to either allow tombstones to be present on an engine that is being bootstrapped, or to give the storage the option to defer writing to the engine once it's bootstrapped. Neither seemed worth the extra work. Fixes #55576. Release note: None 55739: opt: fix normalization of st_distance when use_spheroid parameter used r=rytaft a=rytaft This commit fixes the normalization rule that converts `st_distance` to `st_dwithin` or `st_dwithinexclusive`, which was broken in the case when the `use_spheroid` parameter was used. Prior to this commit, the rule was assigning the `use_spheroid` parameter as the 3rd parameter to `st_dwithin` or `st_dwithinexclusive` and the `distance` parameter as the 4th, but that order does not match the function signatures. This commit fixes the issue by assigning `distance` as the 3rd parameter and `use_spheroid` as the 4th if it exists. Fixes #55675 Release note (bug fix): Fixed an internal error that could occur during query planning when the use_spheroid parameter was used in the ST_Distance function as part of a filter predicate. For example, `SELECT ... WHERE ST_Distance(geog1, geog2, false) < 10` previously caused an error. This has now been fixed. Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com> Co-authored-by: Tobias Grieger <tobias.b.grieger@gmail.com> Co-authored-by: Rebecca Taft <becca@cockroachlabs.com>
itsbilal
added a commit
to itsbilal/cockroach
that referenced
this issue
Nov 11, 2020
Currently, doing direct mutations on a RocksDB instance bypasses custom batching / syncing logic that we've built on top of it. This, or something internal to RocksDB, started leading to some bugs when all direct mutations started passing in WriteOptions.sync = true (see cockroachdb#55240 for when this change went in). In this change, direct mutations still commit the batch with sync=true to guarantee WAL syncing, but they go through the batch commit pipeline too, just like the vast majority of operations already do. Fixes cockroachdb#55362. Release note: None.
itsbilal
added a commit
to itsbilal/cockroach
that referenced
this issue
Nov 11, 2020
Currently, doing direct mutations on a RocksDB instance bypasses custom batching / syncing logic that we've built on top of it. This, or something internal to RocksDB, started leading to some bugs when all direct mutations started passing in WriteOptions.sync = true (see cockroachdb#55240 for when this change went in). In this change, direct mutations still commit the batch with sync=true to guarantee WAL syncing, but they go through the batch commit pipeline too, just like the vast majority of operations already do. Fixes cockroachdb#55362. Release note: None.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(roachtest).engine/switch/encrypted/nodes=3 failed on master@7f02a7ac0949f226d617955337965f681706bace:
More
Artifacts: /engine/switch/encrypted/nodes=3
See this test on roachdash
powered by pkg/cmd/internal/issues
The text was updated successfully, but these errors were encountered: