sql/rowflow: TestRouterDiskSpill failed #55848

cockroach-teamcity · 2020-10-22T06:26:34Z

(sql/rowflow).TestRouterDiskSpill failed on master@4019977188f67504f39422997c32408150d4a88d:

Fatal error:

panic: test-monitor: unexpected 8 leftover bytes [recovered]
	panic: test-monitor: unexpected 8 leftover bytes [recovered]
	panic: test-monitor: unexpected 8 leftover bytes

Stack:

goroutine 199 [running]:
testing.tRunner.func1(0xc000324700)
	/usr/local/go/src/testing/testing.go:874 +0x69f
panic(0x32124c0, 0xc00098c100)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:103 +0x2ae
panic(0x32124c0, 0xc00098c100)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/log.ReportOrPanic(0x4921ea0, 0xc0003be3c0, 0xc00053b500, 0x33a6a10, 0x20, 0xc0007eeec8, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/crash_reporting.go:340 +0x3ff
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop(0xc000263540, 0x4921ea0, 0xc0003be3c0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:406 +0x475
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).Stop(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391
github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*EvalContext).Stop(0xc000304240, 0x4921ea0, 0xc0003be3c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/eval.go:3344 +0x70
runtime.Goexit()
	/usr/local/go/src/runtime/panic.go:563 +0xec
testing.(*common).FailNow(0xc000324700)
	/usr/local/go/src/testing/testing.go:653 +0x5b
testing.(*common).Fatalf(0xc000324700, 0x33bde47, 0x29, 0x0, 0x0, 0x0)
	/usr/local/go/src/testing/testing.go:716 +0x9e
github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill(0xc000324700)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:919 +0x1807
testing.tRunner(0xc000324700, 0x38669d8)
	/usr/local/go/src/testing/testing.go:909 +0x19a
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:960 +0x652

Log preceding fatal error

  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:406 +0x474
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*EvalContext).Stop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391 +0x6f
  runtime.call32()
      /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a
  testing.(*common).Fatalf()
      /usr/local/go/src/testing/testing.go:716 +0x9d
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:919 +0x1806
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Previous write at 0x00c00010e290 by goroutine 82:
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).releaseBytes()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:663 +0xfe
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Close()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:534 +0xc8
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).releaseBudget()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:521 +0xba
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:413 +0xbb
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*routerBase).Start.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391 +0x580

Goroutine 72 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:960 +0x651
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1202 +0xa6
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1200 +0x521
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1117 +0x2ff
  main.main()
      _testmain.go:64 +0x223

Goroutine 82 (finished) created at:
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*routerBase).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers.go:301 +0xcd
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:826 +0x15fc
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
--- FAIL: TestRouterDiskSpill (0.02s)
    routers_test.go:919: expected memory error when consuming rows
    testing.go:853: race detected during execution of test

More

Parameters:

TAGS=
GOFLAGS=-race -parallel=2

make stressrace TESTS=TestRouterDiskSpill PKG=./pkg/sql/rowflow TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

yuzefovich · 2020-10-22T21:52:07Z

#46930 is the culprit. The test fails reliably under stress, I'll adjust it.

cockroach-teamcity · 2020-10-24T17:20:24Z

(sql/rowflow).TestRouterDiskSpill failed on master@7fa301fe1879b8af0249238e3dfd97502a1c694d:

WARNING: DATA RACE
Write at 0x00c0000f7550 by goroutine 81:
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).releaseBytes()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:663 +0xfe
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Close()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:534 +0xc8
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).releaseBudget()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:521 +0xba
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:413 +0xbb
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*routerBase).Start.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391 +0x531

Previous read at 0x00c0000f7550 by goroutine 74:
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:405 +0x521
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*EvalContext).Stop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391 +0x6f
  runtime.call32()
      /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a
  testing.(*common).Fatalf()
      /usr/local/go/src/testing/testing.go:716 +0x9d
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:919 +0x1874
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Goroutine 81 (running) created at:
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*routerBase).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers.go:301 +0xcd
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:826 +0x166a
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Goroutine 74 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:960 +0x651
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1202 +0xa6
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1200 +0x521
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1117 +0x2ff
  main.main()
      _testmain.go:64 +0x223
==================
FAIL	github.com/cockroachdb/cockroach/pkg/sql/rowflow	0.424s

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestRouterDiskSpill PKG=./pkg/sql/rowflow TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-10-25T06:19:20Z

(sql/rowflow).TestRouterDiskSpill failed on master@8aceac3c99c3addece3a9ef9af04cc74715cdb37:

Fatal error:

panic: test-monitor: unexpected 82 leftover bytes [recovered]
	panic: test-monitor: unexpected 82 leftover bytes [recovered]
	panic: test-monitor: unexpected 82 leftover bytes

Stack:

goroutine 190 [running]:
testing.tRunner.func1(0xc0004c8d00)
	/usr/local/go/src/testing/testing.go:874 +0x3a3
panic(0x1e1a8a0, 0xc000912340)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:103 +0x22f
panic(0x1e1a8a0, 0xc000912340)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/log.ReportOrPanic(0x29d6e80, 0xc000719740, 0xc000691500, 0x1fac107, 0x20, 0xc000743138, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/crash_reporting.go:340 +0x212
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop(0xc000526b40, 0x29d6e80, 0xc000719740, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:406 +0x2ca
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).Stop(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391
github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*EvalContext).Stop(0xc00010b8c0, 0x29d6e80, 0xc000719740)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/eval.go:3347 +0x4b
runtime.Goexit()
	/usr/local/go/src/runtime/panic.go:563 +0xec
testing.(*common).FailNow(0xc0004c8d00)
	/usr/local/go/src/testing/testing.go:653 +0x39
testing.(*common).Fatalf(0xc0004c8d00, 0x1fc3068, 0x29, 0x0, 0x0, 0x0)
	/usr/local/go/src/testing/testing.go:716 +0x90
github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill(0xc0004c8d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:919 +0x15da
testing.tRunner(0xc0004c8d00, 0x246b748)
	/usr/local/go/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:960 +0x350

Log preceding fatal error

=== RUN   TestRouterDiskSpill
--- FAIL: TestRouterDiskSpill (0.00s)
    routers_test.go:919: expected memory error when consuming rows

More

Parameters:

TAGS=
GOFLAGS=-parallel=4

make stressrace TESTS=TestRouterDiskSpill PKG=./pkg/sql/rowflow TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-10-25T06:20:16Z

(sql/rowflow).TestRouterDiskSpill failed on master@8aceac3c99c3addece3a9ef9af04cc74715cdb37:

Fatal error:

panic: test-monitor: unexpected 8 leftover bytes [recovered]
	panic: test-monitor: unexpected 8 leftover bytes [recovered]
	panic: test-monitor: unexpected 8 leftover bytes

Stack:

goroutine 160 [running]:
testing.tRunner.func1(0xc000512400)
	/usr/local/go/src/testing/testing.go:874 +0x69f
panic(0x2a25800, 0xc0007b7ec0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:103 +0x2ae
panic(0x2a25800, 0xc0007b7ec0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/log.ReportOrPanic(0x4143640, 0xc0005aefc0, 0xc000491500, 0x2bb7b1a, 0x20, 0xc00053cec8, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/crash_reporting.go:340 +0x3ff
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop(0xc000114320, 0x4143640, 0xc0005aefc0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:406 +0x475
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).Stop(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391
github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*EvalContext).Stop(0xc0003b0240, 0x4143640, 0xc0005aefc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/eval.go:3347 +0x70
runtime.Goexit()
	/usr/local/go/src/runtime/panic.go:563 +0xec
testing.(*common).FailNow(0xc000512400)
	/usr/local/go/src/testing/testing.go:653 +0x5b
testing.(*common).Fatalf(0xc000512400, 0x2bcece0, 0x29, 0x0, 0x0, 0x0)
	/usr/local/go/src/testing/testing.go:716 +0x9e
github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill(0xc000512400)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:919 +0x1875
testing.tRunner(0xc000512400, 0x3077be8)
	/usr/local/go/src/testing/testing.go:909 +0x19a
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:960 +0x652

Log preceding fatal error

  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:406 +0x474
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*EvalContext).Stop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391 +0x6f
  runtime.call32()
      /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a
  testing.(*common).Fatalf()
      /usr/local/go/src/testing/testing.go:716 +0x9d
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:919 +0x1874
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Previous write at 0x00c00025e650 by goroutine 84:
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).releaseBytes()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:663 +0xfe
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Close()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:534 +0xc8
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).releaseBudget()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:521 +0xba
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:413 +0xbb
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*routerBase).Start.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391 +0x580

Goroutine 76 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:960 +0x651
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1202 +0xa6
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1200 +0x521
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1117 +0x2ff
  main.main()
      _testmain.go:64 +0x223

Goroutine 84 (finished) created at:
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*routerBase).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers.go:301 +0xcd
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:826 +0x166a
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
--- FAIL: TestRouterDiskSpill (0.02s)
    routers_test.go:919: expected memory error when consuming rows
    testing.go:853: race detected during execution of test

More

Parameters:

TAGS=
GOFLAGS=-race -parallel=2

make stressrace TESTS=TestRouterDiskSpill PKG=./pkg/sql/rowflow TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-10-26T06:20:47Z

(sql/rowflow).TestRouterDiskSpill failed on master@6184870a438ae34afbcf29dda5452345dc7587d3:

Fatal error:

panic: test-monitor: unexpected 88 leftover bytes [recovered]
	panic: test-monitor: unexpected 88 leftover bytes [recovered]
	panic: test-monitor: unexpected 88 leftover bytes

Stack:

goroutine 113 [running]:
testing.tRunner.func1(0xc0009ca100)
	/usr/local/go/src/testing/testing.go:874 +0x69f
panic(0x2a25cc0, 0xc0005d1320)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:103 +0x2ae
panic(0x2a25cc0, 0xc0005d1320)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/cockroachdb/cockroach/pkg/util/log.ReportOrPanic(0x4143b40, 0xc000253980, 0xc00045d500, 0x2bb7ffa, 0x20, 0xc000528ec8, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/crash_reporting.go:340 +0x3ff
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop(0xc0003c5c20, 0x4143b40, 0xc000253980, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:406 +0x475
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).Stop(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391
github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*EvalContext).Stop(0xc00036a000, 0x4143b40, 0xc000253980)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/eval.go:3347 +0x70
runtime.Goexit()
	/usr/local/go/src/runtime/panic.go:563 +0xec
testing.(*common).FailNow(0xc0009ca100)
	/usr/local/go/src/testing/testing.go:653 +0x5b
testing.(*common).Fatalf(0xc0009ca100, 0x2bcf1c0, 0x29, 0x0, 0x0, 0x0)
	/usr/local/go/src/testing/testing.go:716 +0x9e
github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill(0xc0009ca100)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:919 +0x1875
testing.tRunner(0xc0009ca100, 0x30780f8)
	/usr/local/go/src/testing/testing.go:909 +0x19a
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:960 +0x652

Log preceding fatal error

  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:406 +0x474
  github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*EvalContext).Stop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391 +0x6f
  runtime.call32()
      /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a
  testing.(*common).Fatalf()
      /usr/local/go/src/testing/testing.go:716 +0x9d
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:919 +0x1874
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Previous write at 0x00c0003c4830 by goroutine 81:
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).releaseBytes()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:663 +0xfe
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Close()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:534 +0xc8
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).releaseBudget()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:521 +0xba
  github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:413 +0xbb
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*routerBase).Start.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:391 +0x580

Goroutine 74 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:960 +0x651
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1202 +0xa6
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1200 +0x521
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1117 +0x2ff
  main.main()
      _testmain.go:64 +0x223

Goroutine 81 (finished) created at:
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*routerBase).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers.go:301 +0xcd
  github.com/cockroachdb/cockroach/pkg/sql/rowflow.TestRouterDiskSpill()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers_test.go:826 +0x166a
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
--- FAIL: TestRouterDiskSpill (0.02s)
    routers_test.go:919: expected memory error when consuming rows
    testing.go:853: race detected during execution of test

More

Parameters:

TAGS=
GOFLAGS=-race -parallel=2

make stressrace TESTS=TestRouterDiskSpill PKG=./pkg/sql/rowflow TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-10-27T06:27:55Z

(sql/rowflow).TestRouterDiskSpill failed on master@5e3c201595fc33b0d120057c61413195716f811d:

Fatal error:

panic: pebble: inconsistent reference count: 1

Stack:

goroutine 212 [running]:
github.com/cockroachdb/pebble.(*flushableEntry).readerRef(0xc0005d2050)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/flushable.go:61 +0xf0
github.com/cockroachdb/pebble.(*DB).updateReadStateLocked(0xc0003ec800, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/read_state.go:91 +0x233
github.com/cockroachdb/pebble.(*DB).makeRoomForWrite(0xc0003ec800, 0x0, 0x0, 0xc000235590)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/db.go:1498 +0x8b1
github.com/cockroachdb/pebble.(*DB).AsyncFlush(0xc0003ec800, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/db.go:1049 +0x2a4
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMap).Clear(0xc0004ce150, 0xc000235620, 0xc0002355b8)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/disk_map.go:156 +0x2bf
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMap).Close(0xc0004ce150, 0x4146e00, 0xc0004ce090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/disk_map.go:162 +0x3d
github.com/cockroachdb/cockroach/pkg/sql/rowcontainer.(*DiskRowContainer).Close(0xc000558000, 0x4146e00, 0xc0004ce090)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowcontainer/disk_row_container.go:391 +0xa6
github.com/cockroachdb/cockroach/pkg/sql/rowcontainer.(*DiskBackedRowContainer).Close(0xc0004036d8, 0x4146e00, 0xc0004ce090)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowcontainer/row_container.go:557 +0x1c2
github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*routerBase).Start.func1(0x4146e00, 0xc0004ce090, 0xc0000fe640, 0xc000403500, 0xc000d745e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers.go:384 +0x488
created by github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*routerBase).Start
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowflow/routers.go:301 +0xce

Log preceding fatal error

=== RUN   TestRouterDiskSpill
--- FAIL: TestRouterDiskSpill (0.02s)
    routers_test.go:919: expected memory error when consuming rows

More

Parameters:

TAGS=
GOFLAGS=-race -parallel=2

make stressrace TESTS=TestRouterDiskSpill PKG=./pkg/sql/rowflow TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Oct 22, 2020

cockroach-teamcity added this to the 20.2 milestone Oct 22, 2020

cockroach-teamcity assigned asubiotto Oct 22, 2020

yuzefovich assigned yuzefovich and unassigned asubiotto Oct 22, 2020

yuzefovich mentioned this issue Oct 22, 2020

rowflow: account for more memory usage of the row buffer #55877

Merged

craig bot closed this as completed in b319a0b Oct 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql/rowflow: TestRouterDiskSpill failed #55848

sql/rowflow: TestRouterDiskSpill failed #55848

cockroach-teamcity commented Oct 22, 2020

yuzefovich commented Oct 22, 2020 •

edited

Loading

cockroach-teamcity commented Oct 24, 2020

cockroach-teamcity commented Oct 25, 2020

cockroach-teamcity commented Oct 25, 2020

cockroach-teamcity commented Oct 26, 2020

cockroach-teamcity commented Oct 27, 2020

sql/rowflow: TestRouterDiskSpill failed #55848

sql/rowflow: TestRouterDiskSpill failed #55848

Comments

cockroach-teamcity commented Oct 22, 2020

yuzefovich commented Oct 22, 2020 • edited Loading

cockroach-teamcity commented Oct 24, 2020

cockroach-teamcity commented Oct 25, 2020

cockroach-teamcity commented Oct 25, 2020

cockroach-teamcity commented Oct 26, 2020

cockroach-teamcity commented Oct 27, 2020

yuzefovich commented Oct 22, 2020 •

edited

Loading