oniguruma segfault

[smola]

time="2018-10-05T11:14:31Z" level=info msg="NewConnection: client 305"
time="2018-10-05T11:14:31Z" level=debug msg="executing query" query="select\n    files.file_path, ref_commits.repository_id, files.blob_content\nFROM\n    files\nNATURAL JOIN\n    commit_files\nNATURAL JOIN\n    ref_commits\nWHERE\n    ref_commits.ref_name = 'HEAD'\n    AND ref_commits.history_index BETWEEN 0 AND 5\n    AND is_binary(blob_content) = false\n    AND files.file_path NOT REGEXP '^vendor.*'\n    AND (\n        blob_content REGEXP '(?i)facebook.*[\\'\\\\\"][0-9a-f]{32}[\\'\\\\\"]'\n        OR blob_content REGEXP '(?i)twitter.*[\\'\\\\\"][0-9a-zA-Z]{35,44}[\\'\\\\\"]'\n        OR blob_content REGEXP '(?i)github.*[\\'\\\\\"][0-9a-zA-Z]{35,40}[\\'\\\\\"]'\n        OR blob_content REGEXP 'AKIA[0-9A-Z]{16}'\n        OR blob_content REGEXP '(?i)heroku.*[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}'\n        OR blob_content REGEXP '.*-----BEGIN PRIVATE KEY-----.*'\n        OR blob_content REGEXP '.*-----BEGIN RSA PRIVATE KEY-----.*'\n        OR blob_content REGEXP '.*-----BEGIN DSA PRIVATE KEY-----.*'\n        OR blob_content REGEXP '.*-----BEGIN OPENSSH PRIVATE KEY-----.*'\n    )"
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7f7222ffe000 pc=0x7f7247c97e2f]

runtime stack:
runtime.throw(0x11eb922, 0x2a)
	/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:374 +0x2f2

goroutine 3123 [syscall]:
runtime.cgocall(0xe4f670, 0xc062b577b0, 0x29)
	/usr/local/go/src/runtime/cgocall.go:128 +0x5e fp=0xc062b57778 sp=0xc062b57740 pc=0x4078de
github.com/src-d/gitbase/vendor/github.com/moovweb/rubex._Cfunc_SearchOnigRegex(0xc020cf6000, 0x25181, 0x0, 0x7f71d40019c0, 0x7f71d4001d00, 0x7f71d4001ce0, 0x0, 0x0, 0x0, 0x0)
	_cgo_gotypes.go:285 +0x50 fp=0xc062b577b0 sp=0xc062b57778 pc=0x8a9ed0
github.com/src-d/gitbase/vendor/github.com/moovweb/rubex.(*Regexp).match.func1(0xc020cf6000, 0x25181, 0xc000000000, 0x7f71d40019c0, 0x7f71d4001d00, 0x7f71d4001ce0, 0x0, 0x0, 0x0, 0xc020cf6000)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/github.com/moovweb/rubex/regex.go:209 +0x144 fp=0xc062b57810 sp=0xc062b577b0 pc=0x8b0044
github.com/src-d/gitbase/vendor/github.com/moovweb/rubex.(*Regexp).match(0xc0deade500, 0xc020cf6000, 0x25181, 0x26000, 0x25181, 0x0, 0xc020cf6000)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/github.com/moovweb/rubex/regex.go:209 +0x9e fp=0xc062b57870 sp=0xc062b57810 pc=0x8ab39e
github.com/src-d/gitbase/vendor/github.com/moovweb/rubex.(*Regexp).Match(0xc0deade500, 0xc020cf6000, 0x25181, 0x26000, 0x25181)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/github.com/moovweb/rubex/regex.go:444 +0x57 fp=0xc062b578b8 sp=0xc062b57870 pc=0x8ad887
github.com/src-d/gitbase/vendor/github.com/moovweb/rubex.(*Regexp).MatchString(0xc0deade500, 0xc0a359a000, 0x25181, 0x28)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/github.com/moovweb/rubex/regex.go:449 +0x6a fp=0xc062b578f8 sp=0xc062b578b8 pc=0x8ad91a
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/internal/regex.(*Oniguruma).Match(0xc0441fc378, 0xc0a359a000, 0x25181, 0xc05fb82f10)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/internal/regex/regex_oniguruma.go:14 +0x42 fp=0xc062b57928 sp=0xc062b578f8 pc=0x8b0cc2
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression.(*Regexp).compareRegexp(0xc04b833500, 0xc096206c00, 0xc0689920a0, 0xa, 0xa, 0xa, 0xef96c0, 0x21acf40, 0x0)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression/comparison.go:234 +0x17c fp=0xc062b579b0 sp=0xc062b57928 pc=0x8b722c
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression.(*Regexp).Eval(0xc04b833500, 0xc096206c00, 0xc0689920a0, 0xa, 0xa, 0xef96c0, 0x21acf40, 0x0, 0x0)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression/comparison.go:192 +0x1f6 fp=0xc062b57a18 sp=0xc062b579b0 pc=0x8b6f96
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression.(*Or).Eval(0xc012cdc300, 0xc096206c00, 0xc0689920a0, 0xa, 0xa, 0x0, 0xc0004bd500, 0xc062b57b18, 0x7f72491f6d80)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression/logic.go:116 +0xdb fp=0xc062b57a78 sp=0xc062b57a18 pc=0x8bc91b
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression.(*Or).Eval(0xc012cdc340, 0xc096206c00, 0xc0689920a0, 0xa, 0xa, 0xf50d00, 0xc05c494301, 0xa, 0xc062b57bd0)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression/logic.go:107 +0x62 fp=0xc062b57ad8 sp=0xc062b57a78 pc=0x8bc8a2
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression.(*Or).Eval(0xc012cdc380, 0xc096206c00, 0xc0689920a0, 0xa, 0xa, 0xa, 0x0, 0xa, 0x10)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression/logic.go:107 +0x62 fp=0xc062b57b38 sp=0xc062b57ad8 pc=0x8bc8a2
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression.(*Or).Eval(0xc012cdc3c0, 0xc096206c00, 0xc0689920a0, 0xa, 0xa, 0xc05c494300, 0xc00028b710, 0xc062b57c40, 0x8d8f03)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression/logic.go:107 +0x62 fp=0xc062b57b98 sp=0xc062b57b38 pc=0x8bc8a2
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression.(*Or).Eval(0xc012cdc400, 0xc096206c00, 0xc0689920a0, 0xa, 0xa, 0xa, 0xa, 0x0, 0x0)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression/logic.go:107 +0x62 fp=0xc062b57bf8 sp=0xc062b57b98 pc=0x8bc8a2
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression.(*Or).Eval(0xc012cdc440, 0xc096206c00, 0xc0689920a0, 0xa, 0xa, 0xef96c0, 0x21acf40, 0x0, 0x0)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression/logic.go:107 +0x62 fp=0xc062b57c58 sp=0xc062b57bf8 pc=0x8bc8a2
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression.(*Or).Eval(0xc012cdc480, 0xc096206c00, 0xc0689920a0, 0xa, 0xa, 0x0, 0x21acf40, 0x0, 0x0)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/expression/logic.go:107 +0x62 fp=0xc062b57cb8 sp=0xc062b57c58 pc=0x8bc8a2
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*FilterIter).Next(0xc0339fa1b0, 0x3, 0x13aebc0, 0x0, 0x0, 0x2)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/filter.go:110 +0x97 fp=0xc062b57d28 sp=0xc062b57cb8 pc=0x8cf7e7
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql.(*spanIter).Next(0xc0339fa240, 0xc025d45eb2, 0xc025d45e58, 0xc025d45f60, 0x0, 0xc0339fa270)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/session.go:194 +0x47 fp=0xc062b57d90 sp=0xc062b57d28 pc=0x79eba7
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*iter).Next(0xc012c66c20, 0xc03cd878c0, 0x2241430, 0xc0339fa240, 0xc025d45e58, 0x4de79d)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/project.go:129 +0x38 fp=0xc062b57e00 sp=0xc062b57d90 pc=0x8d8e78
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql.(*spanIter).Next(0xc0339fa270, 0xc025d45eac, 0x3, 0x2, 0xc00f1cac00, 0x0)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/session.go:194 +0x47 fp=0xc062b57e68 sp=0xc062b57e00 pc=0x79eba7
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*exchangeRowIter).iterPartition(0xc00f1cac00, 0x13935c0, 0xc0111c0ce0)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/exchange.go:241 +0x251 fp=0xc062b57f98 sp=0xc062b57e68 pc=0x8ce6d1
github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*exchangeRowIter).start.func1(0xc00f1cac00, 0xc03b16e764, 0x13935c0, 0xc0111c0ce0)
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/exchange.go:168 +0x3f fp=0xc062b57fc0 sp=0xc062b57f98 pc=0x8e0c5f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc062b57fc8 sp=0xc062b57fc0 pc=0x45c9f1
created by github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan.(*exchangeRowIter).start
	/home/francesc/go/src/github.com/src-d/gitbase/vendor/gopkg.in/src-d/go-mysql-server.v0/sql/plan/exchange.go:167 +0x10d

[jfontan]

It seems to be a concurrency problem with oniguruma regexps. I'm not able to reproduce the error in my machine but created a binary with locks around regexp. This will be tested in a many cores machine. If this goes well I'll apply the changes in go-mysql-server and enry.

[jfontan]

We are going to implement another library and check if we have the same problems.

[vmarkovtsev]

Related info: k-takata/Onigmo#63 (comment)

They say it is thread safe on >=6.0. What's the installed libonig.so version in the docker image? If the code runs on an ancient Debian I wouldn't be surprised if it is older.

[vmarkovtsev]

https://github.com/rubinius/oniguruma corresponds to 5.9.2 from 2013

https://github.com/kkos/oniguruma is 6.9, updated and maintained

[smola]

The machine where we saw this had libonig-dev 6.1.3-2 installed... but I think the bug happened with the statically linked binary published to GitHub releases.

I'm not sure where we got that oniguruma version (cc @jfontan). If it was libonig-dev package from Travis, that was probably Ubuntu Trusty's: 5.9.1-1ubuntu1.1.

[smola]

@campoy I see you manually built some binaries on that machine too, do you know if panics happened with oniguruma-enabled builds when built directly from source?

[jfontan]

@smola the version was 6.8.2, from alpine. It was statically linked in a docker container.

Still, oniguruma itself is not the problem, is the golang bindings. The buffers where to write intermediate results are the same for each compiled regexp (region and errorInfo). Most probably the race comes from that call.

We could fix that with one regexp node per thread, adding locking or changing the library to use different region and errorInfo each time it's called. I prefer the first one, less error prone.

https://github.com/moovweb/rubex/blob/master/regex.go#L208

[smola]

As discussed previously, I would also prefer one regexp node per thread. Just imagine thread contention in a 96 cores machine all waiting for a regexp lock...

[campoy]

I do not think there were any crashes with the binary built from source, but not 100% sure.

Also, 96 cores waiting for regexp lock is a great topic for Halloween 😨

[vmarkovtsev]

It's a pity https://github.com/moovweb/rubex is archived so we cannot fix that for everyone...