DGraph crash: Whilst using recursive. #3161

pjebs · 2019-03-18T11:06:53Z

Dgraph version   : v1.0.11
Commit SHA-1     : b2a09c5b
Commit timestamp : 2018-12-17 09:50:56 -0800
Branch           : HEAD
Go version       : go1.11.1

I ran a recursive query.

I0318 10:11:25.678778      75 groups.go:913] Zero leadership changed. Renewing oracle delta stream.
I0318 10:11:27.004691      75 groups.go:850] Leader idx=1 of group=1 is connecting to Zero for txn updates
I0318 10:11:27.011460      75 groups.go:859] Got Zero leader: localhost:5080
E0318 10:11:27.005070      75 pool.go:206] Echo error from localhost:5080. Err: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0318 10:11:27.006006      75 groups.go:889] Error in oracle delta stream. Error: rpc error: code = Canceled desc = context canceled
W0318 10:11:25.679372      75 groups.go:751] No membership update for 10s. Closing connection to Zero.
E0318 10:11:27.039284      75 groups.go:716] Unable to sync memberships. Error: rpc error: code = Canceled desc = context canceled. State: <nil>
I0318 10:11:27.043488      75 groups.go:695] Got address of a Zero leader: localhost:5080
I0318 10:11:27.046948      75 groups.go:708] Starting a new membership stream receive from localhost:5080.
I0318 10:11:27.137171      75 pool.go:217] Connection established with localhost:5080
I0318 10:11:27.138948      75 groups.go:725] Received first state update from Zero: counter:501 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:"localhost:7080" leader:true last_update:1552880537 > > tablets:<key:"_predicate_" value:<group_id:1 predicate:"_predicate_" space:17738 > > tablets:<key:"dgraph.group.acl" value:<group_id:1 predicate:"dgraph.group.acl" space:39 > > tablets:<key:"dgraph.password" value:<group_id:1 predicate:"dgraph.password" space:37 > > tablets:<key:"dgraph.user.group" value:<group_id:1 predicate:"dgraph.user.group" space:43 > > tablets:<key:"dgraph.xid" value:<group_id:1 predicate:"dgraph.xid" space:36 > > tablets:<key:"node" value:<group_id:1 predicate:"node" space:3859 > > tablets:<key:"node.hashid" value:<group_id:1 predicate:"node.hashid" space:6449 > > tablets:<key:"node.owner" value:<group_id:1 predicate:"node.owner" space:2072 > > tablets:<key:"node.parent" value:<group_id:1 predicate:"node.parent" space:3688 > > tablets:<key:"node.search_synopsis" value:<group_id:1 predicate:"node.search_synopsis" space:47 > > tablets:<key:"node.search_title" value:<group_id:1 predicate:"node.search_title" space:41 > > tablets:<key:"node.searchable" value:<group_id:1 predicate:"node.searchable" space:4563 > > tablets:<key:"node.xdata" value:<group_id:1 predicate:"node.xdata" space:5855 > > tablets:<key:"user" value:<group_id:1 predicate:"user" space:139 > > tablets:<key:"user.code" value:<group_id:1 predicate:"user.code" space:229 > > tablets:<key:"user.email" value:<group_id:1 predicate:"user.email" space:278 > > tablets:<key:"user.name" value:<group_id:1 predicate:"user.name" space:227 > > tablets:<key:"user.password" value:<group_id:1 predicate:"user.password" space:271 > > snapshot_ts:315239 > > zeros:<key:1 value:<id:1 addr:"localhost:5080" leader:true > > maxLeaseId:30000 maxTxnTs:370000 maxRaftId:1 cid:"323dfbd8-9a4e-471f-8bda-93616c751184" 
panic: runtime error: index out of range

goroutine 9299121 [running]:
github.com/dgraph-io/dgraph/query.(*SubGraph).expandRecurse(0xc00e09cdc0, 0x158b8a0, 0xc00c02b4a0, 0xffffffffffffffff, 0xc00c74ff90, 0xb20b11)
	/ext-go/1/src/github.com/dgraph-io/dgraph/query/recurse.go:104 +0xbd1
github.com/dgraph-io/dgraph/query.Recurse(0x158b8a0, 0xc00c02b4a0, 0xc00e09cdc0, 0x7f14665b3438, 0xc00dc14cd0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/query/recurse.go:190 +0x79
github.com/dgraph-io/dgraph/query.(*QueryRequest).ProcessQuery.func4(0xc00b7eb200, 0x158b8a0, 0xc00c02b4a0, 0xc00e09cdc0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/query/query.go:2510 +0x3f
created by github.com/dgraph-io/dgraph/query.(*QueryRequest).ProcessQuery
	/ext-go/1/src/github.com/dgraph-io/dgraph/query/query.go:2509 +0x737

The text was updated successfully, but these errors were encountered:

pjebs · 2019-03-18T11:08:24Z

Query:

{  
	find(func: eq(node.hashid, "7nil4uzhw5"))  @recurse   {
    uid
    expand(_all_)
  }
}

pjebs · 2019-03-18T11:32:48Z

schema:

		user: bool .
		user.name: string @index(hash) .
		user.email: string @index(hash) .
		user.password: password .
		user.code: string . 

		node: bool .
		node.hashid: string @index(hash) . 
		node.owner: uid @reverse . 
		node.parent: uid . 
		node.xdata: string . 
		node.searchable: bool . 
		node.search_title: string .
		node.search_synopsis: string .

manishrjain · 2019-03-18T12:48:37Z

@pjebs : Can you try this on the latest version of Dgraph?

pjebs · 2019-03-18T13:20:02Z

I only use docker. Has image been updated?

MichelDiz · 2019-03-18T13:39:38Z

You can use v1.0.13 https://hub.docker.com/r/dgraph/dgraph/tags

pjebs · 2019-03-18T23:15:05Z

Dgraph version   : v1.0.13
Commit SHA-1     : 691b3b35
Commit timestamp : 2019-03-09 19:33:59 -0800
Branch           : HEAD
Go version       : go1.11.5

I0318 23:11:12.637466      21 groups.go:747] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0318 23:11:12.639781      21 groups.go:756] Got Zero leader: localhost:5080
panic: runtime error: index out of range

goroutine 109984 [running]:
github.com/dgraph-io/dgraph/query.(*SubGraph).expandRecurse(0xc0003ba000, 0x15acba0, 0xc0003eac60, 0xffffffffffffffff, 0x31, 0x0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/query/recurse.go:104 +0xbda
github.com/dgraph-io/dgraph/query.Recurse(0x15acba0, 0xc0003eac60, 0xc0003ba000, 0xc00b6067b0, 0x8ee104)
	/ext-go/1/src/github.com/dgraph-io/dgraph/query/recurse.go:189 +0x79
github.com/dgraph-io/dgraph/query.(*QueryRequest).ProcessQuery.func4(0xc000418ea0, 0x15acba0, 0xc0003eac60, 0xc0003ba000)
	/ext-go/1/src/github.com/dgraph-io/dgraph/query/query.go:2547 +0x3f
created by github.com/dgraph-io/dgraph/query.(*QueryRequest).ProcessQuery
	/ext-go/1/src/github.com/dgraph-io/dgraph/query/query.go:2546 +0x737

martinmr · 2019-03-19T00:50:43Z

Can you share more of the data that lead to the breaking query? We have a test for this exact type of query but it's passing both on master and the v1.0 branch so I haven't been able to reproduce it.

pjebs · 2019-03-19T01:40:37Z

How do I share the data?

martinmr · 2019-03-19T17:47:17Z

If you can share a minimal set of triples in the RDF format that lead to this problem via something like Google drive that would be great.

danielmai · 2019-03-19T18:08:13Z

Looks like this example causes the issue on master and latest.

Run Dgraph cluster
Run schema update

user: bool .
user.name: string @index(hash) .
user.email: string @index(hash) .
user.password: password .
user.code: string . 

node: bool .
node.hashid: string @index(hash) . 
node.owner: uid @reverse . 
node.parent: uid . 
node.xdata: string . 
node.searchable: bool . 
node.search_title: string .
node.search_synopsis: string .

Run these two mutations:

{
  set {
    _:a <user> "true" .
    _:a <user.name> "A" .
    _:a <user.email> "a@example.com" .
    _:a <user.password> "apassword" .
    _:a <user.code> "acode" .
  }
}

{
  set {
    _:n <node> "true" .
    _:n <node.hashid> "7nil4uzhw5" .
    _:n <node.owner> <0x1> .
    _:n <node.parent> _:n .
  }
}

Run @recurse query:

{  
  find(func: eq(node.hashid, "7nil4uzhw5")) @recurse {
    uid
    expand(_all_)
  }
}

martinmr · 2019-03-19T18:21:12Z

This bug is present both in the v1.0 and master branches.

martinmr · 2019-03-19T22:53:33Z

#3179 should fix the issue.

martinmr self-assigned this Mar 19, 2019

danielmai added the kind/bug Something is broken. label Mar 19, 2019

martinmr mentioned this issue Mar 19, 2019

Fix bug for queries with @recurse and expand(_all_). #3179

Merged

martinmr closed this as completed in #3179 Mar 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DGraph crash: Whilst using recursive. #3161

DGraph crash: Whilst using recursive. #3161

pjebs commented Mar 18, 2019

pjebs commented Mar 18, 2019

pjebs commented Mar 18, 2019

manishrjain commented Mar 18, 2019

pjebs commented Mar 18, 2019

MichelDiz commented Mar 18, 2019

pjebs commented Mar 18, 2019

martinmr commented Mar 19, 2019

pjebs commented Mar 19, 2019

martinmr commented Mar 19, 2019 •

edited

Loading

danielmai commented Mar 19, 2019 •

edited

Loading

martinmr commented Mar 19, 2019

martinmr commented Mar 19, 2019

DGraph crash: Whilst using recursive. #3161

DGraph crash: Whilst using recursive. #3161

Comments

pjebs commented Mar 18, 2019

pjebs commented Mar 18, 2019

pjebs commented Mar 18, 2019

manishrjain commented Mar 18, 2019

pjebs commented Mar 18, 2019

MichelDiz commented Mar 18, 2019

pjebs commented Mar 18, 2019

martinmr commented Mar 19, 2019

pjebs commented Mar 19, 2019

martinmr commented Mar 19, 2019 • edited Loading

danielmai commented Mar 19, 2019 • edited Loading

martinmr commented Mar 19, 2019

martinmr commented Mar 19, 2019

martinmr commented Mar 19, 2019 •

edited

Loading

danielmai commented Mar 19, 2019 •

edited

Loading