Never call hd/1 in riak_object:most_recent_content/1 with the empty list [JIRA: RIAK-1743]

[russelldb]

Reported here http://lists.basho.com/pipermail/riak-users_lists.basho.com/2014-December/016459.html

Some merge/reconcile bug is causing the empty list to be passed to hd, probably passed to most_recent_content in fact. Investigate the cause. Add a guard earlier to catch the error sooner.

Settings are LWW=true, allow_mult=false.

[ghost]

We're actually seeing this same error.. One key appears to be the culprit... It's crashing in initialcall: riak_kv_Get_fsm:init/1

Which is awesome, in that the request actually then times out...

Any ideas as to a clean way to patch this in the meantime, either delete the key by hand, or something else.

[russelldb]

@Boardom any chance I can see this data item? You can sanitize the content/key as you see fit, but if I could see the vclock it would be really valuable for me to diagnose the root cause.

You'd need to get each key straight from it's vnode, rather than via the API. If you're willing to do this let me know, and I'll link a snippet you can run on an attached riak node to get this data for me.

[ghost]

Sure man, no problems. Shoot me the snippet

On Mon, Jan 12, 2015 at 12:41 PM, Russell Brown notifications@github.com
wrote:

@Boardom https://github.com/boardom any chance I can see this data
item? You can sanitize the content/key as you see fit, but if I could see
the vclock it would be really valuable for me to diagnose the root cause.

You'd need to get each key straight from it's vnode, rather than via the
API. If you're willing to do this let me know, and I'll link a snippet you
can run on an attached riak node to get this data for me.

—
Reply to this email directly or view it on GitHub
#1069 (comment).

[russelldb]

@Boardom if you attach to a node with

riak attach

then run https://gist.github.com/russelldb/0b11fbcea1c6f618d2f5

you well get 3 copies of the object output on the console. If you don't want to paste them here, feel free to email me russelldb@basho.com

[DSomogyi]

Comment for Jira.

[russelldb]

Current research shows this to be an issue with conflicting bucket properties.

{dvv_enabled, true}
{last_write_wins, true}

and an indexing backend (leveldb or memory.)

Work around is just to set {dvv_enabled, false} if you are using lww. We're working on code that means you can't shoot yourself in the foot with this. We have manual repro, working now on automating it ,then the fix.

[Basho-JIRA]

Advisory issued. Need to add bucket validation and fixup to make sure users can't do this by accident in the future.

_[posted via JIRA by Douglas Rohrer]_

[Basho-JIRA]

Assigned Fred per the Riak Social / Mumble

_[posted via JIRA by Patricia Brewer]_

[Basho-JIRA]

Seems the assignment of Fred did not take and this was still assigned to me. Fixed it. [~fdushin]

_[posted via JIRA by Patricia Brewer]_

[Basho-JIRA]

This was reviewed by [~fdushin], [~jmeredith] for the 2.0 merge, and by [~drohrer] for the 2.1 merge.

_[posted via JIRA by Fred Dushin]_